2024-04-25T05:20:41.520 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T05:20:41.520 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832 --name adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} --owner scheduled_adking@teuthology --verbose -- /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/orig.config.yaml 2024-04-25T05:20:41.554 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T05:20:41.640 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832 branch: wip-adk-testing-2024-04-24-1625 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: adking@redhat.com first_in_suite: false job_id: '7672832' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-04-24-1625 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2024-04-24-1625 sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8141 sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-04-24-1625 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhqdfEzSQ81OIS5ZAnfcJw/d+GlfTl96+NsKXGawO/u3CHI7aVBJ6khr5CvFs+uDnjGoxWblUHW/6UOTwRyUZA= smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGxt86lJws1LfuJltAWW8ITFxRFw3JxvhGm7dCUTFH1X9rVnvAh19qF9tLieZDMptXu4LG0A+q07xM97I+d6niU= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_03:53:55 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T05:20:41.640 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/qa; will attempt to use it 2024-04-25T05:20:41.641 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/qa/tasks 2024-04-25T05:20:41.641 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T05:20:41.642 INFO:teuthology.task.internal:Checking packages... 2024-04-25T05:20:41.668 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737' 2024-04-25T05:20:41.668 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T05:20:41.669 INFO:teuthology.packaging:ref: None 2024-04-25T05:20:41.669 INFO:teuthology.packaging:tag: None 2024-04-25T05:20:41.669 INFO:teuthology.packaging:branch: wip-adk-testing-2024-04-24-1625 2024-04-25T05:20:41.669 INFO:teuthology.packaging:sha1: d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:20:41.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-04-24-1625 2024-04-25T05:20:41.815 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3293.gd6f5f4e5 2024-04-25T05:20:41.816 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T05:20:41.823 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T05:20:41.823 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T05:20:41.999 INFO:teuthology.task.internal:Saving configuration 2024-04-25T05:20:42.012 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T05:20:42.123 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T05:20:42.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 05:16:00.138583', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhqdfEzSQ81OIS5ZAnfcJw/d+GlfTl96+NsKXGawO/u3CHI7aVBJ6khr5CvFs+uDnjGoxWblUHW/6UOTwRyUZA='} 2024-04-25T05:20:42.166 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 05:16:00.137202', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGxt86lJws1LfuJltAWW8ITFxRFw3JxvhGm7dCUTFH1X9rVnvAh19qF9tLieZDMptXu4LG0A+q07xM97I+d6niU='} 2024-04-25T05:20:42.166 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T05:20:42.173 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-25T05:20:42.173 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-25T05:20:42.173 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T05:20:42.275 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec23795160>, signals=[15]) 2024-04-25T05:20:42.275 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T05:20:42.507 INFO:teuthology.task.internal:Opening connections... 2024-04-25T05:20:42.507 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T05:20:42.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T05:20:42.588 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-04-25T05:20:42.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T05:20:42.666 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T05:20:42.673 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-04-25T05:20:42.691 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-04-25T05:20:42.692 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T05:20:42.751 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-04-25T05:20:42.752 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T05:20:42.752 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T05:20:42.752 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T05:20:42.752 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-04-25T05:20:42.777 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-04-25T05:20:42.796 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-04-25T05:20:42.797 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-04-25T05:20:42.856 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T05:20:42.857 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T05:20:42.858 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-04-25T05:20:42.877 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T05:20:42.891 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T05:20:43.015 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T05:20:43.016 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-04-25T05:20:43.018 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-04-25T05:20:43.036 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T05:20:43.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T05:20:43.088 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-04-25T05:20:43.090 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-04-25T05:20:43.115 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T05:20:43.336 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T05:20:43.468 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-25T05:20:43.469 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-25T05:20:43.469 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T05:20:43.470 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T05:20:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:20:43.487 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-04-25T05:20:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:20:43.488 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-04-25T05:20:43.546 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-437.el9.x86_64 2024-04-25T05:20:43.547 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.14.0-437.el9.x86_64 2024-04-25T05:20:43.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-04-25T05:20:43.548 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-437.el9.x86_64 2024-04-25T05:20:43.548 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-437.el9.x86_64 2024-04-25T05:20:43.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-04-25T05:20:45.706 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-25T05:20:46.692 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-04-25T05:20:47.408 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-04-25T05:20:49.801 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:02 2024-04-25T05:20:51.441 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.2 MB 00:00 2024-04-25T05:20:53.101 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-25T05:20:54.085 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-25T05:20:56.094 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-04-25T05:21:08.401 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:15 2024-04-25T05:21:10.935 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:14 2024-04-25T05:21:13.567 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T05:21:15.507 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T05:21:15.507 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T05:21:15.595 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:21:15.597 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:21:15.597 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:21:15.690 DEBUG:teuthology.orchestra.run.smithi060:> echo no | sudo yum reinstall kernel || true 2024-04-25T05:21:16.126 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-04-25T05:21:16.209 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 05:21:13 AM UTC. 2024-04-25T05:21:16.333 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:21:16.349 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:16.349 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:Reinstalling: 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 255 k 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 0 2024-04-25T05:21:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:Operation aborted. 2024-04-25T05:21:16.436 INFO:teuthology.orchestra.run.smithi060.stdout:Is this ok [y/N]: 2024-04-25T05:21:16.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum reinstall -y kernel || true 2024-04-25T05:21:16.955 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 05:21:13 AM UTC. 2024-04-25T05:21:17.076 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:Reinstalling: 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:17.077 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 255 k 2024-04-25T05:21:17.078 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 0 2024-04-25T05:21:17.078 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:21:17.386 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.3 MB/s | 255 kB 00:00 2024-04-25T05:21:17.387 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:21:17.387 INFO:teuthology.orchestra.run.smithi060.stdout:Total 825 kB/s | 255 kB 00:00 2024-04-25T05:21:17.403 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:21:17.580 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:21:17.580 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:21:17.585 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:21:17.586 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:21:17.726 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:21:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T05:21:17.813 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:18.124 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T05:21:18.124 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T05:21:18.222 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:21:18.223 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:21:18.223 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:21:18.326 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2024-04-25T05:21:18.546 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:18.546 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T05:21:18.888 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 05:21:16 AM UTC. 2024-04-25T05:21:19.012 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-04-25T05:21:19.031 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-04-25T05:21:19.032 INFO:teuthology.orchestra.run.smithi043.stderr:Operation aborted. 2024-04-25T05:21:19.089 INFO:teuthology.orchestra.run.smithi043.stdout:Is this ok [y/N]: 2024-04-25T05:21:19.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout:Reinstalled: 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:21:19.341 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:21:19.500 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T05:21:19.589 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T05:21:19.590 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T05:21:19.590 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T05:21:19.590 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T05:21:19.590 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T05:21:19.606 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 05:21:16 AM UTC. 2024-04-25T05:21:19.629 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-04-25T05:21:19.645 DEBUG:teuthology.parallel:result is None 2024-04-25T05:21:19.726 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:19.727 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-04-25T05:21:19.728 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-04-25T05:21:19.728 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:21:19.987 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-25T05:21:19.987 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:21:19.987 INFO:teuthology.orchestra.run.smithi043.stdout:Total 982 kB/s | 255 kB 00:00 2024-04-25T05:21:20.006 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:21:20.206 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:21:20.206 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:21:20.211 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:21:20.211 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:21:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:21:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T05:21:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:21.084 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:21.084 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T05:21:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T05:21:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:21.840 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalled: 2024-04-25T05:21:21.840 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T05:21:21.841 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:21:21.841 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:21:22.016 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T05:21:22.064 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T05:21:22.064 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T05:21:22.064 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T05:21:22.064 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T05:21:22.064 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T05:21:22.100 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-04-25T05:21:22.120 DEBUG:teuthology.parallel:result is None 2024-04-25T05:21:22.121 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T05:21:22.128 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T05:21:22.129 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T05:21:22.163 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T05:21:22.181 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T05:21:22.217 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T05:21:22.384 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T05:21:22.384 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T05:21:22.386 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T05:21:22.423 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T05:21:22.485 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T05:21:22.486 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T05:21:22.488 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T05:21:22.515 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T05:21:22.521 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T05:21:22.525 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T05:21:22.533 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T05:21:22.535 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T05:21:22.636 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T05:21:22.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T05:21:22.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T05:21:22.668 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T05:21:22.711 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T05:21:22.712 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T05:21:22.714 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T05:21:22.731 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T05:21:22.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T05:21:22.859 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T05:21:22.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T05:21:22.980 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:21:22.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T05:21:23.044 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T05:21:23.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T05:21:23.146 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T05:21:23.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T05:21:23.271 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:21:23.272 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T05:21:23.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-04-25T05:21:23.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-04-25T05:21:23.382 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T05:21:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T05:21:23.783 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T05:21:23.791 INFO:teuthology.task.internal:Starting timer... 2024-04-25T05:21:23.791 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T05:21:23.822 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T05:21:23.928 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-25T05:21:23.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-04-25T05:21:23.993 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-04-25T05:21:23.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-04-25T05:21:24.043 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-04-25T05:21:24.045 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T05:21:24.045 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-04-25T05:21:24.069 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-04-25T05:21:24.070 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-04-25T05:21:24.092 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-04-25T05:21:24.092 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi060.front.sepia.ceph.com': 'permissive'} 2024-04-25T05:21:24.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-25T05:21:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:21:24.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-25T05:21:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:21:24.160 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T05:21:24.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-04-25T05:21:24.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-04-25T05:21:24.226 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T05:21:24.236 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T05:21:24.252 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T05:21:24.253 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T05:25:00.674 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-04-25T05:25:00.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-25T05:25:00.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T05:25:00.754 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-04-25T05:25:00.837 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-25T05:25:00.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-25T05:25:00.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T05:25:00.916 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-04-25T05:25:00.995 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-04-25T05:25:00.995 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T05:25:01.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T05:25:01.007 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T05:25:01.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T05:25:01.010 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T05:25:01.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T05:25:01.036 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T05:25:01.051 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T05:25:01.066 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T05:25:01.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-04-25T05:25:01.081 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T05:25:01.084 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-04-25T05:25:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T05:25:01.106 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-04-25T05:25:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T05:25:01.117 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-04-25T05:25:01.131 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T05:25:01.145 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T05:25:01.164 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.167 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.199 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-25T05:25:01.203 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T05:25:01.203 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-25T05:25:01.203 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.204 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.204 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.204 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:25:01.205 INFO:teuthology.run_tasks:Running task install... 2024-04-25T05:25:01.213 DEBUG:teuthology.task.install:project ceph 2024-04-25T05:25:01.213 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737'}} 2024-04-25T05:25:01.213 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737'} 2024-04-25T05:25:01.213 INFO:teuthology.task.install:Using flavor: default 2024-04-25T05:25:01.219 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T05:25:01.219 INFO:teuthology.task.install:extra packages: [] 2024-04-25T05:25:01.220 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737', 'tag': None, 'wait_for_package': False} 2024-04-25T05:25:01.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:25:01.222 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737', 'tag': None, 'wait_for_package': False} 2024-04-25T05:25:01.223 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:25:01.395 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/ 2024-04-25T05:25:01.395 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3293.gd6f5f4e5 2024-04-25T05:25:01.401 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/ 2024-04-25T05:25:01.402 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3293.gd6f5f4e5 2024-04-25T05:25:01.519 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T05:25:01.519 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:25:01.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T05:25:01.526 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T05:25:01.527 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:25:01.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T05:25:01.546 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-25T05:25:01.547 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T05:25:01.553 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-25T05:25:01.553 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T05:25:01.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T05:25:01.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T05:25:01.688 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T05:25:01.693 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T05:25:01.752 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-04-25T05:25:01.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-04-25T05:25:01.757 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-04-25T05:25:01.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-04-25T05:25:01.988 INFO:teuthology.orchestra.run.smithi060.stdout:48 files removed 2024-04-25T05:25:01.994 INFO:teuthology.orchestra.run.smithi043.stdout:48 files removed 2024-04-25T05:25:02.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-04-25T05:25:02.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw 2024-04-25T05:25:02.629 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 248 kB/s | 84 kB 00:00 2024-04-25T05:25:02.645 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 235 kB/s | 84 kB 00:00 2024-04-25T05:25:02.996 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-25T05:25:03.012 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-04-25T05:25:03.339 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-25T05:25:03.363 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-25T05:25:03.989 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T05:25:04.472 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-04-25T05:25:05.873 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-25T05:25:06.306 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-25T05:25:10.016 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-25T05:25:10.583 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-25T05:25:11.658 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-25T05:25:12.425 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-25T05:25:12.610 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-25T05:25:17.835 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-25T05:25:20.098 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:25:20.100 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:20.100 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:25:20.100 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:20.100 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:25:20.100 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 11 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.4 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.1 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 5.4 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 21 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 25 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 746 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T05:25:20.101 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 500 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 5.4 M 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 45 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 135 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 161 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 321 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 299 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 99 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T05:25:20.102 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Installing weak dependencies: 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Install 30 Packages 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:20.103 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 60 M 2024-04-25T05:25:20.104 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:25:21.586 INFO:teuthology.orchestra.run.smithi043.stdout:(1/32): ceph-base-19.0.0-3293.gd6f5f4e5.el9.x86 5.6 MB/s | 5.4 MB 00:00 2024-04-25T05:25:21.628 INFO:teuthology.orchestra.run.smithi043.stdout:(2/32): ceph-selinux-19.0.0-3293.gd6f5f4e5.el9. 610 kB/s | 25 kB 00:00 2024-04-25T05:25:21.720 INFO:teuthology.orchestra.run.smithi043.stdout:(3/32): libcephfs2-19.0.0-3293.gd6f5f4e5.el9.x8 7.9 MB/s | 746 kB 00:00 2024-04-25T05:25:21.795 INFO:teuthology.orchestra.run.smithi043.stdout:(4/32): libradosstriper1-19.0.0-3293.gd6f5f4e5. 6.5 MB/s | 500 kB 00:00 2024-04-25T05:25:22.171 INFO:teuthology.orchestra.run.smithi043.stdout:(5/32): librgw2-19.0.0-3293.gd6f5f4e5.el9.x86_6 14 MB/s | 5.4 MB 00:00 2024-04-25T05:25:22.212 INFO:teuthology.orchestra.run.smithi043.stdout:(6/32): python3-ceph-argparse-19.0.0-3293.gd6f5 1.1 MB/s | 45 kB 00:00 2024-04-25T05:25:22.271 INFO:teuthology.orchestra.run.smithi043.stdout:(7/32): python3-ceph-common-19.0.0-3293.gd6f5f4 2.3 MB/s | 135 kB 00:00 2024-04-25T05:25:22.321 INFO:teuthology.orchestra.run.smithi043.stdout:(8/32): python3-cephfs-19.0.0-3293.gd6f5f4e5.el 3.2 MB/s | 161 kB 00:00 2024-04-25T05:25:22.380 INFO:teuthology.orchestra.run.smithi043.stdout:(9/32): python3-rados-19.0.0-3293.gd6f5f4e5.el9 5.4 MB/s | 321 kB 00:00 2024-04-25T05:25:22.447 INFO:teuthology.orchestra.run.smithi043.stdout:(10/32): python3-rbd-19.0.0-3293.gd6f5f4e5.el9. 4.4 MB/s | 299 kB 00:00 2024-04-25T05:25:22.514 INFO:teuthology.orchestra.run.smithi043.stdout:(11/32): python3-rgw-19.0.0-3293.gd6f5f4e5.el9. 1.5 MB/s | 99 kB 00:00 2024-04-25T05:25:22.806 INFO:teuthology.orchestra.run.smithi043.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 367 kB/s | 107 kB 00:00 2024-04-25T05:25:22.939 INFO:teuthology.orchestra.run.smithi043.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-25T05:25:23.056 INFO:teuthology.orchestra.run.smithi043.stdout:(14/32): ceph-radosgw-19.0.0-3293.gd6f5f4e5.el9 4.4 MB/s | 11 MB 00:02 2024-04-25T05:25:23.082 INFO:teuthology.orchestra.run.smithi043.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-04-25T05:25:23.223 INFO:teuthology.orchestra.run.smithi043.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-04-25T05:25:23.307 INFO:teuthology.orchestra.run.smithi043.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 507 kB/s | 42 kB 00:00 2024-04-25T05:25:23.391 INFO:teuthology.orchestra.run.smithi043.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-25T05:25:23.474 INFO:teuthology.orchestra.run.smithi043.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 431 kB/s | 36 kB 00:00 2024-04-25T05:25:23.566 INFO:teuthology.orchestra.run.smithi043.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 518 kB/s | 47 kB 00:00 2024-04-25T05:25:23.725 INFO:teuthology.orchestra.run.smithi043.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 991 kB/s | 662 kB 00:00 2024-04-25T05:25:23.959 INFO:teuthology.orchestra.run.smithi043.stdout:(22/32): ceph-common-19.0.0-3293.gd6f5f4e5.el9. 6.3 MB/s | 21 MB 00:03 2024-04-25T05:25:24.140 INFO:teuthology.orchestra.run.smithi043.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 144 kB/s | 26 kB 00:00 2024-04-25T05:25:24.276 INFO:teuthology.orchestra.run.smithi043.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 358 kB/s | 49 kB 00:00 2024-04-25T05:25:24.301 INFO:teuthology.orchestra.run.smithi043.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 420 kB/s | 309 kB 00:00 2024-04-25T05:25:24.352 INFO:teuthology.orchestra.run.smithi043.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-04-25T05:25:24.461 INFO:teuthology.orchestra.run.smithi043.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-04-25T05:25:24.486 INFO:teuthology.orchestra.run.smithi043.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-25T05:25:24.694 INFO:teuthology.orchestra.run.smithi043.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-04-25T05:25:24.728 INFO:teuthology.orchestra.run.smithi043.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-04-25T05:25:24.870 INFO:teuthology.orchestra.run.smithi043.stdout:(31/32): librados2-19.0.0-3293.gd6f5f4e5.el9.x8 8.9 MB/s | 3.4 MB 00:00 2024-04-25T05:25:25.503 INFO:teuthology.orchestra.run.smithi043.stdout:(32/32): librbd1-19.0.0-3293.gd6f5f4e5.el9.x86_ 3.9 MB/s | 3.1 MB 00:00 2024-04-25T05:25:25.506 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:25:25.506 INFO:teuthology.orchestra.run.smithi043.stdout:Total 11 MB/s | 60 MB 00:05 2024-04-25T05:25:25.612 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:25:25.652 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:25:25.652 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:25:26.146 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:25:26.146 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:25:27.653 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:25:27.730 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-25T05:25:27.767 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-25T05:25:27.792 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-25T05:25:27.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T05:25:27.837 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T05:25:28.151 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T05:25:28.207 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T05:25:28.600 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-25T05:25:28.654 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:25:28.747 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:25:28.836 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 8/34 2024-04-25T05:25:28.864 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 9/34 2024-04-25T05:25:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 9/34 2024-04-25T05:25:28.964 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-25T05:25:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9. 11/34 2024-04-25T05:25:29.078 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 12/34 2024-04-25T05:25:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 13/34 2024-04-25T05:25:29.155 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 13/34 2024-04-25T05:25:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-25T05:25:29.303 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-25T05:25:29.332 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-25T05:25:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-25T05:25:29.406 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-25T05:25:29.435 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-25T05:25:29.467 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-25T05:25:29.798 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-25T05:25:29.839 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 22/34 2024-04-25T05:25:29.930 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 22/34 2024-04-25T05:25:29.960 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 23/34 2024-04-25T05:25:29.997 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-25T05:25:30.503 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-25T05:25:30.533 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 26/34 2024-04-25T05:25:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 26/34 2024-04-25T05:25:30.685 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 27/34 2024-04-25T05:25:30.719 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x8 28/34 2024-04-25T05:25:32.558 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-04-25T05:25:33.131 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:33.173 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:33.747 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:33.785 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 30/34 2024-04-25T05:25:33.878 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 30/34 2024-04-25T05:25:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T05:25:33.879 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T05:25:33.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:33.929 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:25:37.741 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-25T05:25:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:25:39.989 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 11 M 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.4 M 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.1 M 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 5.4 M 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 21 M 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 25 k 2024-04-25T05:25:39.990 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 746 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 500 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 5.4 M 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 45 k 2024-04-25T05:25:39.991 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 135 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 161 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 321 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 299 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 99 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout:Install 30 Packages 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-04-25T05:25:39.992 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 60 M 2024-04-25T05:25:39.993 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:25:41.001 INFO:teuthology.orchestra.run.smithi060.stdout:(1/32): ceph-base-19.0.0-3293.gd6f5f4e5.el9.x86 6.6 MB/s | 5.4 MB 00:00 2024-04-25T05:25:41.059 INFO:teuthology.orchestra.run.smithi060.stdout:(2/32): ceph-selinux-19.0.0-3293.gd6f5f4e5.el9. 433 kB/s | 25 kB 00:00 2024-04-25T05:25:41.168 INFO:teuthology.orchestra.run.smithi060.stdout:(3/32): libcephfs2-19.0.0-3293.gd6f5f4e5.el9.x8 6.7 MB/s | 746 kB 00:00 2024-04-25T05:25:41.268 INFO:teuthology.orchestra.run.smithi060.stdout:(4/32): libradosstriper1-19.0.0-3293.gd6f5f4e5. 4.9 MB/s | 500 kB 00:00 2024-04-25T05:25:41.935 INFO:teuthology.orchestra.run.smithi060.stdout:(5/32): librgw2-19.0.0-3293.gd6f5f4e5.el9.x86_6 8.1 MB/s | 5.4 MB 00:00 2024-04-25T05:25:42.069 INFO:teuthology.orchestra.run.smithi060.stdout:(6/32): ceph-radosgw-19.0.0-3293.gd6f5f4e5.el9. 5.6 MB/s | 11 MB 00:01 2024-04-25T05:25:42.094 INFO:teuthology.orchestra.run.smithi060.stdout:(7/32): python3-ceph-argparse-19.0.0-3293.gd6f5 285 kB/s | 45 kB 00:00 2024-04-25T05:25:42.119 INFO:teuthology.orchestra.run.smithi060.stdout:(8/32): python3-ceph-common-19.0.0-3293.gd6f5f4 2.6 MB/s | 135 kB 00:00 2024-04-25T05:25:42.145 INFO:teuthology.orchestra.run.smithi060.stdout:(9/32): python3-cephfs-19.0.0-3293.gd6f5f4e5.el 3.1 MB/s | 161 kB 00:00 2024-04-25T05:25:42.170 INFO:teuthology.orchestra.run.smithi060.stdout:(10/32): python3-rados-19.0.0-3293.gd6f5f4e5.el 6.2 MB/s | 321 kB 00:00 2024-04-25T05:25:42.195 INFO:teuthology.orchestra.run.smithi060.stdout:(11/32): python3-rbd-19.0.0-3293.gd6f5f4e5.el9. 5.8 MB/s | 299 kB 00:00 2024-04-25T05:25:42.220 INFO:teuthology.orchestra.run.smithi060.stdout:(12/32): python3-rgw-19.0.0-3293.gd6f5f4e5.el9. 1.9 MB/s | 99 kB 00:00 2024-04-25T05:25:42.429 INFO:teuthology.orchestra.run.smithi060.stdout:(13/32): ceph-common-19.0.0-3293.gd6f5f4e5.el9. 9.4 MB/s | 21 MB 00:02 2024-04-25T05:25:42.588 INFO:teuthology.orchestra.run.smithi060.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2024-04-25T05:25:42.775 INFO:teuthology.orchestra.run.smithi060.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 184 kB/s | 107 kB 00:00 2024-04-25T05:25:42.814 INFO:teuthology.orchestra.run.smithi060.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 270 kB/s | 160 kB 00:00 2024-04-25T05:25:42.905 INFO:teuthology.orchestra.run.smithi060.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 462 kB/s | 42 kB 00:00 2024-04-25T05:25:43.039 INFO:teuthology.orchestra.run.smithi060.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-04-25T05:25:43.098 INFO:teuthology.orchestra.run.smithi060.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-04-25T05:25:43.190 INFO:teuthology.orchestra.run.smithi060.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 239 kB/s | 36 kB 00:00 2024-04-25T05:25:43.215 INFO:teuthology.orchestra.run.smithi060.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2024-04-25T05:25:43.249 INFO:teuthology.orchestra.run.smithi060.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 803 kB/s | 47 kB 00:00 2024-04-25T05:25:43.499 INFO:teuthology.orchestra.run.smithi060.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-04-25T05:25:43.566 INFO:teuthology.orchestra.run.smithi060.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 821 kB/s | 309 kB 00:00 2024-04-25T05:25:43.594 INFO:teuthology.orchestra.run.smithi060.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 516 kB/s | 49 kB 00:00 2024-04-25T05:25:43.627 INFO:teuthology.orchestra.run.smithi060.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-25T05:25:43.702 INFO:teuthology.orchestra.run.smithi060.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-25T05:25:43.727 INFO:teuthology.orchestra.run.smithi060.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-25T05:25:43.811 INFO:teuthology.orchestra.run.smithi060.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-04-25T05:25:43.845 INFO:teuthology.orchestra.run.smithi060.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-25T05:25:44.054 INFO:teuthology.orchestra.run.smithi060.stdout:(31/32): librados2-19.0.0-3293.gd6f5f4e5.el9.x8 11 MB/s | 3.4 MB 00:00 2024-04-25T05:25:44.095 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /root 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-04-25T05:25:44.096 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:44.295 INFO:teuthology.orchestra.run.smithi060.stdout:(32/32): librbd1-19.0.0-3293.gd6f5f4e5.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-04-25T05:25:44.301 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:25:44.301 INFO:teuthology.orchestra.run.smithi060.stdout:Total 14 MB/s | 60 MB 00:04 2024-04-25T05:25:44.407 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:25:44.447 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:25:44.447 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:25:44.940 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:25:44.942 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:25:45.113 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/34 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/34 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T05:25:45.138 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:45.140 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T05:25:45.224 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T05:25:45.224 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:25:46.458 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 5/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 6/34 2024-04-25T05:25:46.503 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9. 8/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x8 9/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 10/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 11/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 12/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 13/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-25T05:25:46.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-25T05:25:46.505 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-25T05:25:46.506 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-25T05:25:46.506 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:25:46.506 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-25T05:25:46.506 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/34 2024-04-25T05:25:46.538 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-25T05:25:46.575 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-25T05:25:46.601 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-25T05:25:46.623 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T05:25:46.649 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-25T05:25:46.850 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T05:25:46.904 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T05:25:47.286 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T05:25:47.287 INFO:teuthology.orchestra.run.smithi043.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:47.288 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:25:47.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-25T05:25:47.369 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:25:47.462 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:25:47.551 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 8/34 2024-04-25T05:25:47.579 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 9/34 2024-04-25T05:25:47.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-test 2024-04-25T05:25:47.625 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 9/34 2024-04-25T05:25:47.654 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-25T05:25:47.698 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9. 11/34 2024-04-25T05:25:47.793 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 12/34 2024-04-25T05:25:47.815 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 13/34 2024-04-25T05:25:47.879 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 13/34 2024-04-25T05:25:47.913 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-25T05:25:48.026 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-25T05:25:48.056 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-25T05:25:48.096 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-25T05:25:48.129 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-25T05:25:48.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-25T05:25:48.187 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:31 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:25:48.191 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-25T05:25:48.311 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 49 M 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 163 k 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T05:25:48.312 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:Install 4 Packages 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 50 M 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 211 M 2024-04-25T05:25:48.313 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:25:48.522 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-25T05:25:48.554 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 22/34 2024-04-25T05:25:48.629 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 22/34 2024-04-25T05:25:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 23/34 2024-04-25T05:25:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-25T05:25:48.790 INFO:teuthology.orchestra.run.smithi043.stdout:(1/4): libcephsqlite-19.0.0-3293.gd6f5f4e5.el9. 660 kB/s | 163 kB 00:00 2024-04-25T05:25:48.942 INFO:teuthology.orchestra.run.smithi043.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 763 kB/s | 305 kB 00:00 2024-04-25T05:25:48.971 INFO:teuthology.orchestra.run.smithi043.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 353 kB/s | 64 kB 00:00 2024-04-25T05:25:49.202 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-25T05:25:49.232 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 26/34 2024-04-25T05:25:49.304 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 26/34 2024-04-25T05:25:49.384 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 27/34 2024-04-25T05:25:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x8 28/34 2024-04-25T05:25:51.394 INFO:teuthology.orchestra.run.smithi043.stdout:(4/4): ceph-test-19.0.0-3293.gd6f5f4e5.el9.x86_ 17 MB/s | 49 MB 00:02 2024-04-25T05:25:51.396 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:25:51.396 INFO:teuthology.orchestra.run.smithi043.stdout:Total 16 MB/s | 50 MB 00:03 2024-04-25T05:25:51.424 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:25:51.447 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:25:51.448 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:25:51.787 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:51.831 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:51.839 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:25:51.839 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:25:52.405 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 29/34 2024-04-25T05:25:52.443 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 30/34 2024-04-25T05:25:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:25:52.536 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 30/34 2024-04-25T05:25:52.536 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T05:25:52.536 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T05:25:52.536 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:25:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-25T05:25:52.569 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:25:52.577 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-25T05:25:52.604 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/4 2024-04-25T05:25:57.487 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/4 2024-04-25T05:25:57.563 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/4 2024-04-25T05:25:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/4 2024-04-25T05:25:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/4 2024-04-25T05:25:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/4 2024-04-25T05:25:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-25T05:26:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-25T05:26:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:00.145 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T05:26:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T05:26:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:00.146 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:00.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph 2024-04-25T05:26:01.050 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:44 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:01.180 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 6.6 k 2024-04-25T05:26:01.182 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 2.4 M 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 1.5 M 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 249 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 4.7 M 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 17 M 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T05:26:01.183 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T05:26:01.184 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout:Install 36 Packages 2024-04-25T05:26:01.185 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:01.186 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 30 M 2024-04-25T05:26:01.186 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 108 M 2024-04-25T05:26:01.186 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:01.853 INFO:teuthology.orchestra.run.smithi043.stdout:(1/36): ceph-19.0.0-3293.gd6f5f4e5.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-04-25T05:26:02.029 INFO:teuthology.orchestra.run.smithi043.stdout:(2/36): ceph-mgr-19.0.0-3293.gd6f5f4e5.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-04-25T05:26:02.429 INFO:teuthology.orchestra.run.smithi043.stdout:(3/36): ceph-mon-19.0.0-3293.gd6f5f4e5.el9.x86_ 8.2 MB/s | 4.7 MB 00:00 2024-04-25T05:26:02.479 INFO:teuthology.orchestra.run.smithi043.stdout:(4/36): ceph-mds-19.0.0-3293.gd6f5f4e5.el9.x86_ 2.8 MB/s | 2.4 MB 00:00 2024-04-25T05:26:02.505 INFO:teuthology.orchestra.run.smithi043.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3293.gd6f5 3.2 MB/s | 249 kB 00:00 2024-04-25T05:26:02.530 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:26:02.530 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-04-25T05:26:02.530 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-04-25T05:26:02.531 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-04-25T05:26:02.531 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-04-25T05:26:02.531 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-04-25T05:26:02.531 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-04-25T05:26:02.531 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-04-25T05:26:02.532 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:02.630 INFO:teuthology.orchestra.run.smithi043.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-04-25T05:26:02.672 INFO:teuthology.orchestra.run.smithi043.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-25T05:26:02.714 INFO:teuthology.orchestra.run.smithi043.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-04-25T05:26:02.747 INFO:teuthology.orchestra.run.smithi043.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 5.1 MB/s | 1.2 MB 00:00 2024-04-25T05:26:02.773 INFO:teuthology.orchestra.run.smithi043.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-04-25T05:26:02.798 INFO:teuthology.orchestra.run.smithi043.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-04-25T05:26:03.082 INFO:teuthology.orchestra.run.smithi043.stdout:(12/36): ceph-osd-19.0.0-3293.gd6f5f4e5.el9.x86 16 MB/s | 17 MB 00:01 2024-04-25T05:26:03.106 INFO:teuthology.orchestra.run.smithi043.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 565 kB/s | 188 kB 00:00 2024-04-25T05:26:03.166 INFO:teuthology.orchestra.run.smithi043.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 468 kB/s | 172 kB 00:00 2024-04-25T05:26:03.191 INFO:teuthology.orchestra.run.smithi043.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-04-25T05:26:03.216 INFO:teuthology.orchestra.run.smithi043.stdout:(16/36): python3-markupsafe-1.1.1-12.el9.x86_64 259 kB/s | 35 kB 00:00 2024-04-25T05:26:03.241 INFO:teuthology.orchestra.run.smithi043.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-04-25T05:26:03.266 INFO:teuthology.orchestra.run.smithi043.stdout:(18/36): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-04-25T05:26:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/34 2024-04-25T05:26:03.550 INFO:teuthology.orchestra.run.smithi043.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-04-25T05:26:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/34 2024-04-25T05:26:03.562 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T05:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T05:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T05:26:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T05:26:03.651 INFO:teuthology.orchestra.run.smithi043.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 448 kB/s | 172 kB 00:00 2024-04-25T05:26:03.656 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-25T05:26:03.656 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:26:03.692 INFO:teuthology.orchestra.run.smithi043.stdout:(21/36): luarocks-3.9.2-1.el9.noarch.rpm 317 kB/s | 151 kB 00:00 2024-04-25T05:26:03.743 INFO:teuthology.orchestra.run.smithi043.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-04-25T05:26:03.785 INFO:teuthology.orchestra.run.smithi043.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-04-25T05:26:03.835 INFO:teuthology.orchestra.run.smithi043.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-04-25T05:26:03.860 INFO:teuthology.orchestra.run.smithi043.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-04-25T05:26:03.885 INFO:teuthology.orchestra.run.smithi043.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-04-25T05:26:03.919 INFO:teuthology.orchestra.run.smithi043.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-04-25T05:26:03.944 INFO:teuthology.orchestra.run.smithi043.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 551 kB/s | 46 kB 00:00 2024-04-25T05:26:03.969 INFO:teuthology.orchestra.run.smithi043.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-04-25T05:26:04.028 INFO:teuthology.orchestra.run.smithi043.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-25T05:26:04.053 INFO:teuthology.orchestra.run.smithi043.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-25T05:26:04.087 INFO:teuthology.orchestra.run.smithi043.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-25T05:26:04.112 INFO:teuthology.orchestra.run.smithi043.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-04-25T05:26:04.221 INFO:teuthology.orchestra.run.smithi043.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-25T05:26:04.246 INFO:teuthology.orchestra.run.smithi043.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-04-25T05:26:04.280 INFO:teuthology.orchestra.run.smithi043.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-04-25T05:26:04.281 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:04.282 INFO:teuthology.orchestra.run.smithi043.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-04-25T05:26:04.425 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:04.467 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:04.467 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:04.779 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:04.780 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:04.893 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 5/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_6 6/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 7/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9. 8/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x8 9/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 10/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 11/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 12/34 2024-04-25T05:26:04.894 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 13/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-25T05:26:04.895 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-25T05:26:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-25T05:26:04.896 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-25T05:26:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-25T05:26:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 31/34 2024-04-25T05:26:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-25T05:26:04.897 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/34 2024-04-25T05:26:05.618 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-25T05:26:05.618 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.619 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:05.621 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:05.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-04-25T05:26:06.164 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:06.224 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-25T05:26:06.257 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-25T05:26:06.285 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-25T05:26:06.362 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-25T05:26:06.410 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-25T05:26:06.443 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-25T05:26:06.469 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:29 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:06.540 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-25T05:26:06.600 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:06.600 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:06.600 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:26:06.600 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 49 M 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 163 k 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-04-25T05:26:06.601 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:06.602 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 50 M 2024-04-25T05:26:06.602 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 211 M 2024-04-25T05:26:06.602 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:06.733 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-25T05:26:06.780 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-25T05:26:06.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-25T05:26:06.843 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-25T05:26:06.878 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-25T05:26:06.911 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-25T05:26:06.940 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 305 kB 00:00 2024-04-25T05:26:06.952 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-25T05:26:06.996 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-25T05:26:06.999 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T05:26:07.029 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-25T05:26:07.049 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): libcephsqlite-19.0.0-3293.gd6f5f4e5.el9. 506 kB/s | 163 kB 00:00 2024-04-25T05:26:07.067 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-25T05:26:07.148 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-25T05:26:07.299 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-25T05:26:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-25T05:26:07.447 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-25T05:26:07.484 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-25T05:26:07.544 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-25T05:26:07.607 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-25T05:26:07.807 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-25T05:26:07.881 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-25T05:26:07.963 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-25T05:26:08.061 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-25T05:26:08.096 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-25T05:26:08.221 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-25T05:26:08.384 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9. 31/36 2024-04-25T05:26:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/36 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/36 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T05:26:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:09.103 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): ceph-test-19.0.0-3293.gd6f5f4e5.el9.x86_ 21 MB/s | 49 MB 00:02 2024-04-25T05:26:09.105 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:09.105 INFO:teuthology.orchestra.run.smithi060.stdout:Total 20 MB/s | 50 MB 00:02 2024-04-25T05:26:09.133 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:09.155 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:09.155 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:09.547 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:09.547 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/36 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/36 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T05:26:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:10.099 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:10.180 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-25T05:26:10.228 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-25T05:26:10.264 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/4 2024-04-25T05:26:10.458 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 34/36 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 34/36 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T05:26:10.482 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:10.717 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 35/36 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 35/36 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T05:26:10.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:10.806 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 36/36 2024-04-25T05:26:12.456 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 36/36 2024-04-25T05:26:12.456 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 5/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9. 6/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-25T05:26:12.457 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-25T05:26:12.458 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-25T05:26:12.459 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-25T05:26:13.374 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T05:26:13.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T05:26:13.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:13.377 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:13.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-base 2024-04-25T05:26:14.246 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:57 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:14.330 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:14.374 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:14.376 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:26:14.376 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:14.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install cephadm 2024-04-25T05:26:14.977 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:57 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:15.106 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:15.107 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:15.107 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:15.107 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:15.107 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:15.107 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 763 k 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 763 k 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 770 k 2024-04-25T05:26:15.108 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:15.133 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/4 2024-04-25T05:26:15.221 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/4 2024-04-25T05:26:15.412 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm-19.0.0-3293.gd6f5f4e5.el9.noarch.rpm 2.5 MB/s | 763 kB 00:00 2024-04-25T05:26:15.413 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:15.413 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.4 MB/s | 763 kB 00:00 2024-04-25T05:26:15.413 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:15.419 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:15.419 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:15.425 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:15.425 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:15.500 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:15.999 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:16.078 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:16.652 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:16.652 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:16.652 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:16.652 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:16.652 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:16.653 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:16.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-immutable-object-cache 2024-04-25T05:26:17.404 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:00 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:17.530 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 145 k 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:17.531 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 145 k 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 435 k 2024-04-25T05:26:17.532 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:17.551 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/4 2024-04-25T05:26:17.551 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/4 2024-04-25T05:26:17.551 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/4 2024-04-25T05:26:17.551 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-25T05:26:17.747 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-immutable-object-cache-19.0.0-3293.gd6f5f4 676 kB/s | 145 kB 00:00 2024-04-25T05:26:17.747 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:17.748 INFO:teuthology.orchestra.run.smithi043.stdout:Total 670 kB/s | 145 kB 00:00 2024-04-25T05:26:17.748 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:17.756 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:17.756 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:17.816 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:17.816 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:17.818 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:18.064 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:18.090 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:18.090 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T05:26:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:18.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-04-25T05:26:18.627 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:41 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:18.758 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 6.6 k 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 2.4 M 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 1.5 M 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 249 k 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 4.7 M 2024-04-25T05:26:18.761 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 17 M 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T05:26:18.762 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T05:26:18.763 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout:Install 36 Packages 2024-04-25T05:26:18.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:18.765 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 30 M 2024-04-25T05:26:18.765 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 108 M 2024-04-25T05:26:18.765 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:18.902 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:18.902 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:19.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr 2024-04-25T05:26:19.656 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:02 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:19.697 INFO:teuthology.orchestra.run.smithi060.stdout:(1/36): ceph-19.0.0-3293.gd6f5f4e5.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-04-25T05:26:19.739 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:19.784 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:19.785 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:26:19.785 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:19.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-dashboard 2024-04-25T05:26:19.897 INFO:teuthology.orchestra.run.smithi060.stdout:(2/36): ceph-mgr-19.0.0-3293.gd6f5f4e5.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-04-25T05:26:19.964 INFO:teuthology.orchestra.run.smithi060.stdout:(3/36): ceph-mds-19.0.0-3293.gd6f5f4e5.el9.x86_ 5.3 MB/s | 2.4 MB 00:00 2024-04-25T05:26:20.056 INFO:teuthology.orchestra.run.smithi060.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3293.gd6f5 2.7 MB/s | 249 kB 00:00 2024-04-25T05:26:20.131 INFO:teuthology.orchestra.run.smithi060.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-25T05:26:20.240 INFO:teuthology.orchestra.run.smithi060.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-04-25T05:26:20.307 INFO:teuthology.orchestra.run.smithi060.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T05:26:20.374 INFO:teuthology.orchestra.run.smithi060.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-25T05:26:20.380 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:03 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:20.441 INFO:teuthology.orchestra.run.smithi060.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-04-25T05:26:20.499 INFO:teuthology.orchestra.run.smithi060.stdout:(10/36): ceph-mon-19.0.0-3293.gd6f5f4e5.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-04-25T05:26:20.505 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:20.506 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:20.506 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 3.5 M 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 34 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 16 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T05:26:20.507 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout:Install 12 Packages 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:20.508 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 8.6 M 2024-04-25T05:26:20.509 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 104 M 2024-04-25T05:26:20.509 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:20.566 INFO:teuthology.orchestra.run.smithi060.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.7 MB/s | 215 kB 00:00 2024-04-25T05:26:20.591 INFO:teuthology.orchestra.run.smithi060.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-25T05:26:20.642 INFO:teuthology.orchestra.run.smithi060.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 696 kB/s | 35 kB 00:00 2024-04-25T05:26:20.667 INFO:teuthology.orchestra.run.smithi060.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-25T05:26:20.843 INFO:teuthology.orchestra.run.smithi060.stdout:(15/36): ceph-osd-19.0.0-3293.gd6f5f4e5.el9.x86 18 MB/s | 17 MB 00:00 2024-04-25T05:26:20.868 INFO:teuthology.orchestra.run.smithi060.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 226 kB/s | 51 kB 00:00 2024-04-25T05:26:20.893 INFO:teuthology.orchestra.run.smithi060.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 185 kB/s | 42 kB 00:00 2024-04-25T05:26:20.960 INFO:teuthology.orchestra.run.smithi060.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-25T05:26:21.244 INFO:teuthology.orchestra.run.smithi060.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-04-25T05:26:21.327 INFO:teuthology.orchestra.run.smithi060.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 329 kB/s | 151 kB 00:00 2024-04-25T05:26:21.386 INFO:teuthology.orchestra.run.smithi060.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-04-25T05:26:21.503 INFO:teuthology.orchestra.run.smithi060.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-04-25T05:26:21.509 INFO:teuthology.orchestra.run.smithi043.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3293.gd6 172 kB/s | 34 kB 00:00 2024-04-25T05:26:21.550 INFO:teuthology.orchestra.run.smithi043.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3293.gd6f 68 kB/s | 16 kB 00:00 2024-04-25T05:26:21.645 INFO:teuthology.orchestra.run.smithi060.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 68 kB/s | 18 kB 00:00 2024-04-25T05:26:21.701 INFO:teuthology.orchestra.run.smithi043.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-04-25T05:26:21.753 INFO:teuthology.orchestra.run.smithi060.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 704 kB/s | 358 kB 00:00 2024-04-25T05:26:21.768 INFO:teuthology.orchestra.run.smithi043.stdout:(4/12): ceph-mgr-dashboard-19.0.0-3293.gd6f5f4e 7.7 MB/s | 3.5 MB 00:00 2024-04-25T05:26:21.779 INFO:teuthology.orchestra.run.smithi060.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 84 kB/s | 23 kB 00:00 2024-04-25T05:26:21.801 INFO:teuthology.orchestra.run.smithi043.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.4 MB/s | 1.0 MB 00:00 2024-04-25T05:26:21.912 INFO:teuthology.orchestra.run.smithi060.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 73 kB/s | 19 kB 00:00 2024-04-25T05:26:22.012 INFO:teuthology.orchestra.run.smithi060.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-04-25T05:26:22.043 INFO:teuthology.orchestra.run.smithi043.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-04-25T05:26:22.069 INFO:teuthology.orchestra.run.smithi043.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-04-25T05:26:22.121 INFO:teuthology.orchestra.run.smithi060.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-04-25T05:26:22.279 INFO:teuthology.orchestra.run.smithi060.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 215 kB/s | 79 kB 00:00 2024-04-25T05:26:22.302 INFO:teuthology.orchestra.run.smithi043.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-04-25T05:26:22.388 INFO:teuthology.orchestra.run.smithi060.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-04-25T05:26:22.419 INFO:teuthology.orchestra.run.smithi043.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 263 kB/s | 31 kB 00:00 2024-04-25T05:26:22.470 INFO:teuthology.orchestra.run.smithi043.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 361 kB/s | 144 kB 00:00 2024-04-25T05:26:22.530 INFO:teuthology.orchestra.run.smithi060.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 526 kB/s | 272 kB 00:00 2024-04-25T05:26:22.561 INFO:teuthology.orchestra.run.smithi043.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-04-25T05:26:22.603 INFO:teuthology.orchestra.run.smithi043.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.6 MB/s | 2.0 MB 00:00 2024-04-25T05:26:22.605 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:22.605 INFO:teuthology.orchestra.run.smithi043.stdout:Total 4.1 MB/s | 8.6 MB 00:02 2024-04-25T05:26:22.655 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:22.655 INFO:teuthology.orchestra.run.smithi060.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-04-25T05:26:22.674 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:22.674 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:22.705 INFO:teuthology.orchestra.run.smithi060.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 211 kB/s | 90 kB 00:00 2024-04-25T05:26:22.839 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:22.842 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:22.972 INFO:teuthology.orchestra.run.smithi060.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-04-25T05:26:23.031 INFO:teuthology.orchestra.run.smithi060.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-04-25T05:26:23.239 INFO:teuthology.orchestra.run.smithi060.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 732 kB/s | 427 kB 00:00 2024-04-25T05:26:23.243 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:23.243 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.7 MB/s | 30 MB 00:04 2024-04-25T05:26:23.379 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:23.386 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:23.430 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:23.430 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:23.472 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-25T05:26:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-25T05:26:23.562 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-25T05:26:23.591 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-25T05:26:23.673 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-25T05:26:23.741 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:23.743 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:23.809 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-25T05:26:23.908 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-25T05:26:23.956 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-25T05:26:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-25T05:26:24.016 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9 10/12 2024-04-25T05:26:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-25T05:26:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-25T05:26:25.140 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el 11/12 2024-04-25T05:26:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-25T05:26:25.174 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 12/12 2024-04-25T05:26:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-25T05:26:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-25T05:26:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-25T05:26:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-25T05:26:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-25T05:26:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-25T05:26:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-25T05:26:25.611 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-25T05:26:25.646 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-25T05:26:25.679 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-25T05:26:25.721 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-25T05:26:25.764 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-25T05:26:25.798 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-25T05:26:25.837 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-25T05:26:25.914 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-25T05:26:26.076 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-25T05:26:26.165 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-25T05:26:26.266 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-25T05:26:26.303 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-25T05:26:26.361 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-25T05:26:26.427 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 12/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el 1/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 2/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9 3/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-25T05:26:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-25T05:26:26.632 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-25T05:26:26.682 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-25T05:26:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-25T05:26:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T05:26:26.869 INFO:teuthology.orchestra.run.smithi043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T05:26:26.870 INFO:teuthology.orchestra.run.smithi043.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T05:26:26.870 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T05:26:26.870 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T05:26:26.870 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:26.870 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-25T05:26:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-25T05:26:27.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-25T05:26:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9. 31/36 2024-04-25T05:26:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/36 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 32/36 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T05:26:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:27.656 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:10 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:27.784 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 7.4 M 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T05:26:27.785 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================= 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout:Install 12 Packages 2024-04-25T05:26:27.786 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:27.787 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 38 M 2024-04-25T05:26:27.787 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 209 M 2024-04-25T05:26:27.787 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:28.362 INFO:teuthology.orchestra.run.smithi043.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 899 kB/s | 195 kB 00:00 2024-04-25T05:26:28.453 INFO:teuthology.orchestra.run.smithi043.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-04-25T05:26:28.478 INFO:teuthology.orchestra.run.smithi043.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-25T05:26:28.562 INFO:teuthology.orchestra.run.smithi043.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-04-25T05:26:28.629 INFO:teuthology.orchestra.run.smithi043.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-04-25T05:26:28.688 INFO:teuthology.orchestra.run.smithi043.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-04-25T05:26:28.738 INFO:teuthology.orchestra.run.smithi043.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-25T05:26:28.776 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/36 2024-04-25T05:26:28.803 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 33/36 2024-04-25T05:26:28.803 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:28.803 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T05:26:28.804 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T05:26:28.804 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T05:26:28.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:28.888 INFO:teuthology.orchestra.run.smithi043.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-04-25T05:26:28.947 INFO:teuthology.orchestra.run.smithi043.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-04-25T05:26:29.030 INFO:teuthology.orchestra.run.smithi043.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 12 MB/s | 4.7 MB 00:00 2024-04-25T05:26:29.268 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 34/36 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 34/36 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T05:26:29.292 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:29.339 INFO:teuthology.orchestra.run.smithi043.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-04-25T05:26:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 35/36 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 35/36 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T05:26:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 36/36 2024-04-25T05:26:29.681 INFO:teuthology.orchestra.run.smithi043.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 4.8 MB/s | 7.4 MB 00:01 2024-04-25T05:26:29.682 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:29.682 INFO:teuthology.orchestra.run.smithi043.stdout:Total 20 MB/s | 38 MB 00:01 2024-04-25T05:26:30.040 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:30.068 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:30.068 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:30.502 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:30.503 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 36/36 2024-04-25T05:26:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 3/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 4/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 5/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9. 6/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-25T05:26:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-25T05:26:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-25T05:26:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-25T05:26:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-25T05:26:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-25T05:26:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-25T05:26:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-25T05:26:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-25T05:26:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-25T05:26:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-25T05:26:31.427 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:31.503 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-25T05:26:31.547 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-25T05:26:31.651 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-25T05:26:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-25T05:26:31.878 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-25T05:26:31.905 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-25T05:26:31.985 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:32.164 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T05:26:32.165 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T05:26:32.166 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:32.167 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:32.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-04-25T05:26:32.678 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-25T05:26:32.799 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-25T05:26:33.032 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:56 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:33.114 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:33.159 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:33.160 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:26:33.160 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:33.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-04-25T05:26:33.759 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:56 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:33.885 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:33.886 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:33.886 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:26:33.886 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:33.886 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 763 k 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 763 k 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 770 k 2024-04-25T05:26:33.887 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:34.115 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T05:26:34.186 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-19.0.0-3293.gd6f5f4e5.el9.noarch.rpm 2.5 MB/s | 763 kB 00:00 2024-04-25T05:26:34.186 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:34.187 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.5 MB/s | 763 kB 00:00 2024-04-25T05:26:34.187 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:34.192 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:34.192 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:34.198 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:34.198 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:34.271 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:34.702 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:34.782 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:35.197 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-25T05:26:35.221 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 12/12 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/1 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:35.337 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:35.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-04-25T05:26:36.069 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:59 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:36.194 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:36.195 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:36.195 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-04-25T05:26:36.195 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 145 k 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 145 k 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 435 k 2024-04-25T05:26:36.196 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:36.393 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-19.0.0-3293.gd6f5f4 738 kB/s | 145 kB 00:00 2024-04-25T05:26:36.393 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:36.394 INFO:teuthology.orchestra.run.smithi060.stdout:Total 731 kB/s | 145 kB 00:00 2024-04-25T05:26:36.394 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:36.402 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:36.402 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:36.462 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:36.463 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:36.620 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:36.702 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:36.728 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:36.728 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:36.728 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T05:26:36.728 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:37.559 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5. 1/1 2024-04-25T05:26:37.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:37.560 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:37.560 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:37.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:37.560 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:37.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-04-25T05:26:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 12/12 2024-04-25T05:26:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 1/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T05:26:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-25T05:26:38.219 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-25T05:26:38.219 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:38.219 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:38.219 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:38.220 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:38.303 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:01 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:38.387 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:38.432 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:38.433 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:26:38.433 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:38.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-04-25T05:26:38.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-rook 2024-04-25T05:26:39.027 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:02 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:39.119 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:22 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:39.153 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:39.154 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:39.154 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:26:39.154 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:39.154 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 3.5 M 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 34 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 16 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:39.155 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:26:39.156 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-04-25T05:26:39.156 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:39.156 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 8.6 M 2024-04-25T05:26:39.156 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 104 M 2024-04-25T05:26:39.156 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:39.244 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:39.245 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:39.245 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:39.245 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:39.245 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 49 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T05:26:39.246 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout:Install 13 Packages 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 2.2 M 2024-04-25T05:26:39.247 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 26 M 2024-04-25T05:26:39.248 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:39.440 INFO:teuthology.orchestra.run.smithi043.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 366 kB/s | 19 kB 00:00 2024-04-25T05:26:39.490 INFO:teuthology.orchestra.run.smithi043.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 250 kB/s | 26 kB 00:00 2024-04-25T05:26:39.557 INFO:teuthology.orchestra.run.smithi043.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-25T05:26:39.582 INFO:teuthology.orchestra.run.smithi043.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-25T05:26:39.632 INFO:teuthology.orchestra.run.smithi043.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.1 MB/s | 54 kB 00:00 2024-04-25T05:26:39.657 INFO:teuthology.orchestra.run.smithi043.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-04-25T05:26:39.783 INFO:teuthology.orchestra.run.smithi043.stdout:(7/13): ceph-mgr-rook-19.0.0-3293.gd6f5f4e5.el9 125 kB/s | 49 kB 00:00 2024-04-25T05:26:39.800 INFO:teuthology.orchestra.run.smithi043.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 105 kB/s | 15 kB 00:00 2024-04-25T05:26:39.816 INFO:teuthology.orchestra.run.smithi043.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-04-25T05:26:40.008 INFO:teuthology.orchestra.run.smithi043.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-04-25T05:26:40.034 INFO:teuthology.orchestra.run.smithi043.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 820 kB/s | 206 kB 00:00 2024-04-25T05:26:40.139 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3293.gd6f 44 kB/s | 16 kB 00:00 2024-04-25T05:26:40.192 INFO:teuthology.orchestra.run.smithi043.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-04-25T05:26:40.259 INFO:teuthology.orchestra.run.smithi043.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.2 MB/s | 1.0 MB 00:00 2024-04-25T05:26:40.260 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:40.260 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-04-25T05:26:40.328 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:40.344 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:40.344 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:40.364 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3293.gd6 57 kB/s | 34 kB 00:00 2024-04-25T05:26:40.423 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-04-25T05:26:40.436 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:40.437 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:40.539 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 7.2 MB/s | 863 kB 00:00 2024-04-25T05:26:40.665 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 896 kB/s | 269 kB 00:00 2024-04-25T05:26:40.732 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-04-25T05:26:40.757 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-04-25T05:26:40.816 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-04-25T05:26:40.858 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 741 kB/s | 31 kB 00:00 2024-04-25T05:26:40.916 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-04-25T05:26:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:41.108 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.4 MB/s | 2.0 MB 00:00 2024-04-25T05:26:41.124 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-25T05:26:41.179 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-25T05:26:41.244 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-25T05:26:41.287 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-25T05:26:41.329 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-25T05:26:41.427 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-25T05:26:41.541 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-25T05:26:41.600 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-25T05:26:42.363 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-25T05:26:42.412 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-25T05:26:42.446 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-25T05:26:42.495 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-25T05:26:42.524 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 13/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 13/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-25T05:26:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-25T05:26:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-25T05:26:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-25T05:26:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-25T05:26:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-25T05:26:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-25T05:26:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T05:26:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T05:26:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T05:26:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:43.619 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:43.749 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-dashboard-19.0.0-3293.gd6f5f4 909 kB/s | 3.5 MB 00:03 2024-04-25T05:26:43.752 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:43.752 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.9 MB/s | 8.6 MB 00:04 2024-04-25T05:26:43.805 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:43.825 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:43.825 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:43.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-cephadm 2024-04-25T05:26:43.991 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:43.992 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:44.387 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:27 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:44.512 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 143 k 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T05:26:44.513 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:44.514 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 7.0 M 2024-04-25T05:26:44.515 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 33 M 2024-04-25T05:26:44.515 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:44.550 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-25T05:26:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-25T05:26:44.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-25T05:26:44.754 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-25T05:26:44.836 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-25T05:26:44.946 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3293.gd6f5f4e5.e 597 kB/s | 143 kB 00:00 2024-04-25T05:26:44.970 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-25T05:26:44.971 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 936 kB/s | 248 kB 00:00 2024-04-25T05:26:45.078 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-25T05:26:45.127 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-25T05:26:45.155 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-25T05:26:45.188 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9 10/12 2024-04-25T05:26:45.330 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-04-25T05:26:45.405 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.5 MB/s | 6.0 MB 00:00 2024-04-25T05:26:45.588 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 854 kB/s | 548 kB 00:00 2024-04-25T05:26:45.672 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-04-25T05:26:45.673 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:45.673 INFO:teuthology.orchestra.run.smithi043.stdout:Total 6.0 MB/s | 7.0 MB 00:01 2024-04-25T05:26:45.747 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:45.759 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:45.759 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:45.863 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:45.863 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:46.191 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el 11/12 2024-04-25T05:26:46.337 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 12/12 2024-04-25T05:26:46.403 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-25T05:26:46.450 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-25T05:26:47.005 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-25T05:26:47.088 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-25T05:26:47.159 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-25T05:26:47.186 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 6/6 2024-04-25T05:26:47.668 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 12/12 2024-04-25T05:26:47.668 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el 1/12 2024-04-25T05:26:47.668 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noa 2/12 2024-04-25T05:26:47.668 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9 3/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-25T05:26:47.669 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 6/6 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/6 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-25T05:26:47.929 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T05:26:47.961 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T05:26:47.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T05:26:47.962 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T05:26:47.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:47.962 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T05:26:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:48.182 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:48.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-25T05:26:48.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-fuse 2024-04-25T05:26:48.735 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:11 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:26:48.863 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:26:48.864 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-04-25T05:26:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:26:48.864 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-04-25T05:26:48.864 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:26:48.864 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 7.4 M 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T05:26:48.865 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-04-25T05:26:48.866 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:48.867 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 38 M 2024-04-25T05:26:48.867 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2024-04-25T05:26:48.867 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:26:48.949 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:31 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:49.074 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 890 k 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:49.075 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 970 k 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 2.8 M 2024-04-25T05:26:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:49.374 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 448 kB/s | 80 kB 00:00 2024-04-25T05:26:49.467 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 820 kB/s | 195 kB 00:00 2024-04-25T05:26:49.526 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-25T05:26:49.618 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 33 kB 00:00 2024-04-25T05:26:49.668 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-04-25T05:26:49.691 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-fuse-19.0.0-3293.gd6f5f4e5.el9.x86_ 1.8 MB/s | 890 kB 00:00 2024-04-25T05:26:49.691 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:49.692 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.5 MB/s | 970 kB 00:00 2024-04-25T05:26:49.704 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:49.718 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-25T05:26:49.727 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:49.727 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:49.802 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-25T05:26:49.811 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:49.812 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:49.886 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-25T05:26:50.008 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:50.019 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-04-25T05:26:50.095 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-04-25T05:26:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-25T05:26:50.170 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-04-25T05:26:50.224 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/2 2024-04-25T05:26:50.654 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-25T05:26:50.845 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/2 2024-04-25T05:26:50.846 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/2 2024-04-25T05:26:51.037 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 4.1 MB/s | 7.4 MB 00:01 2024-04-25T05:26:51.038 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:51.038 INFO:teuthology.orchestra.run.smithi060.stdout:Total 18 MB/s | 38 MB 00:02 2024-04-25T05:26:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-25T05:26:51.083 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:51.084 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-25T05:26:51.084 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:51.084 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:51.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-volume 2024-04-25T05:26:51.398 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:26:51.425 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:26:51.425 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:26:51.860 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:26:51.861 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:26:51.865 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:34 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:51.991 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 273 k 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-04-25T05:26:51.992 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 350 k 2024-04-25T05:26:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 1.5 M 2024-04-25T05:26:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:52.220 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 990 kB/s | 77 kB 00:00 2024-04-25T05:26:52.395 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-volume-19.0.0-3293.gd6f5f4e5.el9.no 1.1 MB/s | 273 kB 00:00 2024-04-25T05:26:52.396 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:52.396 INFO:teuthology.orchestra.run.smithi043.stdout:Total 868 kB/s | 350 kB 00:00 2024-04-25T05:26:52.408 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:52.418 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:52.418 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:52.447 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:52.447 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:52.591 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:52.778 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-25T05:26:52.814 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2/2 2024-04-25T05:26:52.823 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:26:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2/2 2024-04-25T05:26:52.831 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:26:52.831 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T05:26:52.831 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:52.884 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-25T05:26:52.936 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-25T05:26:53.040 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-25T05:26:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-25T05:26:53.265 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-25T05:26:53.286 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-25T05:26:53.365 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-25T05:26:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/2 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:53.793 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:54.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-25T05:26:54.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados-devel 2024-04-25T05:26:54.180 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-25T05:26:54.666 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:37 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:54.790 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 127 k 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:26:54.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:54.792 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 127 k 2024-04-25T05:26:54.792 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 456 k 2024-04-25T05:26:54.792 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:55.018 INFO:teuthology.orchestra.run.smithi043.stdout:librados-devel-19.0.0-3293.gd6f5f4e5.el9.x86_64 561 kB/s | 127 kB 00:00 2024-04-25T05:26:55.018 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:55.019 INFO:teuthology.orchestra.run.smithi043.stdout:Total 558 kB/s | 127 kB 00:00 2024-04-25T05:26:55.019 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:55.025 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:55.025 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:55.091 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:55.091 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:55.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:55.498 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T05:26:55.738 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:55.978 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:55.978 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:55.978 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:55.978 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:55.979 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:55.979 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:56.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs2 2024-04-25T05:26:56.585 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-25T05:26:56.611 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 12/12 2024-04-25T05:26:56.720 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:39 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:56.801 INFO:teuthology.orchestra.run.smithi043.stdout:Package libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:56.844 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:56.845 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:26:56.845 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:56.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs-devel 2024-04-25T05:26:57.459 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:40 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:57.583 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:57.584 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:57.584 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 32 k 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 32 k 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 142 k 2024-04-25T05:26:57.585 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:26:57.779 INFO:teuthology.orchestra.run.smithi043.stdout:libcephfs-devel-19.0.0-3293.gd6f5f4e5.el9.x86_6 164 kB/s | 32 kB 00:00 2024-04-25T05:26:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:26:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:Total 162 kB/s | 32 kB 00:00 2024-04-25T05:26:57.780 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:26:57.784 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:26:57.784 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:26:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:26:57.786 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:26:57.875 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:26:57.978 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:26:58.554 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:58.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados2 2024-04-25T05:26:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 12/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f 1/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-25T05:26:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-25T05:26:59.285 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:42 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:26:59.368 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:26:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:26:59.412 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:26:59.413 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T05:26:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:26:59.490 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:26:59.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librbd1 2024-04-25T05:26:59.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-04-25T05:27:00.043 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:43 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:00.126 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:00.169 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:00.170 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:27:00.170 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:00.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rados 2024-04-25T05:27:00.386 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:23 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:00.512 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 49 k 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:27:00.513 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:00.514 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:00.515 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-04-25T05:27:00.515 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:00.515 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.2 M 2024-04-25T05:27:00.515 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-04-25T05:27:00.515 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:00.788 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:43 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:00.869 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:00.912 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:00.913 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:27:00.913 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:00.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rgw 2024-04-25T05:27:01.112 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 26 kB 00:00 2024-04-25T05:27:01.137 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 19 kB 00:00 2024-04-25T05:27:01.162 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): ceph-mgr-rook-19.0.0-3293.gd6f5f4e5.el9 266 kB/s | 49 kB 00:00 2024-04-25T05:27:01.213 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-04-25T05:27:01.238 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-25T05:27:01.263 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-25T05:27:01.347 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-04-25T05:27:01.524 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:44 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:01.530 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-04-25T05:27:01.556 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-25T05:27:01.605 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:01.648 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:27:01.649 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:01.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-cephfs 2024-04-25T05:27:01.864 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 398 kB/s | 206 kB 00:00 2024-04-25T05:27:01.898 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-04-25T05:27:02.215 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 256 kB/s | 90 kB 00:00 2024-04-25T05:27:02.248 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-04-25T05:27:02.249 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:02.250 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-04-25T05:27:02.265 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:45 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:02.320 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:02.336 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:02.337 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:02.347 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:02.390 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:02.391 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:27:02.391 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:02.432 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:02.432 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:02.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rbd 2024-04-25T05:27:02.954 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:03.014 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:46 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:03.094 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-25T05:27:03.138 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:03.139 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-25T05:27:03.139 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:03.195 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-25T05:27:03.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-fuse 2024-04-25T05:27:03.251 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-25T05:27:03.278 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-25T05:27:03.320 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-25T05:27:03.417 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-25T05:27:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-25T05:27:03.589 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-25T05:27:03.762 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:46 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:03.888 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:03.888 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 85 k 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 85 k 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 231 k 2024-04-25T05:27:03.889 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:27:04.115 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-fuse-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 378 kB/s | 85 kB 00:00 2024-04-25T05:27:04.116 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:04.116 INFO:teuthology.orchestra.run.smithi043.stdout:Total 376 kB/s | 85 kB 00:00 2024-04-25T05:27:04.116 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:27:04.124 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:27:04.124 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:27:04.191 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:27:04.191 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:27:04.342 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:27:04.351 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-25T05:27:04.402 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-25T05:27:04.436 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-25T05:27:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:04.484 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-25T05:27:04.555 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 13/13 2024-04-25T05:27:04.842 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:05.069 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:27:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:05.069 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:05.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-mirror 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 13/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-25T05:27:05.437 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-25T05:27:05.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-25T05:27:05.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-25T05:27:05.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-25T05:27:05.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-25T05:27:05.438 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T05:27:05.743 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T05:27:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T05:27:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T05:27:05.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:05.744 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:05.886 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:48 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:05.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-04-25T05:27:06.013 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:06.014 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.1 M 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.1 M 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 13 M 2024-04-25T05:27:06.015 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:27:06.513 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:29 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:06.577 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-mirror-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 5.5 MB/s | 3.1 MB 00:00 2024-04-25T05:27:06.578 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:06.579 INFO:teuthology.orchestra.run.smithi043.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-04-25T05:27:06.579 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:27:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:27:06.589 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:27:06.638 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:06.638 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:06.638 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 143 k 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:06.639 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:06.640 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-04-25T05:27:06.640 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:06.640 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 7.0 M 2024-04-25T05:27:06.640 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-04-25T05:27:06.640 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:06.674 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:27:06.674 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:27:07.020 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-04-25T05:27:07.143 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:27:07.203 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-25T05:27:07.211 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:07.229 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3293.gd6f5f4e5.e 320 kB/s | 143 kB 00:00 2024-04-25T05:27:07.238 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:07.238 INFO:teuthology.orchestra.run.smithi043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:27:07.238 INFO:teuthology.orchestra.run.smithi043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T05:27:07.238 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T05:27:07.238 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T05:27:07.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:07.270 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-04-25T05:27:07.321 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 493 kB/s | 58 kB 00:00 2024-04-25T05:27:07.363 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 382 kB/s | 51 kB 00:00 2024-04-25T05:27:07.363 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:07.363 INFO:teuthology.orchestra.run.smithi060.stdout:Total 9.6 MB/s | 7.0 MB 00:00 2024-04-25T05:27:07.436 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:07.447 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:07.448 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:07.551 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:07.552 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:07.860 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:08.138 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-25T05:27:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-25T05:27:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:08.345 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:27:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:08.345 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:08.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-nbd 2024-04-25T05:27:08.743 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-25T05:27:08.837 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-25T05:27:08.916 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-25T05:27:08.943 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 6/6 2024-04-25T05:27:09.120 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:01:52 ago on Thu 25 Apr 2024 05:25:17 AM UTC. 2024-04-25T05:27:09.245 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 170 k 2024-04-25T05:27:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 170 k 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 494 k 2024-04-25T05:27:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-25T05:27:09.472 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-nbd-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 757 kB/s | 170 kB 00:00 2024-04-25T05:27:09.473 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:09.473 INFO:teuthology.orchestra.run.smithi043.stdout:Total 752 kB/s | 170 kB 00:00 2024-04-25T05:27:09.473 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-25T05:27:09.481 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-25T05:27:09.482 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-25T05:27:09.549 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-25T05:27:09.549 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-25T05:27:09.670 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 6/6 2024-04-25T05:27:09.671 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/6 2024-04-25T05:27:09.671 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-25T05:27:09.671 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-25T05:27:09.671 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-25T05:27:09.671 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-25T05:27:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-25T05:27:09.821 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:09.926 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:09.927 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:10.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-04-25T05:27:10.208 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:27:10.460 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-25T05:27:10.676 DEBUG:teuthology.parallel:result is None 2024-04-25T05:27:10.680 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:33 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:10.805 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 890 k 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-04-25T05:27:10.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:10.807 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 970 k 2024-04-25T05:27:10.807 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.8 M 2024-04-25T05:27:10.807 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:11.273 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 426 kB/s | 80 kB 00:00 2024-04-25T05:27:11.431 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-19.0.0-3293.gd6f5f4e5.el9.x86_ 2.5 MB/s | 890 kB 00:00 2024-04-25T05:27:11.432 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:11.432 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.5 MB/s | 970 kB 00:00 2024-04-25T05:27:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:11.468 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:11.468 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:11.550 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:11.551 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:11.736 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:11.879 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-25T05:27:11.948 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/2 2024-04-25T05:27:12.607 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2/2 2024-04-25T05:27:12.607 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/2 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:12.854 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:13.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-04-25T05:27:13.627 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:36 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:13.753 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:13.753 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:13.753 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-04-25T05:27:13.753 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:13.753 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-3293.gd6f5f4e5.el9 ceph-noarch 273 k 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-04-25T05:27:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:13.755 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 350 k 2024-04-25T05:27:13.755 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.5 M 2024-04-25T05:27:13.755 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:13.895 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 1.0 MB/s | 77 kB 00:00 2024-04-25T05:27:14.103 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-volume-19.0.0-3293.gd6f5f4e5.el9.no 962 kB/s | 273 kB 00:00 2024-04-25T05:27:14.104 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:14.104 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.0 MB/s | 350 kB 00:00 2024-04-25T05:27:14.116 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:14.126 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:14.126 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:14.155 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:14.155 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:14.305 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:14.489 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-25T05:27:14.523 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2/2 2024-04-25T05:27:14.539 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2/2 2024-04-25T05:27:14.539 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:27:14.539 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T05:27:14.539 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:15.209 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 1/2 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-3293.gd6f5f4e5.el9.noarch 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:15.503 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:15.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-04-25T05:27:16.354 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:39 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:16.482 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 127 k 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:27:16.483 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:16.484 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2024-04-25T05:27:16.484 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2024-04-25T05:27:16.484 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:16.709 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-19.0.0-3293.gd6f5f4e5.el9.x86_64 563 kB/s | 127 kB 00:00 2024-04-25T05:27:16.710 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:16.710 INFO:teuthology.orchestra.run.smithi060.stdout:Total 560 kB/s | 127 kB 00:00 2024-04-25T05:27:16.710 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:16.716 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:16.716 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:16.783 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:16.783 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:16.937 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:17.033 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:17.459 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:17.689 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:17.757 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:17.757 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:17.757 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:17.757 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:17.758 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:17.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-04-25T05:27:18.477 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:41 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:18.576 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:18.633 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:18.634 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:18.634 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:18.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-04-25T05:27:19.268 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:42 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:19.393 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:19.393 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:19.393 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 32 k 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:27:19.394 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:19.395 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 32 k 2024-04-25T05:27:19.395 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 142 k 2024-04-25T05:27:19.395 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:19.589 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-19.0.0-3293.gd6f5f4e5.el9.x86_6 163 kB/s | 32 kB 00:00 2024-04-25T05:27:19.589 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:19.590 INFO:teuthology.orchestra.run.smithi060.stdout:Total 162 kB/s | 32 kB 00:00 2024-04-25T05:27:19.590 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:19.594 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:19.594 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:19.596 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:19.596 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:19.689 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:19.795 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:20.192 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:20.396 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:20.396 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:20.396 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:20.396 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:20.397 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:20.397 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:20.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-04-25T05:27:21.095 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:44 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:21.177 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:21.221 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:21.222 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:21.222 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:21.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-04-25T05:27:21.816 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:44 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:21.897 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:21.941 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:21.942 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:21.942 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:22.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-04-25T05:27:22.534 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:45 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:22.615 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:22.658 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:22.659 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:22.659 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:22.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-04-25T05:27:23.249 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:46 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:23.329 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:23.372 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:23.373 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:23.373 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:23.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-04-25T05:27:23.973 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:46 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:24.054 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:24.097 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:24.098 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:24.098 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:24.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-04-25T05:27:24.703 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:47 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:24.784 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 is already installed. 2024-04-25T05:27:24.828 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:24.829 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-04-25T05:27:24.829 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:24.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-04-25T05:27:25.432 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:48 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:25.558 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:25.559 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-25T05:27:25.559 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:25.559 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 85 k 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 85 k 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 231 k 2024-04-25T05:27:25.560 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:25.800 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 356 kB/s | 85 kB 00:00 2024-04-25T05:27:25.801 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:25.801 INFO:teuthology.orchestra.run.smithi060.stdout:Total 354 kB/s | 85 kB 00:00 2024-04-25T05:27:25.801 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:25.809 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:25.809 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:25.876 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:25.876 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:26.031 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:26.132 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:26.485 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:26.710 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:26.710 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:26.710 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:26.711 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:26.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:26.711 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:26.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-04-25T05:27:27.467 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:50 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:27.591 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 3.1 M 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:27.592 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.1 M 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 13 M 2024-04-25T05:27:27.593 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:28.171 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 5.3 MB/s | 3.1 MB 00:00 2024-04-25T05:27:28.171 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:28.172 INFO:teuthology.orchestra.run.smithi060.stdout:Total 5.3 MB/s | 3.1 MB 00:00 2024-04-25T05:27:28.172 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:28.182 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:28.182 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:28.267 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:28.268 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:28.722 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:28.788 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:28.814 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:28.814 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T05:27:28.814 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T05:27:28.814 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T05:27:28.815 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T05:27:28.815 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:29.872 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:29.872 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:29.873 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:29.873 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:29.873 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:29.873 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:30.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-04-25T05:27:30.627 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:53 ago on Thu 25 Apr 2024 05:25:37 AM UTC. 2024-04-25T05:27:30.754 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-04-25T05:27:30.754 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-3293.gd6f5f4e5.el9 ceph 170 k 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-04-25T05:27:30.755 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-04-25T05:27:30.756 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:30.756 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 170 k 2024-04-25T05:27:30.756 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 494 k 2024-04-25T05:27:30.756 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-04-25T05:27:30.975 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-19.0.0-3293.gd6f5f4e5.el9.x86_64.rpm 777 kB/s | 170 kB 00:00 2024-04-25T05:27:30.976 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:27:30.976 INFO:teuthology.orchestra.run.smithi060.stdout:Total 772 kB/s | 170 kB 00:00 2024-04-25T05:27:30.976 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-04-25T05:27:30.984 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-04-25T05:27:30.984 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-04-25T05:27:31.051 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-04-25T05:27:31.051 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-04-25T05:27:31.206 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-04-25T05:27:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:31.687 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 1/1 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-3293.gd6f5f4e5.el9.x86_64 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:27:31.920 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-04-25T05:27:32.163 DEBUG:teuthology.parallel:result is None 2024-04-25T05:27:32.163 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:27:32.304 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T05:27:32.329 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-3293.gd6f5f4e5.el9 2024-04-25T05:27:32.329 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3293.gd6f5f4e5.el9 2024-04-25T05:27:32.330 INFO:teuthology.task.install:The correct ceph version 19.0.0-3293.gd6f5f4e5 is installed. 2024-04-25T05:27:32.331 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:27:32.466 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T05:27:32.491 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-3293.gd6f5f4e5.el9 2024-04-25T05:27:32.492 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3293.gd6f5f4e5.el9 2024-04-25T05:27:32.492 INFO:teuthology.task.install:The correct ceph version 19.0.0-3293.gd6f5f4e5 is installed. 2024-04-25T05:27:32.494 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T05:27:32.494 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:27:32.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T05:27:32.524 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:27:32.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T05:27:32.558 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T05:27:32.558 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:27:32.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T05:27:32.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T05:27:32.659 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:27:32.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T05:27:32.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T05:27:32.749 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T05:27:32.750 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:27:32.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T05:27:32.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T05:27:32.842 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:27:32.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T05:27:32.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T05:27:32.932 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T05:27:32.933 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:27:32.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T05:27:32.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T05:27:33.023 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:27:33.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T05:27:33.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T05:27:33.112 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T05:27:33.213 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737', 'cephadm_mode': 'root'} 2024-04-25T05:27:33.214 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:27:33.214 INFO:tasks.cephadm:Cluster fsid is 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:27:33.214 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T05:27:33.214 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.43', 'mon.c': '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]', 'mon.b': '172.21.15.60'} 2024-04-25T05:27:33.215 INFO:tasks.cephadm:First mon is mon.a on smithi043 2024-04-25T05:27:33.215 INFO:tasks.cephadm:First mgr is y 2024-04-25T05:27:33.215 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T05:27:33.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2024-04-25T05:27:33.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo hostname $(hostname -s) 2024-04-25T05:27:33.264 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T05:27:33.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:27:33.412 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78864/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3293-gd6f5f4e5', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3293.gd6f5f4e5'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 23:41:59.690185', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-04-24-1625', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T05:27:33.546 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-adk-testing-2024-04-24-1625, sha1 d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:27:33.548 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/x86_64/flavors/default/cephadm 2024-04-25T05:27:33.548 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.050 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-r--r--. 1 ubuntu ubuntu 782568 Apr 25 05:27 /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.050 DEBUG:teuthology.orchestra.run.smithi060:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-04-24-1625/d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.509 INFO:teuthology.orchestra.run.smithi060.stdout:-rw-r--r--. 1 ubuntu ubuntu 782568 Apr 25 05:27 /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.509 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.529 DEBUG:teuthology.orchestra.run.smithi060:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T05:27:34.560 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 on all hosts... 2024-04-25T05:27:34.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 pull 2024-04-25T05:27:34.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 pull 2024-04-25T05:27:36.056 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737... 2024-04-25T05:27:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737... 2024-04-25T05:28:12.421 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:28:12.421 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev)", 2024-04-25T05:28:12.421 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1", 2024-04-25T05:28:12.421 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2024-04-25T05:28:12.421 INFO:teuthology.orchestra.run.smithi043.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72" 2024-04-25T05:28:12.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:28:12.422 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:28:13.045 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:28:13.045 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev)", 2024-04-25T05:28:13.045 INFO:teuthology.orchestra.run.smithi060.stdout: "image_id": "3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1", 2024-04-25T05:28:13.045 INFO:teuthology.orchestra.run.smithi060.stdout: "repo_digests": [ 2024-04-25T05:28:13.045 INFO:teuthology.orchestra.run.smithi060.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72" 2024-04-25T05:28:13.046 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:28:13.046 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:28:13.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2024-04-25T05:28:13.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph 2024-04-25T05:28:13.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2024-04-25T05:28:13.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /etc/ceph 2024-04-25T05:28:13.191 INFO:tasks.cephadm:Writing seed config... 2024-04-25T05:28:13.192 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T05:28:13.192 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T05:28:13.192 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T05:28:13.193 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T05:28:13.194 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-25T05:28:13.195 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:28:13.195 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T05:28:13.221 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 84884b5e-02c4-11ef-bc93-c7b262605968 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-25T05:28:13.222 DEBUG:teuthology.orchestra.run.smithi043:mon.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T05:28:13.264 DEBUG:teuthology.orchestra.run.smithi043:mgr.y> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T05:28:13.306 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T05:28:13.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 -v bootstrap --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.43 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:28:13.502 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-25T05:28:13.502 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737', '-v', 'bootstrap', '--fsid', '84884b5e-02c4-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.43', '--skip-admin-label'] 2024-04-25T05:28:13.503 INFO:teuthology.orchestra.run.smithi043.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T05:28:13.503 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying podman|docker is present... 2024-04-25T05:28:13.529 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T05:28:13.529 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying lvm2 is present... 2024-04-25T05:28:13.530 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying time synchronization is in place... 2024-04-25T05:28:13.537 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T05:28:13.537 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T05:28:13.543 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T05:28:13.544 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:28:13.550 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-04-25T05:28:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-04-25T05:28:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-04-25T05:28:13.556 INFO:teuthology.orchestra.run.smithi043.stdout:Repeating the final host check... 2024-04-25T05:28:13.590 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T05:28:13.591 INFO:teuthology.orchestra.run.smithi043.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T05:28:13.591 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl is present 2024-04-25T05:28:13.591 INFO:teuthology.orchestra.run.smithi043.stdout:lvcreate is present 2024-04-25T05:28:13.597 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T05:28:13.597 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T05:28:13.603 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T05:28:13.603 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:28:13.610 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout enabled 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout active 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:Unit chronyd.service is enabled and running 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:Host looks OK 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:Cluster fsid: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:Acquiring lock 140357844141008 on /run/cephadm/84884b5e-02c4-11ef-bc93-c7b262605968.lock 2024-04-25T05:28:13.616 INFO:teuthology.orchestra.run.smithi043.stdout:Lock 140357844141008 acquired on /run/cephadm/84884b5e-02c4-11ef-bc93-c7b262605968.lock 2024-04-25T05:28:13.617 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 3300 ... 2024-04-25T05:28:13.617 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying IP 172.21.15.43 port 6789 ... 2024-04-25T05:28:13.617 INFO:teuthology.orchestra.run.smithi043.stdout:Base mon IP(s) is [172.21.15.43:3300, 172.21.15.43:6789], mon addrv is [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-04-25T05:28:13.620 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.43 metric 100 2024-04-25T05:28:13.620 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.43 metric 100 2024-04-25T05:28:13.623 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T05:28:13.623 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T05:28:13.623 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T05:28:13.625 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T05:28:13.625 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-04-25T05:28:13.626 INFO:teuthology.orchestra.run.smithi043.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T05:28:13.627 INFO:teuthology.orchestra.run.smithi043.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T05:28:13.627 INFO:teuthology.orchestra.run.smithi043.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737... 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stdout 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737... 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:63ee30ee8c4ee93f6b60ee35f3ba499e6c8848e2dae5ce94a9ad4206c3df0116 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Copying config sha256:3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 2024-04-25T05:28:14.107 INFO:teuthology.orchestra.run.smithi043.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T05:28:15.822 INFO:teuthology.orchestra.run.smithi043.stdout:ceph: stdout ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev) 2024-04-25T05:28:15.823 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph version: ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev) 2024-04-25T05:28:15.823 INFO:teuthology.orchestra.run.smithi043.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T05:28:25.269 INFO:teuthology.orchestra.run.smithi043.stdout:stat: stdout 167 167 2024-04-25T05:28:25.269 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial keys... 2024-04-25T05:28:27.869 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQD76SlmD3HiCRAAy9qTUqi8ip95xUrJZPRAmQ== 2024-04-25T05:28:28.958 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQD86Slm4+gtDhAA/7Rfiu1a9X+Ni9BujoKRfw== 2024-04-25T05:28:29.954 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph-authtool: stdout AQD96SlmodiFExAAr13r0hGFOVfudGKi1Bzp2Q== 2024-04-25T05:28:29.955 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial monmap... 2024-04-25T05:28:30.977 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T05:28:30.977 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool for a [v2:172.21.15.43:3300,v1:172.21.15.43:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: set fsid to 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:28:30.978 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mon... 2024-04-25T05:28:31.949 INFO:teuthology.orchestra.run.smithi043.stdout:create mon.a on 2024-04-25T05:28:32.238 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T05:28:32.426 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T05:28:32.865 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target. 2024-04-25T05:28:32.865 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target. 2024-04-25T05:28:33.465 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a 2024-04-25T05:28:33.466 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T05:28:33.639 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target.wants/ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968@.service. 2024-04-25T05:28:33.912 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:33 smithi043 podman[26606]: 2024-04-25 05:28:33.790377299 +0000 UTC m=+0.020112429 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:28:34.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:33 smithi043 podman[26606]: 2024-04-25 05:28:33.913032398 +0000 UTC m=+0.142767566 container create 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:28:34.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 podman[26606]: 2024-04-25 05:28:34.1489872 +0000 UTC m=+0.378722320 container init 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:28:34.181 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 podman[26606]: 2024-04-25 05:28:34.152476214 +0000 UTC m=+0.382211343 container start 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:28:34.254 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T05:28:34.255 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-04-25T05:28:34.262 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T05:28:34.262 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:28:34.262 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-04-25T05:28:34.262 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T05:28:34.263 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon to start... 2024-04-25T05:28:34.263 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mon... 2024-04-25T05:28:34.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 ceph-mon[26635]: mkfs 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:34.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 ceph-mon[26635]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T05:28:34.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 bash[26606]: 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 2024-04-25T05:28:34.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:34 smithi043 systemd[1]: Started Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: monmap epoch 1 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: last_changed 2024-04-25T05:28:30.314004+0000 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: min_mon_release 19 (squid) 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: election_strategy: 1 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: fsmap 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: mgrmap e1: no daemons active 2024-04-25T05:28:35.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:35 smithi043 ceph-mon[26635]: from='client.? 172.21.15.43:0/2143733591' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T05:28:35.383 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout id: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout services: 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.627184s) 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout data: 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T05:28:35.385 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T05:28:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T05:28:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:mon is available 2024-04-25T05:28:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T05:28:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:36 smithi043 ceph-mon[26635]: from='client.? 172.21.15.43:0/2517941274' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T05:28:36.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:36 smithi043 ceph-mon[26635]: from='client.? 172.21.15.43:0/2517941274' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T05:28:36.657 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T05:28:36.658 INFO:teuthology.orchestra.run.smithi043.stdout:Generating new minimal ceph.conf... 2024-04-25T05:28:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:37 smithi043 ceph-mon[26635]: from='client.? 172.21.15.43:0/3261817024' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:28:37.761 INFO:teuthology.orchestra.run.smithi043.stdout:Restarting the monitor... 2024-04-25T05:28:38.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:37 smithi043 systemd[1]: Stopping Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:28:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[26629]: 2024-04-25T05:28:38.054+0000 7ff4e825c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:28:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[26629]: 2024-04-25T05:28:38.054+0000 7ff4e825c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T05:28:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 podman[26882]: 2024-04-25 05:28:38.167288553 +0000 UTC m=+0.236797767 container died 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T05:28:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 podman[26882]: 2024-04-25 05:28:38.536721247 +0000 UTC m=+0.606230462 container cleanup 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T05:28:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 bash[26882]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a 2024-04-25T05:28:39.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:38 smithi043 podman[26896]: 2024-04-25 05:28:38.962816376 +0000 UTC m=+0.799585521 container remove 88d81273c279996b9b291af8358474e36881f08eb5f4048f8f37b1f226e00eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T05:28:39.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T05:28:39.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 systemd[1]: Stopped Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:28:39.333 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 systemd[1]: Starting Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:28:39.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 podman[26990]: 2024-04-25 05:28:39.236417635 +0000 UTC m=+0.018298494 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:28:39.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 podman[26990]: 2024-04-25 05:28:39.347002764 +0000 UTC m=+0.128883644 container create 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T05:28:39.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 podman[26990]: 2024-04-25 05:28:39.549258368 +0000 UTC m=+0.331139202 container init 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T05:28:39.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 podman[26990]: 2024-04-25 05:28:39.55223481 +0000 UTC m=+0.334115650 container start 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev), process ceph-mon, pid 2 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: pidfile_write: ignore empty --pid-file 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: load: jerasure load: lrc 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: RocksDB version: 7.9.2 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Git sha 0 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Compile date 2024-04-25 01:05:22 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: DB SUMMARY 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: DB Session ID: HWXTHFO8SUIFYM17HCF6 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: CURRENT file: CURRENT 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T05:28:39.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95629 ; 2024-04-25T05:28:39.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.error_if_exists: 0 2024-04-25T05:28:39.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.create_if_missing: 0 2024-04-25T05:28:39.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.paranoid_checks: 1 2024-04-25T05:28:39.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T05:28:39.586 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.env: 0x563782c49cc0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.info_log: 0x5637849bdd80 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.statistics: (nil) 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.use_fsync: 0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_log_file_size: 0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T05:28:39.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_fallocate: 1 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.use_direct_reads: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.db_log_dir: 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.wal_dir: 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T05:28:39.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.write_buffer_manager: 0x56378498f0e0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.unordered_write: 0 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T05:28:39.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.row_cache: None 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.wal_filter: None 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.two_write_queues: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.wal_compression: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.atomic_flush: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.log_readahead_size: 0 2024-04-25T05:28:39.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_background_jobs: 2 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_background_compactions: -1 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_subcompactions: 1 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T05:28:39.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_open_files: -1 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_background_flushes: -1 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Compression algorithms supported: 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kZSTD supported: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kXpressCompression supported: 0 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kZlibCompression supported: 1 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kSnappyCompression supported: 1 2024-04-25T05:28:39.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kLZ4Compression supported: 1 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: kBZip2Compression supported: 0 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.merge_operator: 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_filter: None 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5637849bdf40) 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-04-25T05:28:39.593 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_type: 0 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_index_type: 0 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_shortening: 1 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: checksum: 4 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: no_block_cache: 0 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache: 0x5637849c11f0 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_options: 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: capacity : 536870912 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_shard_bits : 4 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: strict_capacity_limit : 0 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_compressed: (nil) 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: persistent_cache: (nil) 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size: 4096 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size_deviation: 10 2024-04-25T05:28:39.594 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_restart_interval: 16 2024-04-25T05:28:39.595 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_block_restart_interval: 1 2024-04-25T05:28:39.595 INFO:journalctl@ceph.mon.a.smithi043.stdout: metadata_block_size: 4096 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: partition_filters: 0 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: use_delta_encoding: 1 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: filter_policy: bloomfilter 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: whole_key_filtering: 1 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: verify_compression: 0 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: format_version: 5 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_index_compression: 1 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_align: 0 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: max_auto_readahead_size: 262144 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: prepopulate_block_cache: 0 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: initial_auto_readahead_size: 8192 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T05:28:39.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression: NoCompression 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.num_levels: 7 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T05:28:39.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T05:28:39.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T05:28:39.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.inplace_update_support: 0 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T05:28:39.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.bloom_locality: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.max_successive_merges: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.ttl: 2592000 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T05:28:39.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enable_blob_files: false 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.min_blob_size: 0 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T05:28:39.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5283a9c9-7418-4383-9911-d596874475d3 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714022919582028, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714022919583489, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91582, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 89630, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11839, "raw_average_key_size": 48, "raw_value_size": 83102, "raw_average_value_size": 339, "num_data_blocks": 12, "num_entries": 245, "num_filter_entries": 245, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714022919, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5283a9c9-7418-4383-9911-d596874475d3", "db_session_id": "HWXTHFO8SUIFYM17HCF6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714022919583612, "job": 1, "event": "recovery_finished"} 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563784ade000 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: DB pointer 0x563784aca000 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** DB Stats ** 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T05:28:39.603 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 91.30 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 91.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.604 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 19.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 19.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache BinnedLRUCache@0x5637849c11f0#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T05:28:39.605 INFO:journalctl@ceph.mon.a.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: starting mon.a rank 0 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???) e1 preinit fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).mds e1 new map 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).mds e1 print_map 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: e1 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T05:28:39.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: legacy client fscid: -1 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: No filesystems configured 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:28:39.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T05:28:39.645 INFO:teuthology.orchestra.run.smithi043.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T05:28:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 bash[26990]: 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e 2024-04-25T05:28:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 systemd[1]: Started Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:28:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T05:28:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: monmap epoch 1 2024-04-25T05:28:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: last_changed 2024-04-25T05:28:30.314004+0000 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: min_mon_release 19 (squid) 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: election_strategy: 1 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: fsmap 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T05:28:39.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:39 smithi043 ceph-mon[27010]: mgrmap e1: no daemons active 2024-04-25T05:28:40.825 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T05:28:40.846 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:28:40.847 INFO:teuthology.orchestra.run.smithi043.stdout:Creating mgr... 2024-04-25T05:28:40.847 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T05:28:40.847 INFO:teuthology.orchestra.run.smithi043.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T05:28:41.095 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y 2024-04-25T05:28:41.095 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Failed to reset failed state of unit ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-25T05:28:41.256 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968.target.wants/ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-84884b5e-02c4-11ef-bc93-c7b262605968@.service. 2024-04-25T05:28:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3830321432' entity='client.admin' 2024-04-25T05:28:41.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 systemd[1]: Starting Ceph mgr.y for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:28:41.760 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 podman[27223]: 2024-04-25 05:28:41.491075087 +0000 UTC m=+0.109227639 container create 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T05:28:41.760 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 podman[27223]: 2024-04-25 05:28:41.399705552 +0000 UTC m=+0.017858063 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 2024-04-25T05:28:41.760 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 podman[27223]: 2024-04-25 05:28:41.693460643 +0000 UTC m=+0.311613151 container init 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:28:41.760 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 podman[27223]: 2024-04-25 05:28:41.695939522 +0000 UTC m=+0.314092038 container start 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T05:28:41.760 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 bash[27223]: 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 2024-04-25T05:28:41.774 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T05:28:41.774 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-04-25T05:28:41.782 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T05:28:41.782 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:28:41.782 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-04-25T05:28:41.783 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T05:28:41.791 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T05:28:41.791 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr to start... 2024-04-25T05:28:41.799 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr... 2024-04-25T05:28:42.055 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 systemd[1]: Started Ceph mgr.y for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:28:42.055 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:41 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:41.859+0000 7f3e7ce32200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:28:42.055 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:42.054+0000 7f3e7ce32200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:28:42.355 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:42.163+0000 7f3e7ce32200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:28:42.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4031414455' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T05:28:42.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:42.495+0000 7f3e7ce32200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:28:42.642 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:42.582+0000 7f3e7ce32200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:28:43.231 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:43.231 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T05:28:43.232 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T05:28:43.233 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T05:28:43.234 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T05:28:43.238 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T05:28:43.239 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T05:28:34.192998+0000", 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:28:43.240 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (1/15)... 2024-04-25T05:28:44.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:43 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:43.860+0000 7f3e7ce32200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:28:44.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:44.033+0000 7f3e7ce32200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:28:44.505 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:44.209+0000 7f3e7ce32200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:28:44.506 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:44.505+0000 7f3e7ce32200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:28:44.810 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:44.599+0000 7f3e7ce32200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:28:44.810 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:44.809+0000 7f3e7ce32200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:28:45.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:45 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:45.007+0000 7f3e7ce32200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:28:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3772794767' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T05:28:45.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:45 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:45.638+0000 7f3e7ce32200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:28:46.370 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:46 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:46.077+0000 7f3e7ce32200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:28:46.370 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:46 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:46.178+0000 7f3e7ce32200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:28:46.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:46.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:28:46.515 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:28:46.516 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T05:28:46.516 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T05:28:46.516 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T05:28:46.516 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T05:28:46.516 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T05:28:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T05:28:46.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:46.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T05:28:46.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T05:28:46.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:46.518 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T05:28:46.519 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.520 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T05:28:46.521 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T05:28:46.522 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T05:28:46.522 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T05:28:46.522 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.522 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T05:28:46.525 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T05:28:46.526 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:46.527 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T05:28:34.192998+0000", 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T05:28:46.528 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:28:46.529 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (2/15)... 2024-04-25T05:28:46.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:46 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:46.386+0000 7f3e7ce32200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:28:47.278 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:46 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:46.932+0000 7f3e7ce32200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:28:47.279 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.026+0000 7f3e7ce32200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:28:47.631 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.278+0000 7f3e7ce32200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:28:47.631 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.414+0000 7f3e7ce32200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:28:47.631 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.512+0000 7f3e7ce32200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:28:47.631 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.630+0000 7f3e7ce32200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:28:48.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:47.942+0000 7f3e7ce32200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:28:48.825 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:48.575+0000 7f3e7ce32200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:28:48.826 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:48.798+0000 7f3e7ce32200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:28:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: Activating manager daemon y 2024-04-25T05:28:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: mgrmap e2: y(active, starting, since 0.00310503s) 2024-04-25T05:28:49.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:28:49.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:28:49.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:28:49.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:28:49.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: Manager daemon y is now available 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' 2024-04-25T05:28:49.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:48 smithi043 ceph-mon[27010]: from='mgr.14100 172.21.15.43:0/3918974038' entity='mgr.y' 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:49.744 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T05:28:49.745 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T05:28:49.749 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T05:28:49.750 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T05:28:34.192998+0000", 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:28:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:mgr not available, waiting (3/15)... 2024-04-25T05:28:49.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1081187408' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T05:28:49.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:49 smithi043 ceph-mon[27010]: mgrmap e3: y(active, since 1.00791s) 2024-04-25T05:28:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:51 smithi043 ceph-mon[27010]: mgrmap e4: y(active, since 2s) 2024-04-25T05:28:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/409583854' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T05:28:53.164 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:53.164 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 0 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T05:28:53.165 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T05:28:53.166 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T05:28:53.169 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T05:28:53.169 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T05:28:53.169 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.169 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T05:28:53.169 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T05:28:53.170 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ], 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T05:28:34.192998+0000", 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout }, 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:28:53.171 INFO:teuthology.orchestra.run.smithi043.stdout:mgr is available 2024-04-25T05:28:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1320175309' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout fsid = 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:54.404 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T05:28:54.405 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling cephadm module... 2024-04-25T05:28:55.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/912632506' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T05:28:55.514 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setuser ceph since I am not root 2024-04-25T05:28:55.514 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setgroup ceph since I am not root 2024-04-25T05:28:55.514 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:55.364+0000 7f8bd276d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:28:55.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:55.559+0000 7f8bd276d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:28:55.814 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:55.665+0000 7f8bd276d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:28:56.087 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:55 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:55.995+0000 7f8bd276d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:28:56.087 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:56 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:56.087+0000 7f8bd276d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:28:56.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/912632506' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T05:28:56.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:56 smithi043 ceph-mon[27010]: mgrmap e5: y(active, since 6s) 2024-04-25T05:28:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:28:57 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2431776668' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T05:28:57.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:57 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:57.354+0000 7f8bd276d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:28:57.491 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-04-25T05:28:57.492 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 5... 2024-04-25T05:28:57.763 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:57 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:57.537+0000 7f8bd276d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:28:57.763 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:57 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:57.715+0000 7f8bd276d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:28:58.328 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:58 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:58.021+0000 7f8bd276d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:28:58.328 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:58 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:58.117+0000 7f8bd276d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:28:58.328 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:58 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:58.328+0000 7f8bd276d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:28:58.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:58 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:58.528+0000 7f8bd276d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:28:59.599 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:59.160+0000 7f8bd276d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:28:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:59.598+0000 7f8bd276d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:28:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:59.698+0000 7f8bd276d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:29:00.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:28:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:28:59.907+0000 7f8bd276d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:29:00.800 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:00 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:00.453+0000 7f8bd276d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:29:00.800 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:00 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:00.547+0000 7f8bd276d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:29:01.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:00 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:00.800+0000 7f8bd276d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:29:01.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:00 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:00.938+0000 7f8bd276d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:29:01.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:01 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:01.037+0000 7f8bd276d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:29:01.471 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:01 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:01.157+0000 7f8bd276d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:29:01.471 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:01 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:01.471+0000 7f8bd276d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:29:02.379 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:02.102+0000 7f8bd276d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:29:02.380 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:02.326+0000 7f8bd276d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:29:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: Active manager daemon y restarted 2024-04-25T05:29:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: Activating manager daemon y 2024-04-25T05:29:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T05:29:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: mgrmap e6: y(active, starting, since 0.00344899s) 2024-04-25T05:29:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: Manager daemon y is now available 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:02 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:03 smithi043 ceph-mon[27010]: Found migration_current of "None". Setting to last migration. 2024-04-25T05:29:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:03 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:29:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:03 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:29:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:03 smithi043 ceph-mon[27010]: mgrmap e7: y(active, since 1.00548s) 2024-04-25T05:29:03.917 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:29:03.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T05:29:03.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T05:29:03.918 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:29:03.918 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 5 is available 2024-04-25T05:29:03.918 INFO:teuthology.orchestra.run.smithi043.stdout:Setting orchestrator backend to cephadm... 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:05.225 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:05 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:04] ENGINE Bus STARTING 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:04] ENGINE Serving on https://172.21.15.43:7150 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:04] ENGINE Serving on http://172.21.15.43:8765 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:04] ENGINE Bus STARTED 2024-04-25T05:29:06.456 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:06 smithi043 ceph-mon[27010]: mgrmap e8: y(active, since 2s) 2024-04-25T05:29:06.460 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T05:29:06.460 INFO:teuthology.orchestra.run.smithi043.stdout:Generating ssh key... 2024-04-25T05:29:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-mon[27010]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: Generating public/private rsa key pair. 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: Your identification has been saved in /tmp/tmpm3qma7zq/key. 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: Your public key has been saved in /tmp/tmpm3qma7zq/key.pub. 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: The key fingerprint is: 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: SHA256:S1XLTMXkTi00zQqEAO9gr3o75sHEbxo09ldtTonuqko ceph-84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: The key's randomart image is: 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: +---[RSA 3072]----+ 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | .... o++=o | 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | . .=.+.oo| 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | o . . +.+..| 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | o + . =.o | 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | * S o * | 2024-04-25T05:29:08.129 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | = * . o + | 2024-04-25T05:29:08.130 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | E = . . . | 2024-04-25T05:29:08.130 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | o+= . . | 2024-04-25T05:29:08.130 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: | .+=+..... | 2024-04-25T05:29:08.130 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:07 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: +----[SHA256]-----+ 2024-04-25T05:29:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:08 smithi043 ceph-mon[27010]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:08 smithi043 ceph-mon[27010]: Generating ssh key... 2024-04-25T05:29:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:08 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:08 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:09 smithi043 ceph-mon[27010]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:09.780 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU1LAg2bCelY6LO+qIbIG+HmoJPApJNKmBxsMkWud+JmZ/7Qiza27CazGFcBvtuV0wG/kC8dJMeRJZEELROR23qU8MKAHuXEEr08oL2kDOVUI6rqIh6EQXxz0g9TFd7G/34G2E9Uu6mbtAscqGgO/Te/kJGJoMIwM0T7xhjxjDewnrDOmpxhVsAQ1dRn+bNxt65ChZBGsNYO/gaNuRucEXRmq9jTH6RYm8k/ovOQ56Ry9i1rR6SaMbocxvoYFFerUtmVtzdcZ5oDtDi4hUXcB3CWB1bwLm+8QSlejVoJ3rqI8Bk42+KJlkYlyefBTrg0yjD95Q77c+QYM0dyBg4R4kimpZlJyB9VKyZuQkZqZHzq5GpusMViXr8wYN3zaX4p4lqKPt4bSTZvHYf5f1xlhwsXt3XcOHDWw5AZ0cCtfjLpYMrsJm3zzZ5UafWylbE48OcMfS9QZ1d6NcPLrL/ltfFVtQ7zwN72RBROHrFvmc+UMKUoT4LZ56nkIFsRHQsH0= ceph-84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:29:09.780 INFO:teuthology.orchestra.run.smithi043.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T05:29:09.780 INFO:teuthology.orchestra.run.smithi043.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T05:29:09.781 INFO:teuthology.orchestra.run.smithi043.stdout:Adding host smithi043... 2024-04-25T05:29:11.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:11 smithi043 ceph-mon[27010]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:11.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:11 smithi043 ceph-mon[27010]: Deploying cephadm binary to smithi043 2024-04-25T05:29:15.387 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Added host 'smithi043' with addr '172.21.15.43' 2024-04-25T05:29:15.387 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mon service... 2024-04-25T05:29:16.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:15 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:16.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:15 smithi043 ceph-mon[27010]: Added host smithi043 2024-04-25T05:29:16.006 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:15 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:17.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:17 smithi043 ceph-mon[27010]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:17.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:17 smithi043 ceph-mon[27010]: Saving service mon spec with placement count:5 2024-04-25T05:29:17.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:17 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:17.350 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T05:29:17.351 INFO:teuthology.orchestra.run.smithi043.stdout:Deploying unmanaged mgr service... 2024-04-25T05:29:18.916 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T05:29:19.174 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:19 smithi043 ceph-mon[27010]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:19.174 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:19 smithi043 ceph-mon[27010]: Saving service mgr spec with placement count:2 2024-04-25T05:29:19.174 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:19 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:19.174 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:19 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:19.175 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:19 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2558350031' entity='client.admin' 2024-04-25T05:29:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:20 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:22.185 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling the dashboard module... 2024-04-25T05:29:22.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/419251254' entity='client.admin' 2024-04-25T05:29:22.764 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:22 smithi043 ceph-mon[27010]: from='mgr.14120 172.21.15.43:0/302611498' entity='mgr.y' 2024-04-25T05:29:23.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:23 smithi043 ceph-mon[27010]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:29:23.803 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/835362502' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T05:29:23.803 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:23 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setuser ceph since I am not root 2024-04-25T05:29:23.804 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:23 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setgroup ceph since I am not root 2024-04-25T05:29:23.804 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:23 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:23.801+0000 7f19ec730200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:29:24.107 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:23.998+0000 7f19ec730200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:29:24.107 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:24.106+0000 7f19ec730200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:29:24.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/835362502' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T05:29:24.730 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-mon[27010]: mgrmap e9: y(active, since 21s) 2024-04-25T05:29:24.730 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:24.430+0000 7f19ec730200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:29:24.730 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:24.520+0000 7f19ec730200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for the mgr to restart... 2024-04-25T05:29:25.754 INFO:teuthology.orchestra.run.smithi043.stdout:Waiting for mgr epoch 9... 2024-04-25T05:29:25.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4037124172' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T05:29:25.876 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:25 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:25.807+0000 7f19ec730200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:29:26.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:25 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:25.986+0000 7f19ec730200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:29:26.473 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:26.168+0000 7f19ec730200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:29:26.473 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:26.471+0000 7f19ec730200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:29:26.780 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:26.567+0000 7f19ec730200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:29:26.781 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:26.779+0000 7f19ec730200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:29:27.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:26.980+0000 7f19ec730200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:29:27.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:27 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:27.606+0000 7f19ec730200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:29:28.356 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:28.045+0000 7f19ec730200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:29:28.356 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:28.145+0000 7f19ec730200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:29:28.357 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:28.356+0000 7f19ec730200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:29:29.255 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:28.904+0000 7f19ec730200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:29:29.256 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.000+0000 7f19ec730200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:29:29.612 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.255+0000 7f19ec730200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:29:29.612 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.393+0000 7f19ec730200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:29:29.612 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.491+0000 7f19ec730200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:29:29.612 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.611+0000 7f19ec730200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:29:30.391 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:29.924+0000 7f19ec730200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:29:30.832 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:30.553+0000 7f19ec730200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:29:30.832 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:29:30.779+0000 7f19ec730200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:29:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: Active manager daemon y restarted 2024-04-25T05:29:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: Activating manager daemon y 2024-04-25T05:29:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: mgrmap e10: y(active, starting, since 0.00328455s) 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: Manager daemon y is now available 2024-04-25T05:29:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:32.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:29:32.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:29:32.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:31 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:31] ENGINE Bus STARTING 2024-04-25T05:29:32.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:31 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:31] ENGINE Serving on http://172.21.15.43:8765 2024-04-25T05:29:32.136 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:31 smithi043 ceph-mon[27010]: mgrmap e11: y(active, since 1.00585s) 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout { 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout } 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:mgr epoch 9 is available 2024-04-25T05:29:32.780 INFO:teuthology.orchestra.run.smithi043.stdout:Generating a dashboard self-signed certificate... 2024-04-25T05:29:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:31] ENGINE Serving on https://172.21.15.43:7150 2024-04-25T05:29:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:29:31] ENGINE Bus STARTED 2024-04-25T05:29:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:33 smithi043 ceph-mon[27010]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T05:29:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:33 smithi043 ceph-mon[27010]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T05:29:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:34.360 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T05:29:34.360 INFO:teuthology.orchestra.run.smithi043.stdout:Creating initial admin user... 2024-04-25T05:29:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:34 smithi043 ceph-mon[27010]: mgrmap e12: y(active, since 2s) 2024-04-25T05:29:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:34 smithi043 ceph-mon[27010]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:34 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:34 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:34 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:36.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:36 smithi043 ceph-mon[27010]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:36.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:36 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:36.681 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$U3Lp2g6Y.l9tqAqaaYlL..aS5bFI0sR46dcY003hjrQqqJqIt/5ty", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714022975, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T05:29:36.681 INFO:teuthology.orchestra.run.smithi043.stdout:Fetching dashboard port number... 2024-04-25T05:29:38.054 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T05:29:38.065 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T05:29:38.065 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout disabled 2024-04-25T05:29:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T05:29:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:systemctl: stdout inactive 2024-04-25T05:29:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:firewalld.service is not enabled 2024-04-25T05:29:38.074 INFO:teuthology.orchestra.run.smithi043.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:Ceph Dashboard is now available at: 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout: URL: https://smithi043:8443/ 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout: User: admin 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout: Password: lgw5azgpq3 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:Saving cluster configuration to /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config directory 2024-04-25T05:29:38.077 INFO:teuthology.orchestra.run.smithi043.stdout:Enabling autotune for osd_memory_target 2024-04-25T05:29:38.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:38 smithi043 ceph-mon[27010]: mgrmap e13: y(active, since 6s) 2024-04-25T05:29:38.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1802096373' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T05:29:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: ceph telemetry on 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout:For more information see: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:40.818 INFO:teuthology.orchestra.run.smithi043.stdout:Bootstrap complete. 2024-04-25T05:29:40.867 INFO:tasks.cephadm:Fetching config... 2024-04-25T05:29:40.867 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:29:40.867 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T05:29:40.886 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T05:29:40.886 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:29:40.886 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T05:29:40.942 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T05:29:40.942 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:29:40.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T05:29:41.008 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T05:29:41.008 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:29:41.009 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T05:29:41.065 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T05:29:41.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU1LAg2bCelY6LO+qIbIG+HmoJPApJNKmBxsMkWud+JmZ/7Qiza27CazGFcBvtuV0wG/kC8dJMeRJZEELROR23qU8MKAHuXEEr08oL2kDOVUI6rqIh6EQXxz0g9TFd7G/34G2E9Uu6mbtAscqGgO/Te/kJGJoMIwM0T7xhjxjDewnrDOmpxhVsAQ1dRn+bNxt65ChZBGsNYO/gaNuRucEXRmq9jTH6RYm8k/ovOQ56Ry9i1rR6SaMbocxvoYFFerUtmVtzdcZ5oDtDi4hUXcB3CWB1bwLm+8QSlejVoJ3rqI8Bk42+KJlkYlyefBTrg0yjD95Q77c+QYM0dyBg4R4kimpZlJyB9VKyZuQkZqZHzq5GpusMViXr8wYN3zaX4p4lqKPt4bSTZvHYf5f1xlhwsXt3XcOHDWw5AZ0cCtfjLpYMrsJm3zzZ5UafWylbE48OcMfS9QZ1d6NcPLrL/ltfFVtQ7zwN72RBROHrFvmc+UMKUoT4LZ56nkIFsRHQsH0= ceph-84884b5e-02c4-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T05:29:41.142 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU1LAg2bCelY6LO+qIbIG+HmoJPApJNKmBxsMkWud+JmZ/7Qiza27CazGFcBvtuV0wG/kC8dJMeRJZEELROR23qU8MKAHuXEEr08oL2kDOVUI6rqIh6EQXxz0g9TFd7G/34G2E9Uu6mbtAscqGgO/Te/kJGJoMIwM0T7xhjxjDewnrDOmpxhVsAQ1dRn+bNxt65ChZBGsNYO/gaNuRucEXRmq9jTH6RYm8k/ovOQ56Ry9i1rR6SaMbocxvoYFFerUtmVtzdcZ5oDtDi4hUXcB3CWB1bwLm+8QSlejVoJ3rqI8Bk42+KJlkYlyefBTrg0yjD95Q77c+QYM0dyBg4R4kimpZlJyB9VKyZuQkZqZHzq5GpusMViXr8wYN3zaX4p4lqKPt4bSTZvHYf5f1xlhwsXt3XcOHDWw5AZ0cCtfjLpYMrsJm3zzZ5UafWylbE48OcMfS9QZ1d6NcPLrL/ltfFVtQ7zwN72RBROHrFvmc+UMKUoT4LZ56nkIFsRHQsH0= ceph-84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:29:41.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU1LAg2bCelY6LO+qIbIG+HmoJPApJNKmBxsMkWud+JmZ/7Qiza27CazGFcBvtuV0wG/kC8dJMeRJZEELROR23qU8MKAHuXEEr08oL2kDOVUI6rqIh6EQXxz0g9TFd7G/34G2E9Uu6mbtAscqGgO/Te/kJGJoMIwM0T7xhjxjDewnrDOmpxhVsAQ1dRn+bNxt65ChZBGsNYO/gaNuRucEXRmq9jTH6RYm8k/ovOQ56Ry9i1rR6SaMbocxvoYFFerUtmVtzdcZ5oDtDi4hUXcB3CWB1bwLm+8QSlejVoJ3rqI8Bk42+KJlkYlyefBTrg0yjD95Q77c+QYM0dyBg4R4kimpZlJyB9VKyZuQkZqZHzq5GpusMViXr8wYN3zaX4p4lqKPt4bSTZvHYf5f1xlhwsXt3XcOHDWw5AZ0cCtfjLpYMrsJm3zzZ5UafWylbE48OcMfS9QZ1d6NcPLrL/ltfFVtQ7zwN72RBROHrFvmc+UMKUoT4LZ56nkIFsRHQsH0= ceph-84884b5e-02c4-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T05:29:41.194 INFO:teuthology.orchestra.run.smithi060.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU1LAg2bCelY6LO+qIbIG+HmoJPApJNKmBxsMkWud+JmZ/7Qiza27CazGFcBvtuV0wG/kC8dJMeRJZEELROR23qU8MKAHuXEEr08oL2kDOVUI6rqIh6EQXxz0g9TFd7G/34G2E9Uu6mbtAscqGgO/Te/kJGJoMIwM0T7xhjxjDewnrDOmpxhVsAQ1dRn+bNxt65ChZBGsNYO/gaNuRucEXRmq9jTH6RYm8k/ovOQ56Ry9i1rR6SaMbocxvoYFFerUtmVtzdcZ5oDtDi4hUXcB3CWB1bwLm+8QSlejVoJ3rqI8Bk42+KJlkYlyefBTrg0yjD95Q77c+QYM0dyBg4R4kimpZlJyB9VKyZuQkZqZHzq5GpusMViXr8wYN3zaX4p4lqKPt4bSTZvHYf5f1xlhwsXt3XcOHDWw5AZ0cCtfjLpYMrsJm3zzZ5UafWylbE48OcMfS9QZ1d6NcPLrL/ltfFVtQ7zwN72RBROHrFvmc+UMKUoT4LZ56nkIFsRHQsH0= ceph-84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:29:41.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T05:29:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2276448186' entity='client.admin' 2024-04-25T05:29:41.458 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:29:42.885 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T05:29:42.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T05:29:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:29:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/505079817' entity='client.admin' 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:44.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:45.588 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi060 2024-04-25T05:29:45.588 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:29:45.588 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/etc/ceph/ceph.conf 2024-04-25T05:29:45.608 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:29:45.608 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:29:45.666 INFO:tasks.cephadm:Adding host smithi060 to orchestrator... 2024-04-25T05:29:45.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch host add smithi060 2024-04-25T05:29:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:45.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:46.239 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:29:46.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:46 smithi043 ceph-mon[27010]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:29:46.623 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:46 smithi043 ceph-mon[27010]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:29:46.623 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:46 smithi043 ceph-mon[27010]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:29:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:47 smithi043 ceph-mon[27010]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi060", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:47 smithi043 ceph-mon[27010]: Deploying cephadm binary to smithi060 2024-04-25T05:29:51.203 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi060' with addr '172.21.15.60' 2024-04-25T05:29:52.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T05:29:52.269 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:29:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:52 smithi043 ceph-mon[27010]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:29:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:52 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:52 smithi043 ceph-mon[27010]: Added host smithi060 2024-04-25T05:29:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:52 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:29:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:52 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:53.000 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:29:53.000 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}] 2024-04-25T05:29:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:53 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:53.692 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T05:29:53.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T05:29:53.932 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:29:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:54 smithi043 ceph-mon[27010]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:29:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:54 smithi043 ceph-mon[27010]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:29:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:55.318 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-04-25T05:29:55.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2372340038' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T05:29:55.980 INFO:tasks.cephadm:Adding mon.a on smithi043 2024-04-25T05:29:55.981 INFO:tasks.cephadm:Adding mon.c on smithi043 2024-04-25T05:29:55.981 INFO:tasks.cephadm:Adding mon.b on smithi060 2024-04-25T05:29:55.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi060:172.21.15.60=b' 2024-04-25T05:29:56.226 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-25T05:29:56.304 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-25T05:29:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:56 smithi043 ceph-mon[27010]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:29:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2372340038' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T05:29:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:56 smithi043 ceph-mon[27010]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:29:57.529 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled mon update... 2024-04-25T05:29:58.193 DEBUG:teuthology.orchestra.run.smithi043:mon.c> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T05:29:58.196 DEBUG:teuthology.orchestra.run.smithi060:mon.b> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T05:29:58.198 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T05:29:58.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T05:29:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:29:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi060:172.21.15.60=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi060:172.21.15.60=b;count:3 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:58 smithi043 ceph-mon[27010]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:29:58.485 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:29:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:29:59.299 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":1,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","modified":"2024-04-25T05:28:30.314004Z","created":"2024-04-25T05:28:30.314004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T05:29:59.301 INFO:teuthology.orchestra.run.smithi060.stderr:dumped monmap epoch 1 2024-04-25T05:29:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:29:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:29:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:29:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:29:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:29:59 smithi043 ceph-mon[27010]: Deploying daemon mon.b on smithi060 2024-04-25T05:30:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:00 smithi043 ceph-mon[27010]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/1475904251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:01.885 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T05:30:01.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T05:30:02.230 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:30:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:02 smithi043 ceph-mon[27010]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:02 smithi060 systemd[1]: Starting Ceph mon.b for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:30:03.091 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:30:03.091 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":1,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","modified":"2024-04-25T05:28:30.314004Z","created":"2024-04-25T05:28:30.314004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T05:30:03.094 INFO:teuthology.orchestra.run.smithi060.stderr:dumped monmap epoch 1 2024-04-25T05:30:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 podman[30863]: 2024-04-25 05:30:03.000008797 +0000 UTC m=+0.106981999 container create 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:30:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 podman[30863]: 2024-04-25 05:30:02.911675785 +0000 UTC m=+0.018648971 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:30:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/708796969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 podman[30863]: 2024-04-25 05:30:03.722421377 +0000 UTC m=+0.829394562 container init 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T05:30:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 podman[30863]: 2024-04-25 05:30:03.726002102 +0000 UTC m=+0.832975290 container start 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T05:30:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev), process ceph-mon, pid 2 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: pidfile_write: ignore empty --pid-file 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: load: jerasure load: lrc 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: RocksDB version: 7.9.2 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Git sha 0 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Compile date 2024-04-25 01:05:22 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: DB SUMMARY 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: DB Session ID: BUK9MFY4UJQ32S92IP2B 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: CURRENT file: CURRENT 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-04-25T05:30:04.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.error_if_exists: 0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.create_if_missing: 0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.paranoid_checks: 1 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.env: 0x5598a212bcc0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.info_log: 0x5598a47cdd80 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.statistics: (nil) 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.use_fsync: 0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_log_file_size: 0 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T05:30:04.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_fallocate: 1 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.use_direct_reads: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.db_log_dir: 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.wal_dir: 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T05:30:04.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.write_buffer_manager: 0x5598a479f0e0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.unordered_write: 0 2024-04-25T05:30:04.003 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T05:30:04.004 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T05:30:04.004 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T05:30:04.004 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T05:30:04.004 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.row_cache: None 2024-04-25T05:30:04.004 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.wal_filter: None 2024-04-25T05:30:04.005 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T05:30:04.005 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T05:30:04.005 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.two_write_queues: 0 2024-04-25T05:30:04.005 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.wal_compression: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.atomic_flush: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.log_readahead_size: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T05:30:04.006 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_background_jobs: 2 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_background_compactions: -1 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_subcompactions: 1 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_open_files: -1 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T05:30:04.007 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_background_flushes: -1 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Compression algorithms supported: 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kZSTD supported: 0 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kXpressCompression supported: 0 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kZlibCompression supported: 1 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kSnappyCompression supported: 1 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kLZ4Compression supported: 1 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: kBZip2Compression supported: 0 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T05:30:04.008 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.merge_operator: 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_filter: None 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5598a47cdf40) 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: cache_index_and_filter_blocks: 1 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: pin_top_level_index_and_filter: 1 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_type: 0 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: data_block_index_type: 0 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_shortening: 1 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T05:30:04.009 INFO:journalctl@ceph.mon.b.smithi060.stdout: checksum: 4 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: no_block_cache: 0 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache: 0x5598a47d11f0 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_name: BinnedLRUCache 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_options: 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: capacity : 536870912 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: num_shard_bits : 4 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: strict_capacity_limit : 0 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: high_pri_pool_ratio: 0.000 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_cache_compressed: (nil) 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: persistent_cache: (nil) 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_size: 4096 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_size_deviation: 10 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_restart_interval: 16 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: index_block_restart_interval: 1 2024-04-25T05:30:04.010 INFO:journalctl@ceph.mon.b.smithi060.stdout: metadata_block_size: 4096 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: partition_filters: 0 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: use_delta_encoding: 1 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: filter_policy: bloomfilter 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: whole_key_filtering: 1 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: verify_compression: 0 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: read_amp_bytes_per_bit: 0 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: format_version: 5 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: enable_index_compression: 1 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: block_align: 0 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: max_auto_readahead_size: 262144 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: prepopulate_block_cache: 0 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: initial_auto_readahead_size: 8192 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T05:30:04.011 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression: NoCompression 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.num_levels: 7 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T05:30:04.012 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T05:30:04.013 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T05:30:04.014 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.inplace_update_support: 0 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T05:30:04.015 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.bloom_locality: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.max_successive_merges: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.ttl: 2592000 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enable_blob_files: false 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.min_blob_size: 0 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T05:30:04.016 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ebb5b9dd-2e99-493d-bafc-88a0638961f7 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023003757704, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023003758146, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714023003, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ebb5b9dd-2e99-493d-bafc-88a0638961f7", "db_session_id": "BUK9MFY4UJQ32S92IP2B", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023003758233, "job": 1, "event": "recovery_finished"} 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T05:30:04.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5598a48ee000 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: DB pointer 0x5598a48da000 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** DB Stats ** 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.018 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** Compaction Stats [default] ** 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T05:30:04.019 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: Block cache BinnedLRUCache@0x5598a47d11f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: using public_addr v2:172.21.15.60:0/0 -> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: starting mon.b rank -1 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(???) e0 preinit fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:04.020 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: e1 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: legacy client fscid: -1 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout: No filesystems configured 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:04.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: Deploying daemon mon.b on smithi060 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/1475904251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/708796969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 ceph-mon[30898]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 bash[30863]: 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 2024-04-25T05:30:04.022 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:03 smithi060 systemd[1]: Started Ceph mon.b for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:30:04.658 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T05:30:04.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T05:30:04.907 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:30:09.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:09.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: mon.a calling monitor election 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: mon.b calling monitor election 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: monmap epoch 2 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: last_changed 2024-04-25T05:30:03.780730+0000 2024-04-25T05:30:09.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: min_mon_release 19 (squid) 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: election_strategy: 1 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: fsmap 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: mgrmap e13: y(active, since 38s) 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: overall HEALTH_OK 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:09.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: mon.a calling monitor election 2024-04-25T05:30:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: mon.b calling monitor election 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: monmap epoch 2 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: last_changed 2024-04-25T05:30:03.780730+0000 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: min_mon_release 19 (squid) 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: election_strategy: 1 2024-04-25T05:30:09.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: fsmap 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: mgrmap e13: y(active, since 38s) 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: overall HEALTH_OK 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:09.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:09.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:30:09.251 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","modified":"2024-04-25T05:30:03.780730Z","created":"2024-04-25T05:28:30.314004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T05:30:09.252 INFO:teuthology.orchestra.run.smithi060.stderr:dumped monmap epoch 2 2024-04-25T05:30:10.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:09 smithi043 ceph-mon[27010]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:10.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:09 smithi043 ceph-mon[27010]: Deploying daemon mon.c on smithi043 2024-04-25T05:30:10.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/3754060642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:10.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:09 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:09 smithi060 ceph-mon[30898]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:09 smithi060 ceph-mon[30898]: Deploying daemon mon.c on smithi043 2024-04-25T05:30:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/3754060642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:09 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:10.782 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T05:30:10.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T05:30:11.035 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:30:11.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:30:10 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:30:10.780+0000 7f19b238c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T05:30:11.791 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:30:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","modified":"2024-04-25T05:30:03.780730Z","created":"2024-04-25T05:28:30.314004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T05:30:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:dumped monmap epoch 2 2024-04-25T05:30:12.093 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:11 smithi060 ceph-mon[30898]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.093 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:11 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/429825919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:11 smithi043 ceph-mon[27010]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:11 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/429825919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:11 smithi043 systemd[1]: Starting Ceph mon.c for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:30:12.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 podman[37471]: 2024-04-25 05:30:12.06262647 +0000 UTC m=+0.018535245 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:30:12.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 podman[37471]: 2024-04-25 05:30:12.179108817 +0000 UTC m=+0.135017621 container create 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T05:30:12.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 podman[37471]: 2024-04-25 05:30:12.428542761 +0000 UTC m=+0.384451526 container init 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 podman[37471]: 2024-04-25 05:30:12.432108989 +0000 UTC m=+0.388017756 container start 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True) 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: ceph version 19.0.0-3293-gd6f5f4e5 (d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737) squid (dev), process ceph-mon, pid 2 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pidfile_write: ignore empty --pid-file 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: load: jerasure load: lrc 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: RocksDB version: 7.9.2 2024-04-25T05:30:12.761 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Git sha 0 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Compile date 2024-04-25 01:05:22 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: DB SUMMARY 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: DB Session ID: SDVUPRI4L8448XLPPZIK 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: CURRENT file: CURRENT 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.error_if_exists: 0 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.create_if_missing: 0 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.paranoid_checks: 1 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T05:30:12.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.env: 0x560f7bd1ccc0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.info_log: 0x560f7e487d80 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.statistics: (nil) 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.use_fsync: 0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_log_file_size: 0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T05:30:12.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_fallocate: 1 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.use_direct_reads: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.db_log_dir: 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.wal_dir: 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T05:30:12.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.write_buffer_manager: 0x560f7e4590e0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.unordered_write: 0 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T05:30:12.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.row_cache: None 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.wal_filter: None 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.two_write_queues: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.wal_compression: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.atomic_flush: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T05:30:12.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.log_readahead_size: 0 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_background_jobs: 2 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_background_compactions: -1 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_subcompactions: 1 2024-04-25T05:30:12.768 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_open_files: -1 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_background_flushes: -1 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Compression algorithms supported: 2024-04-25T05:30:12.769 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kZSTD supported: 0 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kXpressCompression supported: 0 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kZlibCompression supported: 1 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kSnappyCompression supported: 1 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kLZ4Compression supported: 1 2024-04-25T05:30:12.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: kBZip2Compression supported: 0 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.merge_operator: 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_filter: None 2024-04-25T05:30:12.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560f7e487f40) 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_type: 0 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_index_type: 0 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_shortening: 1 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: checksum: 4 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: no_block_cache: 0 2024-04-25T05:30:12.772 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache: 0x560f7e48b1f0 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_options: 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: capacity : 536870912 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: num_shard_bits : 4 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: strict_capacity_limit : 0 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_compressed: (nil) 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: persistent_cache: (nil) 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size: 4096 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size_deviation: 10 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_restart_interval: 16 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_block_restart_interval: 1 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: metadata_block_size: 4096 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: partition_filters: 0 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: use_delta_encoding: 1 2024-04-25T05:30:12.773 INFO:journalctl@ceph.mon.c.smithi043.stdout: filter_policy: bloomfilter 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: whole_key_filtering: 1 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: verify_compression: 0 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: format_version: 5 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_index_compression: 1 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_align: 0 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: max_auto_readahead_size: 262144 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: prepopulate_block_cache: 0 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: initial_auto_readahead_size: 8192 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression: NoCompression 2024-04-25T05:30:12.774 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.num_levels: 7 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:30:12.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T05:30:12.776 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T05:30:12.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T05:30:12.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.inplace_update_support: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.bloom_locality: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.max_successive_merges: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T05:30:12.779 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.ttl: 2592000 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enable_blob_files: false 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.min_blob_size: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T05:30:12.780 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ba6cad58-2f66-46a4-b04a-034f6157d63f 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023012462269, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023012462683, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714023012, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ba6cad58-2f66-46a4-b04a-034f6157d63f", "db_session_id": "SDVUPRI4L8448XLPPZIK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714023012462766, "job": 1, "event": "recovery_finished"} 2024-04-25T05:30:12.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560f7e5a8000 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: DB pointer 0x560f7e594000 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** DB Stats ** 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T05:30:12.782 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T05:30:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Block cache BinnedLRUCache@0x560f7e48b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T05:30:12.784 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: using public_addrv [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: starting mon.c rank -1 at public addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] at bind addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(???) e0 preinit fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: e1 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: legacy client fscid: -1 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout: No filesystems configured 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T05:30:12.785 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.786 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: Deploying daemon mon.b on smithi060 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/1475904251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/708796969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.a calling monitor election 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.b calling monitor election 2024-04-25T05:30:12.787 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: monmap epoch 2 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: last_changed 2024-04-25T05:30:03.780730+0000 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: min_mon_release 19 (squid) 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: election_strategy: 1 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: fsmap 2024-04-25T05:30:12.788 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mgrmap e13: y(active, since 38s) 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: overall HEALTH_OK 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: Deploying daemon mon.c on smithi043 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/3754060642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:12.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:12.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/429825919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:12.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 ceph-mon[37492]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T05:30:12.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 bash[37471]: 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 2024-04-25T05:30:12.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:12 smithi043 systemd[1]: Started Ceph mon.c for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:30:13.328 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T05:30:13.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T05:30:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:30:17.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: mon.a calling monitor election 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: mon.b calling monitor election 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: monmap epoch 3 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: min_mon_release 19 (squid) 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: election_strategy: 1 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: fsmap 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: mgrmap e13: y(active, since 46s) 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-25T05:30:17.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-25T05:30:17.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-25T05:30:17.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: mon.c (rank 2) addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] is down (out of quorum) 2024-04-25T05:30:17.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:17.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:17.927 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:30:17.927 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","modified":"2024-04-25T05:30:12.484487Z","created":"2024-04-25T05:28:30.314004Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: mon.a calling monitor election 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: mon.b calling monitor election 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.928 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: monmap epoch 3 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: min_mon_release 19 (squid) 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: election_strategy: 1 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: fsmap 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: mgrmap e13: y(active, since 46s) 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-25T05:30:17.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-25T05:30:17.930 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-25T05:30:17.930 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: mon.c (rank 2) addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] is down (out of quorum) 2024-04-25T05:30:17.930 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:17.930 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:17.930 INFO:teuthology.orchestra.run.smithi060.stderr:dumped monmap epoch 3 2024-04-25T05:30:18.476 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T05:30:18.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/1870667152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/1870667152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: mon.a calling monitor election 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: mon.b calling monitor election 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: monmap epoch 3 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: min_mon_release 19 (squid) 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: election_strategy: 1 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:19.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: fsmap 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: mgrmap e13: y(active, since 46s) 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: mon.c (rank 2) addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] is down (out of quorum) 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/1870667152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:30:19.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:19.692 INFO:teuthology.orchestra.run.smithi043.stdout:# minimal ceph.conf for 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:19.692 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-04-25T05:30:19.692 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:19.692 INFO:teuthology.orchestra.run.smithi043.stdout: mon_host = [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-25T05:30:20.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mon.c calling monitor election 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mon.c calling monitor election 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mon.b calling monitor election 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mon.a calling monitor election 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: monmap epoch 3 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:20.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: min_mon_release 19 (squid) 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: election_strategy: 1 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:20.325 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: fsmap 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: mgrmap e13: y(active, since 48s) 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: overall HEALTH_OK 2024-04-25T05:30:20.326 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2789149766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:20.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:20.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mon.c calling monitor election 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.329 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mon.c calling monitor election 2024-04-25T05:30:20.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mon.b calling monitor election 2024-04-25T05:30:20.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mon.a calling monitor election 2024-04-25T05:30:20.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-25T05:30:20.330 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: monmap epoch 3 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: min_mon_release 19 (squid) 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: election_strategy: 1 2024-04-25T05:30:20.331 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: fsmap 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: mgrmap e13: y(active, since 48s) 2024-04-25T05:30:20.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: overall HEALTH_OK 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2789149766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.334 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:20.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:20.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.350 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T05:30:20.350 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:30:20.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T05:30:20.379 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:30:20.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:30:20.444 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:30:20.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T05:30:20.469 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mon.c calling monitor election 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mon.c calling monitor election 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mon.b calling monitor election 2024-04-25T05:30:20.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mon.a calling monitor election 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: monmap epoch 3 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: last_changed 2024-04-25T05:30:12.484487+0000 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: created 2024-04-25T05:28:30.314004+0000 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: min_mon_release 19 (squid) 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: election_strategy: 1 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: 1: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.b 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.c 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: fsmap 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: mgrmap e13: y(active, since 48s) 2024-04-25T05:30:20.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: overall HEALTH_OK 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2789149766' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:20.472 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:20.474 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:30:20.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:30:20.542 INFO:tasks.cephadm:Adding mgr.y on smithi043 2024-04-25T05:30:20.542 INFO:tasks.cephadm:Adding mgr.x on smithi060 2024-04-25T05:30:20.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi043=y;smithi060=x' 2024-04-25T05:30:20.823 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:30:21.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[27010]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T05:30:21.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[27010]: Reconfiguring daemon mon.a on smithi043 2024-04-25T05:30:21.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:21.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:21.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[37492]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T05:30:21.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[37492]: Reconfiguring daemon mon.a on smithi043 2024-04-25T05:30:21.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:21.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled mgr update... 2024-04-25T05:30:21.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:21 smithi060 ceph-mon[30898]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T05:30:21.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:21 smithi060 ceph-mon[30898]: Reconfiguring daemon mon.a on smithi043 2024-04-25T05:30:21.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:30:21.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.078 DEBUG:teuthology.orchestra.run.smithi060:mgr.x> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T05:30:22.084 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T05:30:22.084 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:30:22.084 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T05:30:22.100 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T05:30:22.101 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-04-25T05:30:22.156 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 05:30:11.876301040 +0000 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 05:24:41.436383269 +0000 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 05:24:41.436383269 +0000 2024-04-25T05:30:22.157 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 05:24:41.436383269 +0000 2024-04-25T05:30:22.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T05:30:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T05:30:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T05:30:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000108045 s, 4.7 MB/s 2024-04-25T05:30:22.222 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T05:30:22.278 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:22.333 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 05:30:11.877301028 +0000 2024-04-25T05:30:22.334 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 05:24:41.767379080 +0000 2024-04-25T05:30:22.334 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 05:24:41.767379080 +0000 2024-04-25T05:30:22.334 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 05:24:41.767379080 +0000 2024-04-25T05:30:22.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T05:30:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T05:30:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T05:30:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000109244 s, 4.7 MB/s 2024-04-25T05:30:22.398 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T05:30:22.457 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-04-25T05:30:22.512 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T05:30:22.512 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:22.512 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T05:30:22.512 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:22.513 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:22.513 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 05:30:11.877301028 +0000 2024-04-25T05:30:22.513 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 05:24:42.098374890 +0000 2024-04-25T05:30:22.513 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 05:24:42.098374890 +0000 2024-04-25T05:30:22.513 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 05:24:42.098374890 +0000 2024-04-25T05:30:22.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T05:30:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T05:30:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T05:30:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000104988 s, 4.9 MB/s 2024-04-25T05:30:22.577 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T05:30:22.632 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-25 05:30:11.878301015 +0000 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-25 05:24:42.429370700 +0000 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-25 05:24:42.429370700 +0000 2024-04-25T05:30:22.688 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-04-25 05:24:42.429370700 +0000 2024-04-25T05:30:22.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T05:30:22.748 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi060=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: Saving service mgr spec with placement smithi043=y;smithi060=x;count:2 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: Reconfiguring mgr.y (unknown last config time)... 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[27010]: Reconfiguring daemon mgr.y on smithi043 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:22.749 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi060=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: Saving service mgr spec with placement smithi043=y;smithi060=x;count:2 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: Reconfiguring mgr.y (unknown last config time)... 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:22.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:22 smithi043 ceph-mon[37492]: Reconfiguring daemon mgr.y on smithi043 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi060=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: Saving service mgr spec with placement smithi043=y;smithi060=x;count:2 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: Reconfiguring mgr.y (unknown last config time)... 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:22.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:22 smithi060 ceph-mon[30898]: Reconfiguring daemon mgr.y on smithi043 2024-04-25T05:30:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-25T05:30:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-25T05:30:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 9.9577e-05 s, 5.1 MB/s 2024-04-25T05:30:22.753 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T05:30:22.809 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:30:22.809 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T05:30:22.828 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T05:30:22.828 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-25 05:30:02.695504296 +0000 2024-04-25T05:30:22.887 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-25 05:24:41.432542317 +0000 2024-04-25T05:30:22.888 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-25 05:24:41.432542317 +0000 2024-04-25T05:30:22.888 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-25 05:24:41.432542317 +0000 2024-04-25T05:30:22.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T05:30:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-25T05:30:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-25T05:30:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000132314 s, 3.9 MB/s 2024-04-25T05:30:22.957 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T05:30:23.017 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-04-25T05:30:23.075 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T05:30:23.075 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-25 05:30:02.696504286 +0000 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-25 05:24:41.770538911 +0000 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-25 05:24:41.770538911 +0000 2024-04-25T05:30:23.076 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-25 05:24:41.770538911 +0000 2024-04-25T05:30:23.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T05:30:23.142 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-25T05:30:23.143 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-25T05:30:23.143 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000132786 s, 3.9 MB/s 2024-04-25T05:30:23.144 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T05:30:23.201 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-25 05:30:02.696504286 +0000 2024-04-25T05:30:23.258 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-25 05:24:42.126535324 +0000 2024-04-25T05:30:23.259 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-25 05:24:42.126535324 +0000 2024-04-25T05:30:23.259 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-25 05:24:42.126535324 +0000 2024-04-25T05:30:23.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T05:30:23.322 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-25T05:30:23.322 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-25T05:30:23.322 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000109512 s, 4.7 MB/s 2024-04-25T05:30:23.323 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T05:30:23.381 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-04-25 05:30:02.697504276 +0000 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-04-25 05:24:42.467531887 +0000 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-04-25 05:24:42.467531887 +0000 2024-04-25T05:30:23.437 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-04-25 05:24:42.467531887 +0000 2024-04-25T05:30:23.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T05:30:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-04-25T05:30:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-04-25T05:30:23.501 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.5617e-05 s, 5.4 MB/s 2024-04-25T05:30:23.502 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T05:30:23.559 INFO:tasks.cephadm:Deploying osd.0 on smithi043 with /dev/vg_nvme/lv_4... 2024-04-25T05:30:23.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-04-25T05:30:23.806 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:30:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: Reconfiguring mon.c (monmap changed)... 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[27010]: Reconfiguring daemon mon.c on smithi043 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: Reconfiguring mon.c (monmap changed)... 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:24 smithi043 ceph-mon[37492]: Reconfiguring daemon mon.c on smithi043 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: Reconfiguring mon.c (monmap changed)... 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:24 smithi060 ceph-mon[30898]: Reconfiguring daemon mon.c on smithi043 2024-04-25T05:30:26.886 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:26.887 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.887 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.887 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:26.887 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:26.887 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:26.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:26.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T05:30:26.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:27.601 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:30:27.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2024-04-25T05:30:27.886 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:30:27.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:27 smithi043 ceph-mon[27010]: Reconfiguring mon.b (monmap changed)... 2024-04-25T05:30:27.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:27 smithi043 ceph-mon[27010]: Reconfiguring daemon mon.b on smithi060 2024-04-25T05:30:27.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:27 smithi043 ceph-mon[37492]: Reconfiguring mon.b (monmap changed)... 2024-04-25T05:30:27.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:27 smithi043 ceph-mon[37492]: Reconfiguring daemon mon.b on smithi060 2024-04-25T05:30:27.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:27 smithi060 ceph-mon[30898]: Reconfiguring mon.b (monmap changed)... 2024-04-25T05:30:27.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:27 smithi060 ceph-mon[30898]: Reconfiguring daemon mon.b on smithi060 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T05:30:29.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:29.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:29.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:29.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:29 smithi043 ceph-mon[27010]: Deploying daemon mgr.x on smithi060 2024-04-25T05:30:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:29 smithi043 ceph-mon[27010]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:29.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:29 smithi043 ceph-mon[37492]: Deploying daemon mgr.x on smithi060 2024-04-25T05:30:29.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:29 smithi043 ceph-mon[37492]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:29.946 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:29 smithi060 ceph-mon[30898]: Deploying daemon mgr.x on smithi060 2024-04-25T05:30:29.947 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:29 smithi060 ceph-mon[30898]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:29.947 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:29 smithi060 systemd[1]: Starting Ceph mgr.x for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:30:30.232 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:29 smithi060 podman[32737]: 2024-04-25 05:30:29.946966282 +0000 UTC m=+0.113471792 container create 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:30:30.232 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:29 smithi060 podman[32737]: 2024-04-25 05:30:29.851850817 +0000 UTC m=+0.018356342 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:30:30.233 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 podman[32737]: 2024-04-25 05:30:30.165816221 +0000 UTC m=+0.332321731 container init 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:30:30.233 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 podman[32737]: 2024-04-25 05:30:30.168401007 +0000 UTC m=+0.334906536 container start 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5) 2024-04-25T05:30:30.233 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mgr[32756]: -- 172.21.15.60:0/2198208421 <== mon.1 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b84f32a5a0 con 0x55b84f2fec00 2024-04-25T05:30:30.233 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 bash[32737]: 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b 2024-04-25T05:30:30.498 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 systemd[1]: Started Ceph mgr.x for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:30:30.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:30.334+0000 7f9f67328200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:30.833 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:30.834 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.834 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.834 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.834 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.834 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:30.530+0000 7f9f67328200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:30.640+0000 7f9f67328200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:30:30.980 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:30.979+0000 7f9f67328200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:30:31.248 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:31.078+0000 7f9f67328200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:30:32.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]: dispatch 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]': finished 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]: dispatch 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]': finished 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:32.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]: dispatch 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/879024745' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e"}]': finished 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.738 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:32.383+0000 7f9f67328200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:30:32.738 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:32.560+0000 7f9f67328200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:30:32.738 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:32.737+0000 7f9f67328200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:30:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3989496250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:30:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3989496250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:30:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3989496250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:30:33.345 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:33.039+0000 7f9f67328200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:30:33.345 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:33.135+0000 7f9f67328200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:30:33.345 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:33.344+0000 7f9f67328200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:30:33.748 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:33.543+0000 7f9f67328200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:30:34.172 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:33 smithi060 ceph-mon[30898]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:34.376 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:33 smithi043 ceph-mon[27010]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:34.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:33 smithi043 ceph-mon[37492]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:34.498 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:34.171+0000 7f9f67328200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:30:34.916 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:34.609+0000 7f9f67328200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:30:34.917 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:34.708+0000 7f9f67328200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:30:34.917 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:34.916+0000 7f9f67328200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:30:35.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:35.460+0000 7f9f67328200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:30:35.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:35.555+0000 7f9f67328200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:30:36.164 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:35.809+0000 7f9f67328200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:30:36.164 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:35.946+0000 7f9f67328200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:30:36.164 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:36.044+0000 7f9f67328200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:30:36.164 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:36.163+0000 7f9f67328200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:30:36.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:36 smithi060 ceph-mon[30898]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:36.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:36.477+0000 7f9f67328200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:30:36.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:36 smithi043 ceph-mon[27010]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:36.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:36 smithi043 ceph-mon[37492]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:37.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:37.105+0000 7f9f67328200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:30:37.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:30:37.329+0000 7f9f67328200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:30:37.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[27010]: Standby manager daemon x started 2024-04-25T05:30:37.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:30:37.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:30:37.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:30:37.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:30:37.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[37492]: Standby manager daemon x started 2024-04-25T05:30:37.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:30:37.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:30:37.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:30:37.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:37 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:30:37.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-mon[30898]: Standby manager daemon x started 2024-04-25T05:30:37.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:30:37.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:30:37.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:30:37.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:37 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/826060206' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:30:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[27010]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[27010]: mgrmap e14: y(active, since 66s), standbys: x 2024-04-25T05:30:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:30:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[37492]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[37492]: mgrmap e14: y(active, since 66s), standbys: x 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T05:30:38.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:38.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:38 smithi060 ceph-mon[30898]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:38.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:38 smithi060 ceph-mon[30898]: mgrmap e14: y(active, since 66s), standbys: x 2024-04-25T05:30:38.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:30:38.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T05:30:38.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:39.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:39 smithi043 ceph-mon[27010]: Deploying daemon osd.0 on smithi043 2024-04-25T05:30:39.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:39 smithi043 ceph-mon[37492]: Deploying daemon osd.0 on smithi043 2024-04-25T05:30:39.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:39 smithi060 ceph-mon[30898]: Deploying daemon osd.0 on smithi043 2024-04-25T05:30:40.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:40 smithi043 ceph-mon[27010]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:40.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:40 smithi043 ceph-mon[37492]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:40.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:40 smithi060 ceph-mon[30898]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:42.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:42 smithi043 ceph-mon[27010]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:42 smithi043 ceph-mon[37492]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:42.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:42 smithi060 ceph-mon[30898]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:44.733 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[27010]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[37492]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:44 smithi060 ceph-mon[30898]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:46 smithi043 ceph-mon[27010]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:46 smithi043 ceph-mon[37492]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:46 smithi060 ceph-mon[30898]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:48.194 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 0 on host 'smithi043' 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:48.922 DEBUG:teuthology.orchestra.run.smithi043:osd.0> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T05:30:48.924 INFO:tasks.cephadm:Deploying osd.1 on smithi043 with /dev/vg_nvme/lv_3... 2024-04-25T05:30:48.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[27010]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[27010]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[27010]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[27010]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[37492]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[37492]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T05:30:49.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:49.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[37492]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:49.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-mon[37492]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:49.464 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0[41306]: 2024-04-25T05:30:49.170+0000 7fba0258a700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T05:30:49.464 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:30:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0[41306]: 2024-04-25T05:30:49.172+0000 7fb9fe37d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:30:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:49 smithi060 ceph-mon[30898]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T05:30:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:49 smithi060 ceph-mon[30898]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T05:30:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:49 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:49 smithi060 ceph-mon[30898]: from='osd.0 v2:172.21.15.43:6801/3557191248' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:49 smithi060 ceph-mon[30898]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:30:50.155 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:30:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[27010]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[27010]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:30:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[27010]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T05:30:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[37492]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[37492]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[37492]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:50 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:50 smithi060 ceph-mon[30898]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:50 smithi060 ceph-mon[30898]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:30:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:50 smithi060 ceph-mon[30898]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T05:30:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:50 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:50 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:30:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:30:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[27010]: osd.0 v2:172.21.15.43:6801/3557191248 boot 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[27010]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[37492]: osd.0 v2:172.21.15.43:6801/3557191248 boot 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[37492]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T05:30:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:51 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:51 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:30:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:51 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:30:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:51 smithi060 ceph-mon[30898]: osd.0 v2:172.21.15.43:6801/3557191248 boot 2024-04-25T05:30:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:51 smithi060 ceph-mon[30898]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T05:30:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:51 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:30:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:52 smithi060 ceph-mon[30898]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:52 smithi060 ceph-mon[30898]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T05:30:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:52 smithi043 ceph-mon[27010]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:52 smithi043 ceph-mon[27010]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T05:30:52.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:52 smithi043 ceph-mon[37492]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T05:30:52.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:52 smithi043 ceph-mon[37492]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi043 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 16069M 2024-04-25T05:30:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi043 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 16069M 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi043 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 16069M 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:30:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:30:54.565 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:30:54.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2024-04-25T05:30:54.857 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:30:55.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:55.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:55.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:30:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:30:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:30:56.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:56 smithi043 ceph-mon[37492]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:56.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:56 smithi043 ceph-mon[37492]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:56.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:56 smithi043 ceph-mon[27010]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:56 smithi043 ceph-mon[27010]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:56.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:56 smithi060 ceph-mon[30898]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:56 smithi060 ceph-mon[30898]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2513151472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]': finished 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2513151472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]': finished 2024-04-25T05:30:58.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T05:30:58.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2513151472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]: dispatch 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161"}]': finished 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T05:30:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:30:59.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:30:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/511030652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:30:59.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:30:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/511030652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:30:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:30:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/511030652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:00 smithi043 ceph-mon[27010]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:00 smithi043 ceph-mon[37492]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:00.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:00 smithi060 ceph-mon[30898]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:02.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:02 smithi043 ceph-mon[27010]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:02.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:02 smithi043 ceph-mon[37492]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:02 smithi060 ceph-mon[30898]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:04 smithi043 ceph-mon[27010]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:04 smithi043 ceph-mon[37492]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:04 smithi060 ceph-mon[30898]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:05.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:05 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T05:31:05.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:05 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:05 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T05:31:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:05 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:05 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T05:31:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:05 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:06 smithi043 ceph-mon[27010]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:06 smithi043 ceph-mon[27010]: Deploying daemon osd.1 on smithi043 2024-04-25T05:31:06.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:06 smithi043 ceph-mon[37492]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:06.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:06 smithi043 ceph-mon[37492]: Deploying daemon osd.1 on smithi043 2024-04-25T05:31:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:06 smithi060 ceph-mon[30898]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:06 smithi060 ceph-mon[30898]: Deploying daemon osd.1 on smithi043 2024-04-25T05:31:08.798 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:08 smithi043 ceph-mon[27010]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:08.798 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:08 smithi043 ceph-mon[37492]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:08 smithi060 ceph-mon[30898]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:10 smithi043 ceph-mon[37492]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:10 smithi043 ceph-mon[27010]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:10 smithi060 ceph-mon[30898]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:12.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[27010]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:12.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:12.065 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[37492]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:12.065 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.065 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.065 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:11 smithi060 ceph-mon[30898]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:13.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:13 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:13 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.684 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 1 on host 'smithi043' 2024-04-25T05:31:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[27010]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[37492]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:14.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:14 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:14 smithi060 ceph-mon[30898]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T05:31:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:14 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:14 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:14 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:15.348 DEBUG:teuthology.orchestra.run.smithi043:osd.1> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T05:31:15.351 INFO:tasks.cephadm:Deploying osd.2 on smithi043 with /dev/vg_nvme/lv_2... 2024-04-25T05:31:15.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-04-25T05:31:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[37492]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[37492]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[37492]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[37492]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[27010]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[27010]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[27010]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[27010]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:15 smithi060 ceph-mon[30898]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:15 smithi060 ceph-mon[30898]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T05:31:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:15 smithi060 ceph-mon[30898]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T05:31:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:15 smithi060 ceph-mon[30898]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:16.391 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1[47378]: 2024-04-25T05:31:16.086+0000 7f62fa225700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T05:31:16.391 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1[47378]: 2024-04-25T05:31:16.089+0000 7f62f2010700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:31:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:31:17.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:17.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T05:31:17.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' 2024-04-25T05:31:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:16 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T05:31:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: from='osd.1 v2:172.21.15.43:6805/235229295' entity='osd.1' 2024-04-25T05:31:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:16 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:18.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:31:18.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:31:18.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:18.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: osd.1 v2:172.21.15.43:6805/235229295 boot 2024-04-25T05:31:18.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: osd.1 v2:172.21.15.43:6805/235229295 boot 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T05:31:18.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:17 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: osd.1 v2:172.21.15.43:6805/235229295 boot 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T05:31:18.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:17 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:31:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi043 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 8034M 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:20.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi043 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 8034M 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:20.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi043 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 8034M 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:20.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:20.476 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:31:20.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2024-04-25T05:31:20.755 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[27010]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[27010]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:22.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[37492]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[37492]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:22.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:21 smithi060 ceph-mon[30898]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:22.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:21 smithi060 ceph-mon[30898]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:22.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:22.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:22.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:23 smithi043 ceph-mon[27010]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:23 smithi043 ceph-mon[37492]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:24.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:23 smithi060 ceph-mon[30898]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1454751192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]': finished 2024-04-25T05:31:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[27010]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T05:31:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1454751192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]': finished 2024-04-25T05:31:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[37492]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T05:31:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:25.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:24 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1454751192' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]: dispatch 2024-04-25T05:31:25.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088"}]': finished 2024-04-25T05:31:25.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:24 smithi060 ceph-mon[30898]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T05:31:25.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:26.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:25 smithi043 ceph-mon[27010]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:26.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1957340888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:26.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:25 smithi043 ceph-mon[37492]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:26.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1957340888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:26.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:25 smithi060 ceph-mon[30898]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1957340888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:28.030 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:27 smithi043 ceph-mon[27010]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:28.030 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:27 smithi043 ceph-mon[37492]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:28.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:27 smithi060 ceph-mon[30898]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:30.074 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:29 smithi043 ceph-mon[27010]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:30.075 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:29 smithi043 ceph-mon[37492]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:30.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:29 smithi060 ceph-mon[30898]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:32.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[37492]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:32.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T05:31:32.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[37492]: Deploying daemon osd.2 on smithi043 2024-04-25T05:31:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[27010]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T05:31:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:31 smithi043 ceph-mon[27010]: Deploying daemon osd.2 on smithi043 2024-04-25T05:31:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:31 smithi060 ceph-mon[30898]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T05:31:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:31 smithi060 ceph-mon[30898]: Deploying daemon osd.2 on smithi043 2024-04-25T05:31:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:33 smithi043 ceph-mon[27010]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:33 smithi043 ceph-mon[37492]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:34.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:33 smithi060 ceph-mon[30898]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:36.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:36 smithi043 ceph-mon[27010]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:36.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:36 smithi043 ceph-mon[37492]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:36.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:36 smithi060 ceph-mon[30898]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[27010]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[37492]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:38 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:38 smithi060 ceph-mon[30898]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:38 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:40 smithi060 ceph-mon[30898]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:40.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:40 smithi043 ceph-mon[37492]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:40.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:40 smithi043 ceph-mon[27010]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:40.688 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 2 on host 'smithi043' 2024-04-25T05:31:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[37492]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[27010]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.449 DEBUG:teuthology.orchestra.run.smithi043:osd.2> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T05:31:41.452 INFO:tasks.cephadm:Deploying osd.3 on smithi043 with /dev/vg_nvme/lv_1... 2024-04-25T05:31:41.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-04-25T05:31:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:41 smithi060 ceph-mon[30898]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T05:31:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[27010]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[27010]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[27010]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[27010]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:42.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[37492]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:42.386 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[37492]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T05:31:42.386 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[37492]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T05:31:42.386 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[37492]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:42.386 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:42 smithi060 ceph-mon[30898]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:42 smithi060 ceph-mon[30898]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T05:31:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:42 smithi060 ceph-mon[30898]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T05:31:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:42 smithi060 ceph-mon[30898]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:31:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:42 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:42.641 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2[53089]: 2024-04-25T05:31:42.444+0000 7fc498a18700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T05:31:42.641 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2[53089]: 2024-04-25T05:31:42.447+0000 7fc490803700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:31:42.691 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:31:43.484 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T05:31:43.485 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.486 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.486 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' 2024-04-25T05:31:43.486 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:43 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:31:43.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T05:31:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: from='osd.2 v2:172.21.15.43:6809/3416612512' entity='osd.2' 2024-04-25T05:31:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:43 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:44.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:31:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:31:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: osd.2 v2:172.21.15.43:6809/3416612512 boot 2024-04-25T05:31:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T05:31:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: osd.2 v2:172.21.15.43:6809/3416612512 boot 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T05:31:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: osd.2 v2:172.21.15.43:6809/3416612512 boot 2024-04-25T05:31:44.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T05:31:44.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:31:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:45 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:45.619 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:45.619 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:45 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:46.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: pgmap v75: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:46.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi043 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 5356M 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.322 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: pgmap v75: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi043 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.323 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 5356M 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: pgmap v75: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi043 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 5356M 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:31:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:46.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:31:46.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:46.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57149]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57149]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57084]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57084]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57084]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57090]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57090]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57090]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.575 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57090]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57095]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57095]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57095]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57095]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57100]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57100]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.576 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57100]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57103]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57110]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57110]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57110]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57075]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57075]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57075]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57075]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57078]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57078]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57078]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.578 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57078]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57149]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57196]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:46 smithi043 sudo[57196]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33456]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33459]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33459]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T05:31:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:46 smithi060 sudo[33459]: pam_unix(sudo:session): session closed for user root 2024-04-25T05:31:47.229 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:31:47.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:47 smithi060 ceph-mon[30898]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.503 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[27010]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:31:47.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:47 smithi043 ceph-mon[37492]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T05:31:48.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[37492]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:48.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[37492]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T05:31:48.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[37492]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-25T05:31:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[27010]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[27010]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T05:31:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:48 smithi043 ceph-mon[27010]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-25T05:31:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:48 smithi060 ceph-mon[30898]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:48 smithi060 ceph-mon[30898]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T05:31:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:48 smithi060 ceph-mon[30898]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-25T05:31:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:49 smithi060 ceph-mon[30898]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:49 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:49 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:49 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:49.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[27010]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:49.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:49.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:49.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:49.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[37492]: from='client.24197 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:31:49.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:31:49.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:31:49.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:49 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:50.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:50 smithi043 ceph-mon[27010]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:50.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:50 smithi043 ceph-mon[37492]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:50 smithi060 ceph-mon[30898]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]: dispatch 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]': finished 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[27010]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]: dispatch 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]': finished 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[37492]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T05:31:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:51 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:31:51.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]: dispatch 2024-04-25T05:31:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3639390617' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2"}]': finished 2024-04-25T05:31:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:51 smithi060 ceph-mon[30898]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T05:31:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:51 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:31:52.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:52 smithi043 ceph-mon[27010]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:52.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3939757104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:52.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:52 smithi043 ceph-mon[37492]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:52.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3939757104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:52.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:52 smithi060 ceph-mon[30898]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3939757104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:31:54.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:54 smithi043 ceph-mon[27010]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:54.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:54 smithi043 ceph-mon[37492]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:54.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:54 smithi060 ceph-mon[30898]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:56.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:56 smithi043 ceph-mon[27010]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:56.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:56 smithi043 ceph-mon[37492]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:56 smithi060 ceph-mon[30898]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:58.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[27010]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:58.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T05:31:58.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:58.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[37492]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:58.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T05:31:58.744 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:58.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:58 smithi060 ceph-mon[30898]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:31:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T05:31:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:31:59.865 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:31:59 smithi043 ceph-mon[27010]: Deploying daemon osd.3 on smithi043 2024-04-25T05:31:59.865 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:31:59 smithi043 ceph-mon[37492]: Deploying daemon osd.3 on smithi043 2024-04-25T05:31:59.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:31:59 smithi060 ceph-mon[30898]: Deploying daemon osd.3 on smithi043 2024-04-25T05:32:00.836 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:00 smithi043 ceph-mon[37492]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:00.836 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:00 smithi043 ceph-mon[27010]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:00.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:00 smithi060 ceph-mon[30898]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:02.778 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:02 smithi043 ceph-mon[27010]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:02.778 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:02 smithi043 ceph-mon[37492]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:02 smithi060 ceph-mon[30898]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:04.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:04.651 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:32:04.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:32:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:32:05.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:05 smithi060 ceph-mon[30898]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T05:32:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:05 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:05 smithi060 ceph-mon[30898]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[37492]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T05:32:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[37492]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[27010]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T05:32:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:06.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:05 smithi043 ceph-mon[27010]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:07.664 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 3 on host 'smithi043' 2024-04-25T05:32:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:07 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T05:32:08.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:08.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:07 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:08.361 DEBUG:teuthology.orchestra.run.smithi043:osd.3> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T05:32:08.364 INFO:tasks.cephadm:Deploying osd.4 on smithi060 with /dev/vg_nvme/lv_4... 2024-04-25T05:32:08.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-04-25T05:32:08.620 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:32:08.858 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[37492]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T05:32:08.859 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[37492]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T05:32:08.859 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:08.859 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[37492]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:08.859 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[37492]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:08.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[27010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T05:32:08.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[27010]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T05:32:08.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:08.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[27010]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:08.861 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-mon[27010]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:08.862 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:32:08.775+0000 7f8905efb700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T05:32:08.862 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:32:08 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:32:08.778+0000 7f8901cee700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:32:08.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:08 smithi060 ceph-mon[30898]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T05:32:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:08 smithi060 ceph-mon[30898]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T05:32:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:08 smithi060 ceph-mon[30898]: from='osd.3 v2:172.21.15.43:6813/3391766005' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:08 smithi060 ceph-mon[30898]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-04-25T05:32:09.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[37492]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:32:09.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[37492]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T05:32:09.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[37492]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[27010]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[27010]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:09 smithi043 ceph-mon[27010]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:09.950 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:09 smithi060 ceph-mon[30898]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-04-25T05:32:09.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:09 smithi060 ceph-mon[30898]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T05:32:09.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:09 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:09 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:09.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:09 smithi060 ceph-mon[30898]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[37492]: osd.3 v2:172.21.15.43:6813/3391766005 boot 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[37492]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[27010]: osd.3 v2:172.21.15.43:6813/3391766005 boot 2024-04-25T05:32:11.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[27010]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T05:32:11.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:11.133 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:10 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:32:11.133 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:10 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:32:11.133 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:10 smithi060 ceph-mon[30898]: osd.3 v2:172.21.15.43:6813/3391766005 boot 2024-04-25T05:32:11.133 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:10 smithi060 ceph-mon[30898]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T05:32:11.133 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:32:11.137 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:32:11.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_4 2024-04-25T05:32:11.425 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:32:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T05:32:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi043 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 4017M 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi043 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 4017M 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:12.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi043 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 4017M 2024-04-25T05:32:12.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:12.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:12.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:12 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:13.114 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:13.115 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:12 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:13.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:13 smithi060 ceph-mon[30898]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:14.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:13 smithi043 ceph-mon[27010]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:13 smithi043 ceph-mon[37492]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/1944875614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]': finished 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/3337920476' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/1944875614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]': finished 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:15.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/3337920476' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/1944875614' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]: dispatch 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c"}]': finished 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/3337920476' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:16.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:16 smithi043 ceph-mon[37492]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:16 smithi043 ceph-mon[27010]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:16 smithi060 ceph-mon[30898]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:18 smithi043 ceph-mon[27010]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:18 smithi043 ceph-mon[37492]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:18.892 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:18 smithi060 ceph-mon[30898]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:20.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:20 smithi060 ceph-mon[30898]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:20 smithi043 ceph-mon[27010]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:20.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:20 smithi043 ceph-mon[37492]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:21.893 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T05:32:21.894 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T05:32:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T05:32:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:22.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:22 smithi060 ceph-mon[30898]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:22.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:22 smithi060 ceph-mon[30898]: Deploying daemon osd.4 on smithi060 2024-04-25T05:32:23.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:22 smithi043 ceph-mon[27010]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:23.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:22 smithi043 ceph-mon[27010]: Deploying daemon osd.4 on smithi060 2024-04-25T05:32:23.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:22 smithi043 ceph-mon[37492]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:23.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:22 smithi043 ceph-mon[37492]: Deploying daemon osd.4 on smithi060 2024-04-25T05:32:24.970 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:24 smithi060 ceph-mon[30898]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:24 smithi043 ceph-mon[27010]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:25.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:24 smithi043 ceph-mon[37492]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:26 smithi060 ceph-mon[30898]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[27010]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[37492]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:28 smithi060 ceph-mon[30898]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:28 smithi043 ceph-mon[27010]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:29.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:28 smithi043 ceph-mon[37492]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:30.023 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 4 on host 'smithi060' 2024-04-25T05:32:30.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:30 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:30 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:30.675 DEBUG:teuthology.orchestra.run.smithi060:osd.4> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T05:32:30.677 INFO:tasks.cephadm:Deploying osd.5 on smithi060 with /dev/vg_nvme/lv_3... 2024-04-25T05:32:30.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-04-25T05:32:31.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-mon[30898]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T05:32:31.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-mon[30898]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:31.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-mon[30898]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T05:32:31.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:31.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-mon[30898]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:31.949 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:32:31.660+0000 7f368dcc3700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T05:32:31.949 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:32:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:32:31.667+0000 7f3689ab6700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:32:32.127 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:32:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[27010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T05:32:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[27010]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[27010]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T05:32:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[27010]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[37492]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T05:32:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[37492]: from='osd.4 v2:172.21.15.60:6800/1310286745' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[37492]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T05:32:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:31 smithi043 ceph-mon[37492]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:33.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:32 smithi060 ceph-mon[30898]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:33.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:32 smithi060 ceph-mon[30898]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:33.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:32 smithi060 ceph-mon[30898]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T05:32:33.001 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:32 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[27010]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[27010]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[27010]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T05:32:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[37492]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-25T05:32:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[37492]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[37492]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T05:32:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:32 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: osd.4 v2:172.21.15.60:6800/1310286745 boot 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:33 smithi060 ceph-mon[30898]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T05:32:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:32:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:32:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: osd.4 v2:172.21.15.60:6800/1310286745 boot 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[37492]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:32:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: osd.4 v2:172.21.15.60:6800/1310286745 boot 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:34.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:33 smithi043 ceph-mon[27010]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T05:32:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:34 smithi060 ceph-mon[30898]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:34 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi060 2024-04-25T05:32:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:34 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 17093M 2024-04-25T05:32:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:34 smithi060 ceph-mon[30898]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T05:32:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[27010]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi060 2024-04-25T05:32:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 17093M 2024-04-25T05:32:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[27010]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T05:32:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[37492]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi060 2024-04-25T05:32:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 17093M 2024-04-25T05:32:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:34 smithi043 ceph-mon[37492]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T05:32:35.304 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:32:35.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_3 2024-04-25T05:32:35.576 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:32:36.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:36 smithi060 ceph-mon[30898]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:36.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:36 smithi060 ceph-mon[30898]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T05:32:36.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:36 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:36.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:36 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:36.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:36 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[27010]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[27010]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[37492]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[37492]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:32:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:36 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:37.975 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:37 smithi060 ceph-mon[30898]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:37.975 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:37 smithi060 ceph-mon[30898]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T05:32:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:37 smithi043 ceph-mon[37492]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:37 smithi043 ceph-mon[37492]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T05:32:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:37 smithi043 ceph-mon[27010]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:32:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:37 smithi043 ceph-mon[27010]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T05:32:38.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:38 smithi060 ceph-mon[30898]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:38 smithi043 ceph-mon[27010]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:38 smithi043 ceph-mon[37492]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/3849958208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]': finished 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:39.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/2705445520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/3849958208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]': finished 2024-04-25T05:32:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T05:32:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/2705445520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/3849958208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]: dispatch 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2"}]': finished 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:40.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/2705445520' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:32:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:40 smithi060 ceph-mon[30898]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-25T05:32:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:40 smithi043 ceph-mon[27010]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-25T05:32:41.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:40 smithi043 ceph-mon[37492]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-25T05:32:42.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:42 smithi060 ceph-mon[30898]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:32:43.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:42 smithi043 ceph-mon[27010]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:32:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:42 smithi043 ceph-mon[37492]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:32:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:43 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:32:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:43 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:32:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:43 smithi060 ceph-mon[30898]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T05:32:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:43 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:32:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:32:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[27010]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T05:32:44.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:32:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:32:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[37492]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T05:32:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:43 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:44 smithi060 ceph-mon[30898]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-25T05:32:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:44 smithi043 ceph-mon[27010]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-25T05:32:45.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:44 smithi043 ceph-mon[37492]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-25T05:32:45.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:45 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T05:32:45.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:45 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:46.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T05:32:46.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:46.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:45 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T05:32:46.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:45 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:46.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:46 smithi060 ceph-mon[30898]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:32:46.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:46 smithi060 ceph-mon[30898]: Deploying daemon osd.5 on smithi060 2024-04-25T05:32:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:46 smithi043 ceph-mon[27010]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:32:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:46 smithi043 ceph-mon[27010]: Deploying daemon osd.5 on smithi060 2024-04-25T05:32:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:46 smithi043 ceph-mon[37492]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:32:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:46 smithi043 ceph-mon[37492]: Deploying daemon osd.5 on smithi060 2024-04-25T05:32:47.852 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:47 smithi060 ceph-mon[30898]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:47 smithi043 ceph-mon[27010]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:47 smithi043 ceph-mon[37492]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:49.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:49 smithi060 ceph-mon[30898]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:50.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:49 smithi043 ceph-mon[27010]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:50.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:49 smithi043 ceph-mon[37492]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:51 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:52.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:32:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:32:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:51 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:54.429 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 5 on host 'smithi060' 2024-04-25T05:32:54.741 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:54.741 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.742 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:54 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:54.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:54.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:54.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:54 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:32:55.052 DEBUG:teuthology.orchestra.run.smithi060:osd.5> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T05:32:55.054 INFO:tasks.cephadm:Deploying osd.6 on smithi060 with /dev/vg_nvme/lv_2... 2024-04-25T05:32:55.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-04-25T05:32:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-mon[30898]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T05:32:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-mon[30898]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-mon[30898]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T05:32:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-mon[30898]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:55.750 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:32:55.586+0000 7f65ddfec700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T05:32:55.750 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:32:55 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:32:55.592+0000 7f65d9ddf700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:32:55.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[27010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T05:32:55.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[27010]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[27010]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[27010]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[37492]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[37492]: from='osd.5 v2:172.21.15.60:6804/3586186657' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[37492]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:55 smithi043 ceph-mon[37492]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:32:56.671 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:32:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:56 smithi060 ceph-mon[30898]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:56 smithi060 ceph-mon[30898]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:56 smithi060 ceph-mon[30898]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T05:32:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:56 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:56 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:56.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[27010]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[27010]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[27010]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[37492]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-25T05:32:56.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[37492]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:32:56.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[37492]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T05:32:56.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:56.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:56 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:57.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:32:57.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:32:57.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: osd.5 v2:172.21.15.60:6804/3586186657 boot 2024-04-25T05:32:57.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[37492]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: osd.5 v2:172.21.15.60:6804/3586186657 boot 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:57.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:57 smithi043 ceph-mon[27010]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: osd.5 v2:172.21.15.60:6804/3586186657 boot 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:32:57.951 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:57 smithi060 ceph-mon[30898]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T05:32:58.664 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:32:58.664 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.664 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:58.665 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:58 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:32:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:32:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:58 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:32:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:32:59.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:59 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi060 2024-04-25T05:32:59.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:59 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 8546M 2024-04-25T05:32:59.799 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:32:59 smithi060 ceph-mon[30898]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T05:32:59.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_2 2024-04-25T05:32:59.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi060 2024-04-25T05:32:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 8546M 2024-04-25T05:32:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[27010]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T05:32:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi060 2024-04-25T05:32:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 8546M 2024-04-25T05:32:59.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:32:59 smithi043 ceph-mon[37492]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T05:33:00.050 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:00.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:00 smithi060 ceph-mon[30898]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:00 smithi060 ceph-mon[30898]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T05:33:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:00 smithi043 ceph-mon[27010]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:00.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:00 smithi043 ceph-mon[27010]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T05:33:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:00 smithi043 ceph-mon[37492]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:00 smithi043 ceph-mon[37492]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T05:33:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[27010]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T05:33:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:01.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[37492]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T05:33:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:01 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:01.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:01 smithi060 ceph-mon[30898]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T05:33:01.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:01 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:01.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:01 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:01.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:01 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:02.691 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:02 smithi060 ceph-mon[30898]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:02.691 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:02 smithi060 ceph-mon[30898]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:02.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:02 smithi043 ceph-mon[27010]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:02.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:02 smithi043 ceph-mon[27010]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:02.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:02 smithi043 ceph-mon[37492]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:02.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:02 smithi043 ceph-mon[37492]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/2040862454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]': finished 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[27010]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/2040862454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]': finished 2024-04-25T05:33:03.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[37492]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T05:33:03.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:03 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/2040862454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:03 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]: dispatch 2024-04-25T05:33:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:03 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa"}]': finished 2024-04-25T05:33:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:03 smithi060 ceph-mon[30898]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T05:33:03.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:03 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/1876779509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/1876779509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T05:33:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:04 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/1876779509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T05:33:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:04 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:06 smithi043 ceph-mon[27010]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-25T05:33:06.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:06 smithi043 ceph-mon[37492]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-25T05:33:06.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:06 smithi060 ceph-mon[30898]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-04-25T05:33:08.771 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:08 smithi060 ceph-mon[30898]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:08.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:08 smithi043 ceph-mon[27010]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:08 smithi043 ceph-mon[37492]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:10 smithi060 ceph-mon[30898]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T05:33:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[27010]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T05:33:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[37492]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T05:33:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:11.707 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:11 smithi060 ceph-mon[30898]: Deploying daemon osd.6 on smithi060 2024-04-25T05:33:11.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:11 smithi043 ceph-mon[27010]: Deploying daemon osd.6 on smithi060 2024-04-25T05:33:11.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:11 smithi043 ceph-mon[37492]: Deploying daemon osd.6 on smithi060 2024-04-25T05:33:12.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:12 smithi060 ceph-mon[30898]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:12.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:12 smithi043 ceph-mon[27010]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:12.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:12 smithi043 ceph-mon[37492]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[27010]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[37492]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:14 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:14.963 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:14 smithi060 ceph-mon[30898]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:14.964 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:14 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:14.964 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:14 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:15.969 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:15 smithi060 ceph-mon[30898]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T05:33:15.969 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:15.969 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:15.969 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:15.969 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:15 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[27010]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T05:33:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[37492]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T05:33:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:15 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:16.975 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:16 smithi060 ceph-mon[30898]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:17.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:16 smithi043 ceph-mon[27010]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:17.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:16 smithi043 ceph-mon[37492]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:18 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:18 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.337 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 6 on host 'smithi060' 2024-04-25T05:33:19.895 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:19 smithi060 ceph-mon[30898]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:19.896 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:19 smithi060 ceph-mon[30898]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:19.896 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.896 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:19.896 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:19 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:19.923 DEBUG:teuthology.orchestra.run.smithi060:osd.6> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T05:33:19.925 INFO:tasks.cephadm:Deploying osd.7 on smithi060 with /dev/vg_nvme/lv_1... 2024-04-25T05:33:19.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-04-25T05:33:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[27010]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[27010]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:20.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:20.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[37492]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:20.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[37492]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T05:33:20.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:20.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:20.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:19 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:20.949 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-mon[30898]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T05:33:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T05:33:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[27010]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: from='osd.6 v2:172.21.15.60:6808/1675137598' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:20 smithi043 ceph-mon[37492]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:21.154 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:21.228 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:33:20.947+0000 7f51d7dbf700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T05:33:21.229 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:33:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:33:20.953+0000 7f51cfbaa700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:33:21.818 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:21 smithi060 ceph-mon[30898]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:21.818 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:21 smithi060 ceph-mon[30898]: osdmap e49: 7 total, 6 up, 7 in 2024-04-25T05:33:21.818 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:21.818 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:21 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:21.818 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:21 smithi060 ceph-mon[30898]: from='osd.6 ' entity='osd.6' 2024-04-25T05:33:22.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[37492]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:22.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[37492]: osdmap e49: 7 total, 6 up, 7 in 2024-04-25T05:33:22.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[37492]: from='osd.6 ' entity='osd.6' 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[27010]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[27010]: osdmap e49: 7 total, 6 up, 7 in 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:22.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:21 smithi043 ceph-mon[27010]: from='osd.6 ' entity='osd.6' 2024-04-25T05:33:22.972 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: osd.6 v2:172.21.15.60:6808/1675137598 boot 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T05:33:22.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:22 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:23.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:23.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:23.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: osd.6 v2:172.21.15.60:6808/1675137598 boot 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: osd.6 v2:172.21.15.60:6808/1675137598 boot 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T05:33:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:22 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:33:23.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: osdmap e51: 7 total, 7 up, 7 in 2024-04-25T05:33:23.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: osdmap e51: 7 total, 7 up, 7 in 2024-04-25T05:33:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: osdmap e51: 7 total, 7 up, 7 in 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:24.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:24.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:24.908 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:24 smithi060 ceph-mon[30898]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:24.908 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:24 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi060 2024-04-25T05:33:24.908 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:24 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 5697M 2024-04-25T05:33:24.908 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:24 smithi060 ceph-mon[30898]: osdmap e52: 7 total, 7 up, 7 in 2024-04-25T05:33:24.908 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:24 smithi060 ceph-mon[30898]: osdmap e53: 7 total, 7 up, 7 in 2024-04-25T05:33:24.911 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-04-25T05:33:24.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi060:vg_nvme/lv_1 2024-04-25T05:33:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[27010]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi060 2024-04-25T05:33:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 5697M 2024-04-25T05:33:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[27010]: osdmap e52: 7 total, 7 up, 7 in 2024-04-25T05:33:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[27010]: osdmap e53: 7 total, 7 up, 7 in 2024-04-25T05:33:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[37492]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi060 2024-04-25T05:33:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 5697M 2024-04-25T05:33:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[37492]: osdmap e52: 7 total, 7 up, 7 in 2024-04-25T05:33:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:24 smithi043 ceph-mon[37492]: osdmap e53: 7 total, 7 up, 7 in 2024-04-25T05:33:25.182 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:26.856 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:26 smithi060 ceph-mon[30898]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:26.857 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:26 smithi060 ceph-mon[30898]: osdmap e54: 7 total, 7 up, 7 in 2024-04-25T05:33:26.857 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:26.857 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:26.857 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:26 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[27010]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[27010]: osdmap e54: 7 total, 7 up, 7 in 2024-04-25T05:33:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[37492]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[37492]: osdmap e54: 7 total, 7 up, 7 in 2024-04-25T05:33:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T05:33:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T05:33:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:26 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:27.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:27 smithi060 ceph-mon[30898]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:28.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:27 smithi043 ceph-mon[27010]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:28.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:27 smithi043 ceph-mon[37492]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:28.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: osdmap e55: 7 total, 7 up, 7 in 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/3777909438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]': finished 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: osdmap e56: 8 total, 7 up, 8 in 2024-04-25T05:33:28.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:28 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: osdmap e55: 7 total, 7 up, 7 in 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/3777909438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]': finished 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: osdmap e56: 8 total, 7 up, 8 in 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: osdmap e55: 7 total, 7 up, 7 in 2024-04-25T05:33:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/3777909438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]: dispatch 2024-04-25T05:33:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2"}]': finished 2024-04-25T05:33:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: osdmap e56: 8 total, 7 up, 8 in 2024-04-25T05:33:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:28 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:29.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/266077766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:30.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/266077766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/266077766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T05:33:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:30 smithi060 ceph-mon[30898]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-04-25T05:33:31.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:30 smithi043 ceph-mon[27010]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-04-25T05:33:31.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:30 smithi043 ceph-mon[37492]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-04-25T05:33:32.929 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:32 smithi060 ceph-mon[30898]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:32 smithi043 ceph-mon[27010]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:33.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:32 smithi043 ceph-mon[37492]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:34.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:34 smithi060 ceph-mon[30898]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:33:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:34 smithi043 ceph-mon[27010]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:33:35.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:34 smithi043 ceph-mon[37492]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:33:35.968 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:35 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T05:33:35.968 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:35 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:36.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:35 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T05:33:36.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:35 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:36.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:35 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T05:33:36.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:35 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:36.912 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:36 smithi060 ceph-mon[30898]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:36.912 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:36 smithi060 ceph-mon[30898]: Deploying daemon osd.7 on smithi060 2024-04-25T05:33:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:36 smithi043 ceph-mon[27010]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:36 smithi043 ceph-mon[27010]: Deploying daemon osd.7 on smithi060 2024-04-25T05:33:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:36 smithi043 ceph-mon[37492]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T05:33:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:36 smithi043 ceph-mon[37492]: Deploying daemon osd.7 on smithi060 2024-04-25T05:33:38.973 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:38 smithi060 ceph-mon[30898]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T05:33:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:38 smithi043 ceph-mon[27010]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T05:33:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:38 smithi043 ceph-mon[37492]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-25T05:33:40.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:40 smithi060 ceph-mon[30898]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:40 smithi043 ceph-mon[27010]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:40 smithi043 ceph-mon[37492]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:41 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:42.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:42.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:42.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:42.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:42.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:42.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:41 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:42.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:42 smithi060 ceph-mon[30898]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:43.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:42 smithi043 ceph-mon[27010]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:43.055 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:42 smithi043 ceph-mon[37492]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:44.587 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 7 on host 'smithi060' 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:44.913 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:44.914 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:44.914 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:44.914 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:44.914 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:44 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:45.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:44 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:45.214 DEBUG:teuthology.orchestra.run.smithi060:osd.7> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T05:33:45.216 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-25T05:33:45.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T05:33:45.477 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:33:45.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[37492]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T05:33:45.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[37492]: osdmap e57: 8 total, 7 up, 8 in 2024-04-25T05:33:45.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:45.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[37492]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[37492]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[27010]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T05:33:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[27010]: osdmap e57: 8 total, 7 up, 8 in 2024-04-25T05:33:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[27010]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:45 smithi043 ceph-mon[27010]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-mon[30898]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T05:33:45.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-mon[30898]: osdmap e57: 8 total, 7 up, 8 in 2024-04-25T05:33:45.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:45.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-mon[30898]: from='osd.7 v2:172.21.15.60:6812/3982012700' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-mon[30898]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-04-25T05:33:45.999 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:33:45.700+0000 7f2b267b4700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T05:33:46.000 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:33:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:33:45.708+0000 7f2b225a7700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T05:33:46.213 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:33:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: osdmap e58: 8 total, 7 up, 8 in 2024-04-25T05:33:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/769994458' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: osdmap e58: 8 total, 7 up, 8 in 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/769994458' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: osdmap e58: 8 total, 7 up, 8 in 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/769994458' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:47.059 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":58,"num_osds":8,"num_up_osds":7,"osd_up_since":1714023201,"num_in_osds":8,"osd_in_since":1714023208,"num_remapped_pgs":0} 2024-04-25T05:33:47.826 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:47 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:47.826 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:47 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:47.826 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:47 smithi060 ceph-mon[30898]: osd.7 v2:172.21.15.60:6812/3982012700 boot 2024-04-25T05:33:47.826 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:47 smithi060 ceph-mon[30898]: osdmap e59: 8 total, 8 up, 8 in 2024-04-25T05:33:47.826 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:47 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:48.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[27010]: osd.7 v2:172.21.15.60:6812/3982012700 boot 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[27010]: osdmap e59: 8 total, 8 up, 8 in 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:48.089 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:48.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[37492]: osd.7 v2:172.21.15.60:6812/3982012700 boot 2024-04-25T05:33:48.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[37492]: osdmap e59: 8 total, 8 up, 8 in 2024-04-25T05:33:48.090 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:47 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:33:48.326 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: osdmap e60: 8 total, 8 up, 8 in 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:48 smithi060 ceph-mon[30898]: osdmap e61: 8 total, 8 up, 8 in 2024-04-25T05:33:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:33:49.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:49.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: osdmap e60: 8 total, 8 up, 8 in 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[37492]: osdmap e61: 8 total, 8 up, 8 in 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T05:33:49.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: osdmap e60: 8 total, 8 up, 8 in 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:49.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:48 smithi043 ceph-mon[27010]: osdmap e61: 8 total, 8 up, 8 in 2024-04-25T05:33:49.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[27010]: Detected new or changed devices on smithi060 2024-04-25T05:33:49.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 4273M 2024-04-25T05:33:49.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/44749991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:49.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[27010]: osdmap e62: 8 total, 8 up, 8 in 2024-04-25T05:33:49.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[37492]: Detected new or changed devices on smithi060 2024-04-25T05:33:49.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 4273M 2024-04-25T05:33:49.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/44749991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:49.847 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:49 smithi043 ceph-mon[37492]: osdmap e62: 8 total, 8 up, 8 in 2024-04-25T05:33:49.875 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1714023226,"num_in_osds":8,"osd_in_since":1714023208,"num_remapped_pgs":1} 2024-04-25T05:33:49.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T05:33:49.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:49 smithi060 ceph-mon[30898]: Detected new or changed devices on smithi060 2024-04-25T05:33:49.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:49 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 4273M 2024-04-25T05:33:49.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/44749991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T05:33:49.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:49 smithi060 ceph-mon[30898]: osdmap e62: 8 total, 8 up, 8 in 2024-04-25T05:33:50.162 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:33:50.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:50 smithi043 ceph-mon[27010]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:50.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:50 smithi043 ceph-mon[27010]: osdmap e63: 8 total, 8 up, 8 in 2024-04-25T05:33:50.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:50 smithi043 ceph-mon[37492]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:50.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:50 smithi043 ceph-mon[37492]: osdmap e63: 8 total, 8 up, 8 in 2024-04-25T05:33:50.921 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:33:50.921 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":63,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","created":"2024-04-25T05:28:34.192515+0000","modified":"2024-04-25T05:33:50.648363+0000","last_up_change":"2024-04-25T05:33:46.700068+0000","last_in_change":"2024-04-25T05:33:28.520304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T05:31:44.863613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef370ce4-85d8-41ed-9655-2e2eaab3b48e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":3557191248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3557191248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3557191248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":3557191248}]},"public_addr":"172.21.15.43:6801/3557191248","cluster_addr":"172.21.15.43:6802/3557191248","heartbeat_back_addr":"172.21.15.43:6804/3557191248","heartbeat_front_addr":"172.21.15.43:6803/3557191248","state":["exists","up"]},{"osd":1,"uuid":"ed504d08-dc59-4584-bd5f-8059e02a4161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":235229295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":235229295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":235229295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":235229295}]},"public_addr":"172.21.15.43:6805/235229295","cluster_addr":"172.21.15.43:6806/235229295","heartbeat_back_addr":"172.21.15.43:6808/235229295","heartbeat_front_addr":"172.21.15.43:6807/235229295","state":["exists","up"]},{"osd":2,"uuid":"11ff9a46-8686-4b36-a530-5cc1dca1f088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":3416612512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3416612512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3416612512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3416612512}]},"public_addr":"172.21.15.43:6809/3416612512","cluster_addr":"172.21.15.43:6810/3416612512","heartbeat_back_addr":"172.21.15.43:6812/3416612512","heartbeat_front_addr":"172.21.15.43:6811/3416612512","state":["exists","up"]},{"osd":3,"uuid":"f3314c45-11c4-46ee-be0b-6f639faa7da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":3391766005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3391766005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3391766005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6815","nonce":3391766005}]},"public_addr":"172.21.15.43:6813/3391766005","cluster_addr":"172.21.15.43:6814/3391766005","heartbeat_back_addr":"172.21.15.43:6816/3391766005","heartbeat_front_addr":"172.21.15.43:6815/3391766005","state":["exists","up"]},{"osd":4,"uuid":"8c5337b2-94d1-4784-9321-db771de75f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1310286745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6801","nonce":1310286745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1310286745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1310286745}]},"public_addr":"172.21.15.60:6800/1310286745","cluster_addr":"172.21.15.60:6801/1310286745","heartbeat_back_addr":"172.21.15.60:6803/1310286745","heartbeat_front_addr":"172.21.15.60:6802/1310286745","state":["exists","up"]},{"osd":5,"uuid":"fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3586186657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3586186657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":3586186657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3586186657}]},"public_addr":"172.21.15.60:6804/3586186657","cluster_addr":"172.21.15.60:6805/3586186657","heartbeat_back_addr":"172.21.15.60:6807/3586186657","heartbeat_front_addr":"172.21.15.60:6806/3586186657","state":["exists","up"]},{"osd":6,"uuid":"5b946188-0dd1-4e68-bcfc-a3daa342edaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1675137598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":1675137598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1675137598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1675137598}]},"public_addr":"172.21.15.60:6808/1675137598","cluster_addr":"172.21.15.60:6809/1675137598","heartbeat_back_addr":"172.21.15.60:6811/1675137598","heartbeat_front_addr":"172.21.15.60:6810/1675137598","state":["exists","up"]},{"osd":7,"uuid":"dec84bb8-9d2b-469f-9641-62ff40b9b9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3982012700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":3982012700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":3982012700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3982012700}]},"public_addr":"172.21.15.60:6812/3982012700","cluster_addr":"172.21.15.60:6813/3982012700","heartbeat_back_addr":"172.21.15.60:6815/3982012700","heartbeat_front_addr":"172.21.15.60:6814/3982012700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:14.212622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:32:42.395465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:03.539581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:27.046046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:32:30.665378+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:32:55.038764+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:19.709604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:44.960822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/4077364595":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/830746728":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/786479021":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1835545567":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1408327650":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/2239747746":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/4084484879":"2024-04-26T05:29:30.781325+0000","172.21.15.43:0/1247294293":"2024-04-26T05:29:30.781325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T05:33:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:50 smithi060 ceph-mon[30898]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:50 smithi060 ceph-mon[30898]: osdmap e63: 8 total, 8 up, 8 in 2024-04-25T05:33:51.651 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T05:33:51.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T05:33:51.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1718466893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:33:51.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1718466893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:33:51.902 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:33:51.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1718466893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:33:52.659 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-04-25T05:33:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:52 smithi060 ceph-mon[30898]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2380089201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T05:33:53.058 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:52 smithi043 ceph-mon[27010]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:53.058 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2380089201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T05:33:53.059 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:52 smithi043 ceph-mon[37492]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:33:53.059 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2380089201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T05:33:53.398 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi043 2024-04-25T05:33:53.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi043=foo.a' 2024-04-25T05:33:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:53.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:53 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:53.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:53 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:53.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:53 smithi060 ceph-mon[30898]: osdmap e64: 8 total, 8 up, 8 in 2024-04-25T05:33:53.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:53 smithi060 ceph-mon[30898]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-25T05:33:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[27010]: osdmap e64: 8 total, 8 up, 8 in 2024-04-25T05:33:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[27010]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-25T05:33:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[37492]: osdmap e64: 8 total, 8 up, 8 in 2024-04-25T05:33:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:53 smithi043 ceph-mon[37492]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-04-25T05:33:54.407 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled rgw.foo.a update... 2024-04-25T05:33:55.096 DEBUG:teuthology.orchestra.run.smithi043:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@rgw.foo.a.service 2024-04-25T05:33:55.099 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi060 2024-04-25T05:33:55.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-04-25T05:33:55.344 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.714 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.715 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:55 smithi060 ceph-mon[30898]: Deploying daemon rgw.foo.a on smithi043 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-25T05:33:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[27010]: Deploying daemon rgw.foo.a on smithi043 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi043=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-25T05:33:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:55.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:33:55.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:55 smithi043 ceph-mon[37492]: Deploying daemon rgw.foo.a on smithi043 2024-04-25T05:33:56.663 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'datapool' created 2024-04-25T05:33:56.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[27010]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/2353234145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:56.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:56.892 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:56 smithi043 systemd[1]: Starting Ceph rgw.foo.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:33:56.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[37492]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:56.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/2353234145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:56.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:56.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:56 smithi060 ceph-mon[30898]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T05:33:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/2353234145' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-25T05:33:57.183 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:56 smithi043 podman[64022]: 2024-04-25 05:33:56.832231647 +0000 UTC m=+0.018777374 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:33:57.183 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:56 smithi043 podman[64022]: 2024-04-25 05:33:56.930367918 +0000 UTC m=+0.116913702 container create f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T05:33:57.183 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:57 smithi043 podman[64022]: 2024-04-25 05:33:57.183116565 +0000 UTC m=+0.369662280 container init f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T05:33:57.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- rbd pool init datapool 2024-04-25T05:33:57.548 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:33:57.641 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:57 smithi043 podman[64022]: 2024-04-25 05:33:57.186147271 +0000 UTC m=+0.372692986 container start f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T05:33:57.641 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:57 smithi043 bash[64022]: f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 2024-04-25T05:33:57.641 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:33:57 smithi043 systemd[1]: Started Ceph rgw.foo.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:33:57.838 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:33:57.838 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:33:57.838 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: osdmap e65: 8 total, 8 up, 8 in 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:57.839 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:57 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:58.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:33:58.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: osdmap e65: 8 total, 8 up, 8 in 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:33:58.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: osdmap e65: 8 total, 8 up, 8 in 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:58.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:57 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: pgmap v188: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: osdmap e66: 8 total, 8 up, 8 in 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/1225883007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:59.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: pgmap v188: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:59.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:59.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: osdmap e66: 8 total, 8 up, 8 in 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/1225883007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: pgmap v188: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: Saving service rgw.foo.a spec with placement smithi043=foo.a;count:1 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:33:59.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: osdmap e66: 8 total, 8 up, 8 in 2024-04-25T05:33:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-25T05:33:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/1225883007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:59.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-25T05:33:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-25T05:33:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-25T05:33:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: osdmap e67: 8 total, 8 up, 8 in 2024-04-25T05:33:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:33:59.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:00.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:00.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:33:59 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:00.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-25T05:34:00.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-25T05:34:00.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: osdmap e67: 8 total, 8 up, 8 in 2024-04-25T05:34:00.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/112953228' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: osdmap e67: 8 total, 8 up, 8 in 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:00.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:33:59 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: pgmap v191: 36 pgs: 15 creating+peering, 19 unknown, 2 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: Checking dashboard <-> RGW credentials 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: osdmap e68: 8 total, 8 up, 8 in 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:00.991 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: pgmap v191: 36 pgs: 15 creating+peering, 19 unknown, 2 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: Checking dashboard <-> RGW credentials 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: osdmap e68: 8 total, 8 up, 8 in 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:01.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: pgmap v191: 36 pgs: 15 creating+peering, 19 unknown, 2 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: Checking dashboard <-> RGW credentials 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: osdmap e68: 8 total, 8 up, 8 in 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-25T05:34:01.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.60 --placement '1;smithi060=iscsi.a' 2024-04-25T05:34:01.511 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:01.952 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:01.952 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:01.953 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:01 smithi060 ceph-mon[30898]: osdmap e69: 8 total, 8 up, 8 in 2024-04-25T05:34:01.953 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:01 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[27010]: osdmap e69: 8 total, 8 up, 8 in 2024-04-25T05:34:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-25T05:34:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[37492]: osdmap e69: 8 total, 8 up, 8 in 2024-04-25T05:34:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:01 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:02.230 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled iscsi.datapool update... 2024-04-25T05:34:02.814 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: pgmap v194: 68 pgs: 22 creating+peering, 37 unknown, 9 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: osdmap e70: 8 total, 8 up, 8 in 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:02.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:02 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:02.841 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-25T05:34:02.841 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:34:02.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-25T05:34:02.870 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:34:02.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-25T05:34:02.896 DEBUG:teuthology.orchestra.run.smithi060:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@iscsi.iscsi.a.service 2024-04-25T05:34:02.938 INFO:tasks.cephadm:Adding prometheus.a on smithi060 2024-04-25T05:34:02.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi060=a' 2024-04-25T05:34:03.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: pgmap v194: 68 pgs: 22 creating+peering, 37 unknown, 9 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:03.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:34:03.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: osdmap e70: 8 total, 8 up, 8 in 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: pgmap v194: 68 pgs: 22 creating+peering, 37 unknown, 9 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: osdmap e70: 8 total, 8 up, 8 in 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-25T05:34:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:02 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:03.218 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:03.936 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled prometheus update... 2024-04-25T05:34:03.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:03 smithi060 ceph-mon[30898]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.60", "placement": "1;smithi060=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:03.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:03 smithi060 ceph-mon[30898]: Saving service iscsi.datapool spec with placement smithi060=iscsi.a;count:1 2024-04-25T05:34:03.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:03.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:03.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:03 smithi060 ceph-mon[30898]: osdmap e71: 8 total, 8 up, 8 in 2024-04-25T05:34:04.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[37492]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.60", "placement": "1;smithi060=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:04.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[37492]: Saving service iscsi.datapool spec with placement smithi060=iscsi.a;count:1 2024-04-25T05:34:04.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:04.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[37492]: osdmap e71: 8 total, 8 up, 8 in 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[27010]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.60", "placement": "1;smithi060=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[27010]: Saving service iscsi.datapool spec with placement smithi060=iscsi.a;count:1 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-25T05:34:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:03 smithi043 ceph-mon[27010]: osdmap e71: 8 total, 8 up, 8 in 2024-04-25T05:34:04.509 DEBUG:teuthology.orchestra.run.smithi060:prometheus.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-25T05:34:04.511 INFO:tasks.cephadm:Adding node-exporter.a on smithi043 2024-04-25T05:34:04.511 INFO:tasks.cephadm:Adding node-exporter.b on smithi060 2024-04-25T05:34:04.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi043=a;smithi060=b' 2024-04-25T05:34:04.804 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:04 smithi060 ceph-mon[30898]: pgmap v197: 100 pgs: 17 creating+peering, 27 unknown, 56 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T05:34:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:04 smithi060 ceph-mon[30898]: osdmap e72: 8 total, 8 up, 8 in 2024-04-25T05:34:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:04 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:05.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[37492]: pgmap v197: 100 pgs: 17 creating+peering, 27 unknown, 56 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T05:34:05.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[37492]: osdmap e72: 8 total, 8 up, 8 in 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[27010]: pgmap v197: 100 pgs: 17 creating+peering, 27 unknown, 56 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[27010]: osdmap e72: 8 total, 8 up, 8 in 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-25T05:34:05.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:04 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:05.546 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled node-exporter update... 2024-04-25T05:34:05.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:05.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: Saving service prometheus spec with placement smithi060=a;count:1 2024-04-25T05:34:05.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:05.937 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:05.938 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: osdmap e73: 8 total, 8 up, 8 in 2024-04-25T05:34:05.938 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:05.938 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:05.938 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:05 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.141 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a[64036]: 2024-04-25T05:34:05.774+0000 7eff9ac15ac0 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-25T05:34:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: Saving service prometheus spec with placement smithi060=a;count:1 2024-04-25T05:34:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: osdmap e73: 8 total, 8 up, 8 in 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: Saving service prometheus spec with placement smithi060=a;count:1 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: osdmap e73: 8 total, 8 up, 8 in 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-25T05:34:06.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:05 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.258 DEBUG:teuthology.orchestra.run.smithi043:node-exporter.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-25T05:34:06.261 DEBUG:teuthology.orchestra.run.smithi060:node-exporter.b> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-25T05:34:06.263 INFO:tasks.cephadm:Adding alertmanager.a on smithi043 2024-04-25T05:34:06.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi043=a' 2024-04-25T05:34:06.564 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: pgmap v200: 132 pgs: 9 creating+peering, 31 unknown, 92 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi060=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: Saving service node-exporter spec with placement smithi043=a;smithi060=b;count:2 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: osdmap e74: 8 total, 8 up, 8 in 2024-04-25T05:34:06.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.943 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.943 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:06.943 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-25T05:34:06.943 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-25T05:34:06.943 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:06 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: pgmap v200: 132 pgs: 9 creating+peering, 31 unknown, 92 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi060=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: Saving service node-exporter spec with placement smithi043=a;smithi060=b;count:2 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: osdmap e74: 8 total, 8 up, 8 in 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-25T05:34:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: pgmap v200: 132 pgs: 9 creating+peering, 31 unknown, 92 active+clean; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi060=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: Saving service node-exporter spec with placement smithi043=a;smithi060=b;count:2 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/957152661' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/312817651' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: osdmap e74: 8 total, 8 up, 8 in 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-25T05:34:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:06 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:07 smithi060 ceph-mon[30898]: Deploying daemon iscsi.iscsi.a on smithi060 2024-04-25T05:34:08.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:07 smithi043 ceph-mon[27010]: Deploying daemon iscsi.iscsi.a on smithi060 2024-04-25T05:34:08.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:07 smithi043 ceph-mon[37492]: Deploying daemon iscsi.iscsi.a on smithi060 2024-04-25T05:34:08.308 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled alertmanager update... 2024-04-25T05:34:08.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:08 smithi060 systemd[1]: Starting Ceph iscsi.iscsi.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:34:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:08 smithi060 ceph-mon[30898]: pgmap v202: 132 pgs: 9 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-04-25T05:34:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:08 smithi060 ceph-mon[30898]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:08 smithi060 ceph-mon[30898]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-04-25T05:34:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:08 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:09.066 DEBUG:teuthology.orchestra.run.smithi043:alertmanager.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-25T05:34:09.069 INFO:tasks.cephadm:Adding grafana.a on smithi060 2024-04-25T05:34:09.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi060=a' 2024-04-25T05:34:09.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[27010]: pgmap v202: 132 pgs: 9 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-04-25T05:34:09.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[27010]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[27010]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[37492]: pgmap v202: 132 pgs: 9 creating+peering, 14 unknown, 109 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[37492]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[37492]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-04-25T05:34:09.093 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:08 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:09.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 podman[58000]: 2024-04-25 05:34:09.223284524 +0000 UTC m=+0.542053911 container create 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T05:34:09.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 podman[58000]: 2024-04-25 05:34:09.131340485 +0000 UTC m=+0.450109890 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:34:09.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 podman[58000]: 2024-04-25 05:34:09.458836787 +0000 UTC m=+0.777606178 container init 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T05:34:09.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 podman[58000]: 2024-04-25 05:34:09.462042674 +0000 UTC m=+0.780812054 container start 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T05:34:09.803 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:09.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Started the configuration object watcher 2024-04-25T05:34:09.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Checking for config object changes every 1s 2024-04-25T05:34:09.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Processing osd blocklist entries for this node 2024-04-25T05:34:09.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 bash[58000]: 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 2024-04-25T05:34:09.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:09 smithi060 systemd[1]: Started Ceph iscsi.iscsi.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:34:10.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Reading the configuration object to update local LIO configuration 2024-04-25T05:34:10.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Configuration does not have an entry for this host(smithi060) - nothing to define to LIO 2024-04-25T05:34:10.372 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-25T05:34:10.567 INFO:teuthology.orchestra.run.smithi060.stdout:Scheduled grafana update... 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: pgmap v203: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: osdmap e75: 8 total, 8 up, 8 in 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/175073876' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T05:34:11.002 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:10 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: pgmap v203: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-04-25T05:34:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: osdmap e75: 8 total, 8 up, 8 in 2024-04-25T05:34:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/175073876' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: pgmap v203: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 5.0 KiB/s wr, 197 op/s 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: osdmap e75: 8 total, 8 up, 8 in 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/175073876' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T05:34:11.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:10 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:11.310 DEBUG:teuthology.orchestra.run.smithi060:grafana.a> sudo journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@grafana.a.service 2024-04-25T05:34:11.312 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T05:34:11.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T05:34:11.569 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:11.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:11 smithi060 ceph-mon[30898]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-25T05:34:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:11 smithi060 ceph-mon[30898]: Deploying daemon prometheus.a on smithi060 2024-04-25T05:34:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:11 smithi060 ceph-mon[30898]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:11 smithi060 ceph-mon[30898]: Saving service grafana spec with placement smithi060=a;count:1 2024-04-25T05:34:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:11 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[27010]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[27010]: Deploying daemon prometheus.a on smithi060 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[27010]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[27010]: Saving service grafana spec with placement smithi060=a;count:1 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[37492]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[37492]: Deploying daemon prometheus.a on smithi060 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[37492]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi060=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T05:34:12.023 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[37492]: Saving service grafana spec with placement smithi060=a;count:1 2024-04-25T05:34:12.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:11 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:12.442 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-25T05:34:12.442 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBU6ylmvukjGhAA/vvQNJ5ZdMum8WJVWOxnMQ== 2024-04-25T05:34:12.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[27010]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 5.7 KiB/s wr, 261 op/s 2024-04-25T05:34:12.742 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[27010]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2395665566' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[37492]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 5.7 KiB/s wr, 261 op/s 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[37492]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2395665566' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.743 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:12 smithi060 ceph-mon[30898]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 5.7 KiB/s wr, 261 op/s 2024-04-25T05:34:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:12 smithi060 ceph-mon[30898]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-25T05:34:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2395665566' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:13.229 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-25T05:34:13.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T05:34:13.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T05:34:13.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T05:34:13.512 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:34:14.565 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2024-04-25T05:34:14.565 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBW6ylmdVheIRAA7D89ZKuZ6FNn5fW9Kjk3Mw== 2024-04-25T05:34:14.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:14 smithi060 ceph-mon[30898]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 201 op/s 2024-04-25T05:34:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.60:0/3877312776' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:15.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[27010]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 201 op/s 2024-04-25T05:34:15.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.60:0/3877312776' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[37492]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.5 KiB/s wr, 201 op/s 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.60:0/3877312776' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T05:34:15.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T05:34:15.756 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-04-25T05:34:15.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-25T05:34:15.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T05:34:15.793 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T05:34:15.793 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T05:34:15.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T05:34:16.053 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:16.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:16 smithi043 ceph-mon[27010]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 4.0 KiB/s wr, 177 op/s 2024-04-25T05:34:16.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:16 smithi043 ceph-mon[37492]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 4.0 KiB/s wr, 177 op/s 2024-04-25T05:34:16.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:16 smithi060 ceph-mon[30898]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 4.0 KiB/s wr, 177 op/s 2024-04-25T05:34:17.727 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":149142905}]},"active_addr":"172.21.15.43:6800/149142905","active_change":"2024-04-25T05:29:30.781431+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.43:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2082036420}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3423182021}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3953560108}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3958950586}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":3735093998}]}]} 2024-04-25T05:34:17.732 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T05:34:17.732 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T05:34:17.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T05:34:17.986 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:17.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1839956365' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T05:34:18.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1839956365' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T05:34:18.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1839956365' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T05:34:18.814 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:18.814 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":75,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","created":"2024-04-25T05:28:34.192515+0000","modified":"2024-04-25T05:34:09.648466+0000","last_up_change":"2024-04-25T05:33:46.700068+0000","last_in_change":"2024-04-25T05:33:28.520304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T05:31:44.863613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-25T05:33:56.045664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-25T05:33:57.225173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-25T05:33:58.748103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-25T05:34:00.700767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-25T05:34:02.767681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef370ce4-85d8-41ed-9655-2e2eaab3b48e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":3557191248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3557191248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3557191248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":3557191248}]},"public_addr":"172.21.15.43:6801/3557191248","cluster_addr":"172.21.15.43:6802/3557191248","heartbeat_back_addr":"172.21.15.43:6804/3557191248","heartbeat_front_addr":"172.21.15.43:6803/3557191248","state":["exists","up"]},{"osd":1,"uuid":"ed504d08-dc59-4584-bd5f-8059e02a4161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":235229295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":235229295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":235229295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":235229295}]},"public_addr":"172.21.15.43:6805/235229295","cluster_addr":"172.21.15.43:6806/235229295","heartbeat_back_addr":"172.21.15.43:6808/235229295","heartbeat_front_addr":"172.21.15.43:6807/235229295","state":["exists","up"]},{"osd":2,"uuid":"11ff9a46-8686-4b36-a530-5cc1dca1f088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":3416612512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3416612512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3416612512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3416612512}]},"public_addr":"172.21.15.43:6809/3416612512","cluster_addr":"172.21.15.43:6810/3416612512","heartbeat_back_addr":"172.21.15.43:6812/3416612512","heartbeat_front_addr":"172.21.15.43:6811/3416612512","state":["exists","up"]},{"osd":3,"uuid":"f3314c45-11c4-46ee-be0b-6f639faa7da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":3391766005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3391766005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3391766005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6815","nonce":3391766005}]},"public_addr":"172.21.15.43:6813/3391766005","cluster_addr":"172.21.15.43:6814/3391766005","heartbeat_back_addr":"172.21.15.43:6816/3391766005","heartbeat_front_addr":"172.21.15.43:6815/3391766005","state":["exists","up"]},{"osd":4,"uuid":"8c5337b2-94d1-4784-9321-db771de75f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1310286745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6801","nonce":1310286745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1310286745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1310286745}]},"public_addr":"172.21.15.60:6800/1310286745","cluster_addr":"172.21.15.60:6801/1310286745","heartbeat_back_addr":"172.21.15.60:6803/1310286745","heartbeat_front_addr":"172.21.15.60:6802/1310286745","state":["exists","up"]},{"osd":5,"uuid":"fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3586186657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3586186657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":3586186657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3586186657}]},"public_addr":"172.21.15.60:6804/3586186657","cluster_addr":"172.21.15.60:6805/3586186657","heartbeat_back_addr":"172.21.15.60:6807/3586186657","heartbeat_front_addr":"172.21.15.60:6806/3586186657","state":["exists","up"]},{"osd":6,"uuid":"5b946188-0dd1-4e68-bcfc-a3daa342edaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1675137598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":1675137598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1675137598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1675137598}]},"public_addr":"172.21.15.60:6808/1675137598","cluster_addr":"172.21.15.60:6809/1675137598","heartbeat_back_addr":"172.21.15.60:6811/1675137598","heartbeat_front_addr":"172.21.15.60:6810/1675137598","state":["exists","up"]},{"osd":7,"uuid":"dec84bb8-9d2b-469f-9641-62ff40b9b9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3982012700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":3982012700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":3982012700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3982012700}]},"public_addr":"172.21.15.60:6812/3982012700","cluster_addr":"172.21.15.60:6813/3982012700","heartbeat_back_addr":"172.21.15.60:6815/3982012700","heartbeat_front_addr":"172.21.15.60:6814/3982012700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:14.212622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:52.000246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:03.539581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:27.046046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:56.302280+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:34:01.571236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:19.709604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:44.960822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/4077364595":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/830746728":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/786479021":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1835545567":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1408327650":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/2239747746":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/4084484879":"2024-04-26T05:29:30.781325+0000","172.21.15.43:0/1247294293":"2024-04-26T05:29:30.781325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T05:34:18.814 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:18 smithi043 ceph-mon[27010]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.9 KiB/s wr, 125 op/s 2024-04-25T05:34:18.814 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:18 smithi043 ceph-mon[37492]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.9 KiB/s wr, 125 op/s 2024-04-25T05:34:18.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:18 smithi060 ceph-mon[30898]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 2.9 KiB/s wr, 125 op/s 2024-04-25T05:34:19.541 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T05:34:19.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T05:34:19.795 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:19.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/966800343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:19.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/966800343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:19.948 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/966800343' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:19.948 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:19 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:34:20.546 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:20.546 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":75,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","created":"2024-04-25T05:28:34.192515+0000","modified":"2024-04-25T05:34:09.648466+0000","last_up_change":"2024-04-25T05:33:46.700068+0000","last_in_change":"2024-04-25T05:33:28.520304+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T05:31:44.863613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-25T05:33:56.045664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-25T05:33:57.225173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-25T05:33:58.748103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-25T05:34:00.700767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-25T05:34:02.767681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef370ce4-85d8-41ed-9655-2e2eaab3b48e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":3557191248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3557191248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3557191248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":3557191248}]},"public_addr":"172.21.15.43:6801/3557191248","cluster_addr":"172.21.15.43:6802/3557191248","heartbeat_back_addr":"172.21.15.43:6804/3557191248","heartbeat_front_addr":"172.21.15.43:6803/3557191248","state":["exists","up"]},{"osd":1,"uuid":"ed504d08-dc59-4584-bd5f-8059e02a4161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":235229295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":235229295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":235229295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":235229295}]},"public_addr":"172.21.15.43:6805/235229295","cluster_addr":"172.21.15.43:6806/235229295","heartbeat_back_addr":"172.21.15.43:6808/235229295","heartbeat_front_addr":"172.21.15.43:6807/235229295","state":["exists","up"]},{"osd":2,"uuid":"11ff9a46-8686-4b36-a530-5cc1dca1f088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":3416612512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3416612512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3416612512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3416612512}]},"public_addr":"172.21.15.43:6809/3416612512","cluster_addr":"172.21.15.43:6810/3416612512","heartbeat_back_addr":"172.21.15.43:6812/3416612512","heartbeat_front_addr":"172.21.15.43:6811/3416612512","state":["exists","up"]},{"osd":3,"uuid":"f3314c45-11c4-46ee-be0b-6f639faa7da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":3391766005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3391766005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3391766005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6815","nonce":3391766005}]},"public_addr":"172.21.15.43:6813/3391766005","cluster_addr":"172.21.15.43:6814/3391766005","heartbeat_back_addr":"172.21.15.43:6816/3391766005","heartbeat_front_addr":"172.21.15.43:6815/3391766005","state":["exists","up"]},{"osd":4,"uuid":"8c5337b2-94d1-4784-9321-db771de75f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1310286745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6801","nonce":1310286745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1310286745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1310286745}]},"public_addr":"172.21.15.60:6800/1310286745","cluster_addr":"172.21.15.60:6801/1310286745","heartbeat_back_addr":"172.21.15.60:6803/1310286745","heartbeat_front_addr":"172.21.15.60:6802/1310286745","state":["exists","up"]},{"osd":5,"uuid":"fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3586186657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3586186657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":3586186657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3586186657}]},"public_addr":"172.21.15.60:6804/3586186657","cluster_addr":"172.21.15.60:6805/3586186657","heartbeat_back_addr":"172.21.15.60:6807/3586186657","heartbeat_front_addr":"172.21.15.60:6806/3586186657","state":["exists","up"]},{"osd":6,"uuid":"5b946188-0dd1-4e68-bcfc-a3daa342edaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1675137598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":1675137598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1675137598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1675137598}]},"public_addr":"172.21.15.60:6808/1675137598","cluster_addr":"172.21.15.60:6809/1675137598","heartbeat_back_addr":"172.21.15.60:6811/1675137598","heartbeat_front_addr":"172.21.15.60:6810/1675137598","state":["exists","up"]},{"osd":7,"uuid":"dec84bb8-9d2b-469f-9641-62ff40b9b9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3982012700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":3982012700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":3982012700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3982012700}]},"public_addr":"172.21.15.60:6812/3982012700","cluster_addr":"172.21.15.60:6813/3982012700","heartbeat_back_addr":"172.21.15.60:6815/3982012700","heartbeat_front_addr":"172.21.15.60:6814/3982012700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:14.212622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:52.000246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:03.539581+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:27.046046+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:56.302280+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:34:01.571236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:19.709604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:33:44.960822+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/4077364595":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/830746728":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/786479021":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1835545567":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/1408327650":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/2239747746":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/4084484879":"2024-04-26T05:29:30.781325+0000","172.21.15.43:0/1247294293":"2024-04-26T05:29:30.781325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T05:34:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:20 smithi043 ceph-mon[27010]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 818 B/s wr, 44 op/s 2024-04-25T05:34:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2931477601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:20 smithi043 ceph-mon[37492]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 818 B/s wr, 44 op/s 2024-04-25T05:34:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2931477601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:20.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:20 smithi060 ceph-mon[30898]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 818 B/s wr, 44 op/s 2024-04-25T05:34:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2931477601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:34:21.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T05:34:21.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T05:34:21.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T05:34:21.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T05:34:21.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T05:34:21.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T05:34:21.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T05:34:21.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T05:34:21.834 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.835 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.835 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.836 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.836 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.837 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.841 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:34:21.973 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:21 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:34:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:21 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:21 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:34:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:21 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:34:22.684 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:22 smithi060 systemd[1]: Starting Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:34:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:22 smithi043 ceph-mon[27010]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 641 B/s wr, 35 op/s 2024-04-25T05:34:22.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:22 smithi043 ceph-mon[37492]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 641 B/s wr, 35 op/s 2024-04-25T05:34:22.983 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:22 smithi060 ceph-mon[30898]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 641 B/s wr, 35 op/s 2024-04-25T05:34:22.983 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:22 smithi060 podman[58869]: 2024-04-25 05:34:22.6188737 +0000 UTC m=+0.018885600 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-25T05:34:22.983 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:22 smithi060 podman[58869]: 2024-04-25 05:34:22.747227171 +0000 UTC m=+0.147239087 container create be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:22.983 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:22 smithi060 podman[58869]: 2024-04-25 05:34:22.98365866 +0000 UTC m=+0.383670542 container init be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:22 smithi060 podman[58869]: 2024-04-25 05:34:22.986824988 +0000 UTC m=+0.386836862 container start be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.011Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.011Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.011Z caller=main.go:570 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi060 (none))" 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.011Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-25T05:34:23.249 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.011Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.012Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.012Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.013Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.013Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.016Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.016Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.21µs 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.016Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.016Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.016Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.08µs wal_replay_duration=229.296µs wbl_replay_duration=120ns total_replay_duration=272.753µs 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.017Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.017Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-25T05:34:23.250 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.017Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-25T05:34:23.251 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.038Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.804557ms db_storage=769ns remote_storage=927ns web_handler=373ns query_engine=759ns scrape=393.264µs scrape_sd=86.751µs notify=853ns notify_sd=2.47µs rules=20.048856ms tracing=5.889µs 2024-04-25T05:34:23.251 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.038Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-25T05:34:23.251 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:34:23.038Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-25T05:34:23.251 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 bash[58869]: be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab 2024-04-25T05:34:23.251 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:34:23 smithi060 systemd[1]: Started Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:34:23.941 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:34:23.941 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:34:23.941 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: osdmap e76: 8 total, 8 up, 8 in 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: osdmap e76: 8 total, 8 up, 8 in 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.942 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:23 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: osdmap e76: 8 total, 8 up, 8 in 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' 2024-04-25T05:34:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:23 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-25T05:34:24.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: ignoring --setuser ceph since I am not root 2024-04-25T05:34:24.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: ignoring --setgroup ceph since I am not root 2024-04-25T05:34:24.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mgr[32756]: -- 172.21.15.60:0/2287278857 <== mon.1 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f726e665a0 con 0x55f726e3b000 2024-04-25T05:34:24.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:24.348+0000 7f23d58eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:34:24.511 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setuser ceph since I am not root 2024-04-25T05:34:24.511 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ignoring --setgroup ceph since I am not root 2024-04-25T05:34:24.511 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:24.355+0000 7fa19b5f9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T05:34:24.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[37492]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:34:24.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:34:24.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:34:24.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[37492]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-25T05:34:24.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[37492]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[27010]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[27010]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-mon[27010]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:24.553+0000 7fa19b5f9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:34:24.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:24.662+0000 7fa19b5f9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:34:24.971 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:24.540+0000 7f23d58eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:34:24.971 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:24.647+0000 7f23d58eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T05:34:24.971 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mon[30898]: pgmap v212: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:34:24.971 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:34:24.971 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:34:24.972 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mon[30898]: from='mgr.14152 172.21.15.43:0/2103360626' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-25T05:34:24.972 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-mon[30898]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-25T05:34:25.248 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:24.971+0000 7f23d58eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:34:25.249 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:25.060+0000 7f23d58eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:34:25.350 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:24 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:24.993+0000 7fa19b5f9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T05:34:25.350 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:25 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:25.084+0000 7fa19b5f9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T05:34:25.378 INFO:teuthology.orchestra.run.smithi043.stdout:55834574888 2024-04-25T05:34:25.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T05:34:25.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:25 smithi060 ceph-mon[30898]: osdmap e77: 8 total, 8 up, 8 in 2024-04-25T05:34:26.064 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:25 smithi043 ceph-mon[27010]: osdmap e77: 8 total, 8 up, 8 in 2024-04-25T05:34:26.064 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:25 smithi043 ceph-mon[37492]: osdmap e77: 8 total, 8 up, 8 in 2024-04-25T05:34:26.092 INFO:teuthology.orchestra.run.smithi043.stdout:77309411363 2024-04-25T05:34:26.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T05:34:26.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:26.369+0000 7fa19b5f9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:34:26.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:26.571+0000 7fa19b5f9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:34:26.662 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:26.308+0000 7f23d58eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T05:34:26.662 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:26.485+0000 7f23d58eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T05:34:26.662 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:26.662+0000 7f23d58eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:34:26.836 INFO:teuthology.orchestra.run.smithi043.stdout:253403070475 2024-04-25T05:34:26.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T05:34:27.070 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:26.752+0000 7fa19b5f9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T05:34:27.071 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:27 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:27.070+0000 7fa19b5f9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:34:27.248 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:26.965+0000 7f23d58eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T05:34:27.249 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:27.062+0000 7f23d58eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:34:27.339 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:27 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:27.173+0000 7fa19b5f9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T05:34:27.618 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:27 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:27.401+0000 7fa19b5f9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:34:27.748 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:27.273+0000 7f23d58eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T05:34:27.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:27.474+0000 7f23d58eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:34:27.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:27 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:27.626+0000 7fa19b5f9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T05:34:28.305 INFO:teuthology.orchestra.run.smithi043.stdout:171798691860 2024-04-25T05:34:28.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T05:34:28.487 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:28.270+0000 7fa19b5f9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:34:28.498 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:28.103+0000 7f23d58eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T05:34:28.859 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:28.546+0000 7f23d58eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:34:28.859 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:28.646+0000 7f23d58eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:34:29.062 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:28.744+0000 7fa19b5f9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T05:34:29.062 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:28 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:28.847+0000 7fa19b5f9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T05:34:29.062 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:29.062+0000 7fa19b5f9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:34:29.248 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:28.859+0000 7f23d58eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T05:34:29.290 INFO:teuthology.orchestra.run.smithi043.stdout:34359738414 2024-04-25T05:34:29.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T05:34:29.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:29.406+0000 7f23d58eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:34:29.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:29.501+0000 7f23d58eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:34:29.862 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:29.626+0000 7fa19b5f9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T05:34:29.862 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:29.723+0000 7fa19b5f9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T05:34:29.890 INFO:teuthology.orchestra.run.smithi043.stdout:214748364815 2024-04-25T05:34:29.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T05:34:30.115 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:29.755+0000 7f23d58eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:34:30.115 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:29.893+0000 7f23d58eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:34:30.115 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:29.992+0000 7f23d58eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:34:30.115 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:30.114+0000 7f23d58eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:34:30.143 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:29.989+0000 7fa19b5f9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T05:34:30.591 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:30.154+0000 7fa19b5f9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T05:34:30.592 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:30.264+0000 7fa19b5f9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T05:34:30.592 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:30.388+0000 7fa19b5f9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T05:34:30.610 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:30.610 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:30.611 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:30.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:30.638 INFO:teuthology.orchestra.run.smithi043.stdout:133143986200 2024-04-25T05:34:30.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T05:34:30.748 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:30.428+0000 7f23d58eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:34:30.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:30 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:30.724+0000 7fa19b5f9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T05:34:31.095 INFO:teuthology.orchestra.run.smithi043.stdout:111669149725 2024-04-25T05:34:31.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T05:34:31.349 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:31.061+0000 7f23d58eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:34:31.349 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:31.287+0000 7f23d58eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:34:31.349 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: [25/Apr/2024:05:34:31] ENGINE Bus STARTING 2024-04-25T05:34:31.634 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:31.402+0000 7fa19b5f9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T05:34:31.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: Standby manager daemon x restarted 2024-04-25T05:34:31.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: Standby manager daemon x started 2024-04-25T05:34:31.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:34:31.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[37492]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: Standby manager daemon x restarted 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: Standby manager daemon x started 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:34:31.635 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-mon[27010]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: CherryPy Checker: 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: The Application mounted at '' has an empty config. 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: [25/Apr/2024:05:34:31] ENGINE Serving on http://:::9283 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x[32752]: [25/Apr/2024:05:34:31] ENGINE Bus STARTED 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: Standby manager daemon x restarted 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: Standby manager daemon x started 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-25T05:34:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:31 smithi060 ceph-mon[30898]: from='mgr.? 172.21.15.60:0/2955823834' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:31.633+0000 7fa19b5f9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:31] ENGINE Bus STARTING 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: CherryPy Checker: 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: The Application mounted at '' has an empty config. 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:34:31.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:31] ENGINE Serving on http://:::9283 2024-04-25T05:34:31.893 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:31 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:31] ENGINE Bus STARTED 2024-04-25T05:34:32.004 INFO:teuthology.orchestra.run.smithi043.stdout:253403070474 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: Active manager daemon y restarted 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: Activating manager daemon y 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: osdmap e78: 8 total, 8 up, 8 in 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: mgrmap e19: y(active, starting, since 0.00960048s), standbys: x 2024-04-25T05:34:32.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: Active manager daemon y restarted 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: Activating manager daemon y 2024-04-25T05:34:32.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: osdmap e78: 8 total, 8 up, 8 in 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: mgrmap e19: y(active, starting, since 0.00960048s), standbys: x 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:34:32.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:34:32.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: Manager daemon y is now available 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:34:32.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/956119410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: Manager daemon y is now available 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:34:32.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:34:32.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/956119410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: mgrmap e18: y(active, since 5m), standbys: x 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: Active manager daemon y restarted 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: Activating manager daemon y 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: osdmap e78: 8 total, 8 up, 8 in 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: mgrmap e19: y(active, starting, since 0.00960048s), standbys: x 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T05:34:32.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: Manager daemon y is now available 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-25T05:34:32.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-25T05:34:32.658 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/956119410' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:32.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:34:33.169 INFO:teuthology.orchestra.run.smithi043.stdout:34359738413 2024-04-25T05:34:33.392 INFO:teuthology.orchestra.run.smithi043.stdout:55834574886 2024-04-25T05:34:33.620 INFO:teuthology.orchestra.run.smithi043.stdout:77309411363 2024-04-25T05:34:33.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: mgrmap e20: y(active, since 1.01567s), standbys: x 2024-04-25T05:34:33.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:34:32] ENGINE Bus STARTING 2024-04-25T05:34:33.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:34:32] ENGINE Serving on http://172.21.15.43:8765 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:34:32] ENGINE Serving on https://172.21.15.43:7150 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: [25/Apr/2024:05:34:32] ENGINE Bus STARTED 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2915026128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2611377637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2114968818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: mgrmap e20: y(active, since 1.01567s), standbys: x 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: [25/Apr/2024:05:34:32] ENGINE Bus STARTING 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: [25/Apr/2024:05:34:32] ENGINE Serving on http://172.21.15.43:8765 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: [25/Apr/2024:05:34:32] ENGINE Serving on https://172.21.15.43:7150 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: [25/Apr/2024:05:34:32] ENGINE Bus STARTED 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2915026128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2611377637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:33.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2114968818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:34:33.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: mgrmap e20: y(active, since 1.01567s), standbys: x 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: [25/Apr/2024:05:34:32] ENGINE Bus STARTING 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: [25/Apr/2024:05:34:32] ENGINE Serving on http://172.21.15.43:8765 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: [25/Apr/2024:05:34:32] ENGINE Serving on https://172.21.15.43:7150 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: [25/Apr/2024:05:34:32] ENGINE Bus STARTED 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2915026128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2611377637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:33.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2114968818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:34:34.432 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070475 got 253403070474 for osd.7 2024-04-25T05:34:34.652 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:34:34.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:34:34.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:34:34.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[27010]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:34.653 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[27010]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-25T05:34:34.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[27010]: osdmap e79: 8 total, 8 up, 8 in 2024-04-25T05:34:34.928 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:34:34.928 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[37492]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:34.928 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[37492]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-25T05:34:34.928 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:34 smithi043 ceph-mon[37492]: osdmap e79: 8 total, 8 up, 8 in 2024-04-25T05:34:34.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411363 for osd.2 2024-04-25T05:34:34.963 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:34 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:34:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:34 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:34:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:34 smithi060 ceph-mon[30898]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:34 smithi060 ceph-mon[30898]: mgrmap e21: y(active, since 2s), standbys: x 2024-04-25T05:34:34.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:34 smithi060 ceph-mon[30898]: osdmap e79: 8 total, 8 up, 8 in 2024-04-25T05:34:35.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T05:34:35.854 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574886 for osd.1 2024-04-25T05:34:35.878 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:35.992 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[27010]: osdmap e80: 8 total, 8 up, 8 in 2024-04-25T05:34:35.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:35.993 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:35.993 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[37492]: osdmap e80: 8 total, 8 up, 8 in 2024-04-25T05:34:35.993 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:35.993 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:35.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:35 smithi060 ceph-mon[30898]: osdmap e80: 8 total, 8 up, 8 in 2024-04-25T05:34:35.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:35.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:36.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738413 for osd.0 2024-04-25T05:34:36.358 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:36.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 3761M 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 3761M 2024-04-25T05:34:37.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T05:34:37.455 INFO:teuthology.orchestra.run.smithi043.stdout:253403070476 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: mgrmap e22: y(active, since 4s), standbys: x 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 3761M 2024-04-25T05:34:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:38.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/852091308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:34:38.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1541863534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:38.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/852091308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:34:38.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1541863534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/852091308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:34:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1541863534' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:34:38.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070475 got 253403070476 for osd.7 2024-04-25T05:34:38.805 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:38.917 INFO:teuthology.orchestra.run.smithi043.stdout:171798691861 2024-04-25T05:34:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:39.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:39 smithi043 ceph-mon[27010]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:39.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:39 smithi043 ceph-mon[37492]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:39 smithi060 ceph-mon[30898]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:34:39.533 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691860 got 171798691861 for osd.5 2024-04-25T05:34:39.533 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:39.611 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:39.612 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:39.854 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:34:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:34:40.706 INFO:teuthology.orchestra.run.smithi043.stdout:133143986202 2024-04-25T05:34:40.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:40 smithi043 ceph-mon[37492]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-25T05:34:40.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:40 smithi043 ceph-mon[37492]: osdmap e81: 8 total, 8 up, 8 in 2024-04-25T05:34:40.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:40 smithi043 ceph-mon[27010]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-25T05:34:40.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:40 smithi043 ceph-mon[27010]: osdmap e81: 8 total, 8 up, 8 in 2024-04-25T05:34:40.957 INFO:teuthology.orchestra.run.smithi043.stdout:55834574890 2024-04-25T05:34:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:40 smithi060 ceph-mon[30898]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-25T05:34:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:40 smithi060 ceph-mon[30898]: osdmap e81: 8 total, 8 up, 8 in 2024-04-25T05:34:41.794 INFO:teuthology.orchestra.run.smithi043.stdout:214748364817 2024-04-25T05:34:41.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3581945005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:34:41.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2828907472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:41.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3581945005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:34:41.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2828907472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:41.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3581945005' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:34:41.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2828907472' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:34:42.140 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986202 for osd.4 2024-04-25T05:34:42.140 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:42.986 INFO:teuthology.orchestra.run.smithi043.stdout:111669149727 2024-04-25T05:34:42.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:42 smithi043 ceph-mon[27010]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:42.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1762432408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:34:42.987 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:42 smithi043 ceph-mon[37492]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:42.987 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1762432408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:34:42.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:34:42.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:42 smithi060 ceph-mon[30898]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:42.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1762432408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:34:43.307 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364815 got 214748364817 for osd.6 2024-04-25T05:34:43.307 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:43.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/188629610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:34:44.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:44.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/188629610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:34:44.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:44.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/188629610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:34:44.042 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574890 for osd.1 2024-04-25T05:34:44.043 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:44.076 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:44.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149725 got 111669149727 for osd.3 2024-04-25T05:34:44.623 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:44.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:44 smithi043 ceph-mon[27010]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-25T05:34:44.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:44 smithi043 ceph-mon[27010]: osdmap e82: 8 total, 8 up, 8 in 2024-04-25T05:34:44.878 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:44 smithi043 ceph-mon[37492]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-25T05:34:44.878 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:44 smithi043 ceph-mon[37492]: osdmap e82: 8 total, 8 up, 8 in 2024-04-25T05:34:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:44 smithi060 ceph-mon[30898]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-04-25T05:34:44.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:44 smithi060 ceph-mon[30898]: osdmap e82: 8 total, 8 up, 8 in 2024-04-25T05:34:45.503 INFO:teuthology.orchestra.run.smithi043.stdout:34359738417 2024-04-25T05:34:46.245 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738417 for osd.0 2024-04-25T05:34:46.245 DEBUG:teuthology.parallel:result is None 2024-04-25T05:34:46.245 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T05:34:46.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:34:46.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:46 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:46 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2267677071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:46 smithi060 ceph-mon[30898]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2267677071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[27010]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2267677071' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:34:46.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:46 smithi043 ceph-mon[37492]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-25T05:34:47.274 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:47.277 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:34:47.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 3761M 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:34:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:47 smithi060 ceph-mon[30898]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 3761M 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[27010]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 3761M 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: Updating smithi043:/etc/ceph/ceph.conf 2024-04-25T05:34:47.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:47 smithi043 ceph-mon[37492]: Updating smithi060:/etc/ceph/ceph.conf 2024-04-25T05:34:48.107 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-25T05:34:45.655949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1260,"num_read_kb":1074,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235868,"kb_used_data":7228,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725348,"statfs":{"total":767960285184,"available":767718756352,"internally_reserved":0,"allocated":7401472,"data_stored":4163441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":216,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002163"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661166+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:34:39.661166+0000","last_peered":"2024-04-25T05:34:39.661166+0000","last_clean":"2024-04-25T05:34:39.661166+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:34:39.661166+0000","last_undegraded":"2024-04-25T05:34:39.661166+0000","last_fullsized":"2024-04-25T05:34:39.661166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.921834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660295+0000","last_change":"2024-04-25T05:34:02.700825+0000","last_active":"2024-04-25T05:34:39.660295+0000","last_peered":"2024-04-25T05:34:39.660295+0000","last_clean":"2024-04-25T05:34:39.660295+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:34:39.660295+0000","last_undegraded":"2024-04-25T05:34:39.660295+0000","last_fullsized":"2024-04-25T05:34:39.660295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.901521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447130+0000","last_change":"2024-04-25T05:33:58.677608+0000","last_active":"2024-04-25T05:34:40.447130+0000","last_peered":"2024-04-25T05:34:40.447130+0000","last_clean":"2024-04-25T05:34:40.447130+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:34:40.447130+0000","last_undegraded":"2024-04-25T05:34:40.447130+0000","last_fullsized":"2024-04-25T05:34:40.447130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.073153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661088+0000","last_change":"2024-04-25T05:34:00.692628+0000","last_active":"2024-04-25T05:34:39.661088+0000","last_peered":"2024-04-25T05:34:39.661088+0000","last_clean":"2024-04-25T05:34:39.661088+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:34:39.661088+0000","last_undegraded":"2024-04-25T05:34:39.661088+0000","last_fullsized":"2024-04-25T05:34:39.661088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.003557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661625+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:34:39.661625+0000","last_peered":"2024-04-25T05:34:39.661625+0000","last_clean":"2024-04-25T05:34:39.661625+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:34:39.661625+0000","last_undegraded":"2024-04-25T05:34:39.661625+0000","last_fullsized":"2024-04-25T05:34:39.661625+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.498112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661088+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:34:39.661088+0000","last_peered":"2024-04-25T05:34:39.661088+0000","last_clean":"2024-04-25T05:34:39.661088+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:34:39.661088+0000","last_undegraded":"2024-04-25T05:34:39.661088+0000","last_fullsized":"2024-04-25T05:34:39.661088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.555566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661597+0000","last_change":"2024-04-25T05:33:58.679726+0000","last_active":"2024-04-25T05:34:39.661597+0000","last_peered":"2024-04-25T05:34:39.661597+0000","last_clean":"2024-04-25T05:34:39.661597+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:34:39.661597+0000","last_undegraded":"2024-04-25T05:34:39.661597+0000","last_fullsized":"2024-04-25T05:34:39.661597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.377018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":108,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661746+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:34:39.661746+0000","last_peered":"2024-04-25T05:34:39.661746+0000","last_clean":"2024-04-25T05:34:39.661746+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:34:39.661746+0000","last_undegraded":"2024-04-25T05:34:39.661746+0000","last_fullsized":"2024-04-25T05:34:39.661746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.111490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446558+0000","last_change":"2024-04-25T05:34:04.724350+0000","last_active":"2024-04-25T05:34:40.446558+0000","last_peered":"2024-04-25T05:34:40.446558+0000","last_clean":"2024-04-25T05:34:40.446558+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:34:40.446558+0000","last_undegraded":"2024-04-25T05:34:40.446558+0000","last_fullsized":"2024-04-25T05:34:40.446558+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.943468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449077+0000","last_change":"2024-04-25T05:34:02.709504+0000","last_active":"2024-04-25T05:34:40.449077+0000","last_peered":"2024-04-25T05:34:40.449077+0000","last_clean":"2024-04-25T05:34:40.449077+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:34:40.449077+0000","last_undegraded":"2024-04-25T05:34:40.449077+0000","last_fullsized":"2024-04-25T05:34:40.449077+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.151709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449187+0000","last_change":"2024-04-25T05:33:58.687700+0000","last_active":"2024-04-25T05:34:40.449187+0000","last_peered":"2024-04-25T05:34:40.449187+0000","last_clean":"2024-04-25T05:34:40.449187+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:34:40.449187+0000","last_undegraded":"2024-04-25T05:34:40.449187+0000","last_fullsized":"2024-04-25T05:34:40.449187+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.471958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":76,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661416+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:34:39.661416+0000","last_peered":"2024-04-25T05:34:39.661416+0000","last_clean":"2024-04-25T05:34:39.661416+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:34:39.661416+0000","last_undegraded":"2024-04-25T05:34:39.661416+0000","last_fullsized":"2024-04-25T05:34:39.661416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.181248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446648+0000","last_change":"2024-04-25T05:34:04.722769+0000","last_active":"2024-04-25T05:34:40.446648+0000","last_peered":"2024-04-25T05:34:40.446648+0000","last_clean":"2024-04-25T05:34:40.446648+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:34:40.446648+0000","last_undegraded":"2024-04-25T05:34:40.446648+0000","last_fullsized":"2024-04-25T05:34:40.446648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.673388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448466+0000","last_change":"2024-04-25T05:34:02.701665+0000","last_active":"2024-04-25T05:34:40.448466+0000","last_peered":"2024-04-25T05:34:40.448466+0000","last_clean":"2024-04-25T05:34:40.448466+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:34:40.448466+0000","last_undegraded":"2024-04-25T05:34:40.448466+0000","last_fullsized":"2024-04-25T05:34:40.448466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.814291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447594+0000","last_change":"2024-04-25T05:33:58.688919+0000","last_active":"2024-04-25T05:34:40.447594+0000","last_peered":"2024-04-25T05:34:40.447594+0000","last_clean":"2024-04-25T05:34:40.447594+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:34:40.447594+0000","last_undegraded":"2024-04-25T05:34:40.447594+0000","last_fullsized":"2024-04-25T05:34:40.447594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.028583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661513+0000","last_change":"2024-04-25T05:34:00.693330+0000","last_active":"2024-04-25T05:34:39.661513+0000","last_peered":"2024-04-25T05:34:39.661513+0000","last_clean":"2024-04-25T05:34:39.661513+0000","last_became_active":"2024-04-25T05:34:00.693003+0000","last_became_peered":"2024-04-25T05:34:00.693003+0000","last_unstale":"2024-04-25T05:34:39.661513+0000","last_undegraded":"2024-04-25T05:34:39.661513+0000","last_fullsized":"2024-04-25T05:34:39.661513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.762449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662183+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:34:39.662183+0000","last_peered":"2024-04-25T05:34:39.662183+0000","last_clean":"2024-04-25T05:34:39.662183+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:34:39.662183+0000","last_undegraded":"2024-04-25T05:34:39.662183+0000","last_fullsized":"2024-04-25T05:34:39.662183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.388118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.718071+0000","last_change":"2024-04-25T05:34:02.708603+0000","last_active":"2024-04-25T05:34:39.718071+0000","last_peered":"2024-04-25T05:34:39.718071+0000","last_clean":"2024-04-25T05:34:39.718071+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:34:39.718071+0000","last_undegraded":"2024-04-25T05:34:39.718071+0000","last_fullsized":"2024-04-25T05:34:39.718071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.908190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447588+0000","last_change":"2024-04-25T05:33:58.683934+0000","last_active":"2024-04-25T05:34:40.447588+0000","last_peered":"2024-04-25T05:34:40.447588+0000","last_clean":"2024-04-25T05:34:40.447588+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:34:40.447588+0000","last_undegraded":"2024-04-25T05:34:40.447588+0000","last_fullsized":"2024-04-25T05:34:40.447588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.927389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446756+0000","last_change":"2024-04-25T05:34:00.690831+0000","last_active":"2024-04-25T05:34:40.446756+0000","last_peered":"2024-04-25T05:34:40.446756+0000","last_clean":"2024-04-25T05:34:40.446756+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:34:40.446756+0000","last_undegraded":"2024-04-25T05:34:40.446756+0000","last_fullsized":"2024-04-25T05:34:40.446756+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.940787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662227+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:34:39.662227+0000","last_peered":"2024-04-25T05:34:39.662227+0000","last_clean":"2024-04-25T05:34:39.662227+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:34:39.662227+0000","last_undegraded":"2024-04-25T05:34:39.662227+0000","last_fullsized":"2024-04-25T05:34:39.662227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661601+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:34:39.661601+0000","last_peered":"2024-04-25T05:34:39.661601+0000","last_clean":"2024-04-25T05:34:39.661601+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:34:39.661601+0000","last_undegraded":"2024-04-25T05:34:39.661601+0000","last_fullsized":"2024-04-25T05:34:39.661601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.792766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661499+0000","last_change":"2024-04-25T05:33:58.677738+0000","last_active":"2024-04-25T05:34:39.661499+0000","last_peered":"2024-04-25T05:34:39.661499+0000","last_clean":"2024-04-25T05:34:39.661499+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:34:39.661499+0000","last_undegraded":"2024-04-25T05:34:39.661499+0000","last_fullsized":"2024-04-25T05:34:39.661499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.705248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":70,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661782+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:34:39.661782+0000","last_peered":"2024-04-25T05:34:39.661782+0000","last_clean":"2024-04-25T05:34:39.661782+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:34:39.661782+0000","last_undegraded":"2024-04-25T05:34:39.661782+0000","last_fullsized":"2024-04-25T05:34:39.661782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.537702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447918+0000","last_change":"2024-04-25T05:34:04.725886+0000","last_active":"2024-04-25T05:34:40.447918+0000","last_peered":"2024-04-25T05:34:40.447918+0000","last_clean":"2024-04-25T05:34:40.447918+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:34:40.447918+0000","last_undegraded":"2024-04-25T05:34:40.447918+0000","last_fullsized":"2024-04-25T05:34:40.447918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448655+0000","last_change":"2024-04-25T05:34:02.701003+0000","last_active":"2024-04-25T05:34:40.448655+0000","last_peered":"2024-04-25T05:34:40.448655+0000","last_clean":"2024-04-25T05:34:40.448655+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:34:40.448655+0000","last_undegraded":"2024-04-25T05:34:40.448655+0000","last_fullsized":"2024-04-25T05:34:40.448655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.795737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660446+0000","last_change":"2024-04-25T05:33:58.678204+0000","last_active":"2024-04-25T05:34:39.660446+0000","last_peered":"2024-04-25T05:34:39.660446+0000","last_clean":"2024-04-25T05:34:39.660446+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:34:39.660446+0000","last_undegraded":"2024-04-25T05:34:39.660446+0000","last_fullsized":"2024-04-25T05:34:39.660446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.231648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447222+0000","last_change":"2024-04-25T05:34:01.002306+0000","last_active":"2024-04-25T05:34:40.447222+0000","last_peered":"2024-04-25T05:34:40.447222+0000","last_clean":"2024-04-25T05:34:40.447222+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:34:40.447222+0000","last_undegraded":"2024-04-25T05:34:40.447222+0000","last_fullsized":"2024-04-25T05:34:40.447222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.875946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660818+0000","last_change":"2024-04-25T05:34:04.722159+0000","last_active":"2024-04-25T05:34:39.660818+0000","last_peered":"2024-04-25T05:34:39.660818+0000","last_clean":"2024-04-25T05:34:39.660818+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:34:39.660818+0000","last_undegraded":"2024-04-25T05:34:39.660818+0000","last_fullsized":"2024-04-25T05:34:39.660818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.853709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447421+0000","last_change":"2024-04-25T05:34:02.710777+0000","last_active":"2024-04-25T05:34:40.447421+0000","last_peered":"2024-04-25T05:34:40.447421+0000","last_clean":"2024-04-25T05:34:40.447421+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:34:40.447421+0000","last_undegraded":"2024-04-25T05:34:40.447421+0000","last_fullsized":"2024-04-25T05:34:40.447421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.845754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660629+0000","last_change":"2024-04-25T05:33:58.676498+0000","last_active":"2024-04-25T05:34:39.660629+0000","last_peered":"2024-04-25T05:34:39.660629+0000","last_clean":"2024-04-25T05:34:39.660629+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:34:39.660629+0000","last_undegraded":"2024-04-25T05:34:39.660629+0000","last_fullsized":"2024-04-25T05:34:39.660629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.714656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448541+0000","last_change":"2024-04-25T05:34:00.691266+0000","last_active":"2024-04-25T05:34:40.448541+0000","last_peered":"2024-04-25T05:34:40.448541+0000","last_clean":"2024-04-25T05:34:40.448541+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:34:40.448541+0000","last_undegraded":"2024-04-25T05:34:40.448541+0000","last_fullsized":"2024-04-25T05:34:40.448541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.730290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448060+0000","last_change":"2024-04-25T05:34:04.725684+0000","last_active":"2024-04-25T05:34:40.448060+0000","last_peered":"2024-04-25T05:34:40.448060+0000","last_clean":"2024-04-25T05:34:40.448060+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:34:40.448060+0000","last_undegraded":"2024-04-25T05:34:40.448060+0000","last_fullsized":"2024-04-25T05:34:40.448060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.515944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446894+0000","last_change":"2024-04-25T05:34:02.701292+0000","last_active":"2024-04-25T05:34:40.446894+0000","last_peered":"2024-04-25T05:34:40.446894+0000","last_clean":"2024-04-25T05:34:40.446894+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:34:40.446894+0000","last_undegraded":"2024-04-25T05:34:40.446894+0000","last_fullsized":"2024-04-25T05:34:40.446894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447095+0000","last_change":"2024-04-25T05:33:58.678452+0000","last_active":"2024-04-25T05:34:40.447095+0000","last_peered":"2024-04-25T05:34:40.447095+0000","last_clean":"2024-04-25T05:34:40.447095+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:34:40.447095+0000","last_undegraded":"2024-04-25T05:34:40.447095+0000","last_fullsized":"2024-04-25T05:34:40.447095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.043078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660721+0000","last_change":"2024-04-25T05:34:00.692546+0000","last_active":"2024-04-25T05:34:39.660721+0000","last_peered":"2024-04-25T05:34:39.660721+0000","last_clean":"2024-04-25T05:34:39.660721+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:34:39.660721+0000","last_undegraded":"2024-04-25T05:34:39.660721+0000","last_fullsized":"2024-04-25T05:34:39.660721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.114492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661287+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:34:39.661287+0000","last_peered":"2024-04-25T05:34:39.661287+0000","last_clean":"2024-04-25T05:34:39.661287+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:34:39.661287+0000","last_undegraded":"2024-04-25T05:34:39.661287+0000","last_fullsized":"2024-04-25T05:34:39.661287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.872916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448376+0000","last_change":"2024-04-25T05:34:02.701784+0000","last_active":"2024-04-25T05:34:40.448376+0000","last_peered":"2024-04-25T05:34:40.448376+0000","last_clean":"2024-04-25T05:34:40.448376+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:34:40.448376+0000","last_undegraded":"2024-04-25T05:34:40.448376+0000","last_fullsized":"2024-04-25T05:34:40.448376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.034685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448186+0000","last_change":"2024-04-25T05:33:58.679682+0000","last_active":"2024-04-25T05:34:40.448186+0000","last_peered":"2024-04-25T05:34:40.448186+0000","last_clean":"2024-04-25T05:34:40.448186+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:34:40.448186+0000","last_undegraded":"2024-04-25T05:34:40.448186+0000","last_fullsized":"2024-04-25T05:34:40.448186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661628+0000","last_change":"2024-04-25T05:34:00.999704+0000","last_active":"2024-04-25T05:34:39.661628+0000","last_peered":"2024-04-25T05:34:39.661628+0000","last_clean":"2024-04-25T05:34:39.661628+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:34:39.661628+0000","last_undegraded":"2024-04-25T05:34:39.661628+0000","last_fullsized":"2024-04-25T05:34:39.661628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.051072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446242+0000","last_change":"2024-04-25T05:34:04.727071+0000","last_active":"2024-04-25T05:34:40.446242+0000","last_peered":"2024-04-25T05:34:40.446242+0000","last_clean":"2024-04-25T05:34:40.446242+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:34:40.446242+0000","last_undegraded":"2024-04-25T05:34:40.446242+0000","last_fullsized":"2024-04-25T05:34:40.446242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.039265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661612+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:34:39.661612+0000","last_peered":"2024-04-25T05:34:39.661612+0000","last_clean":"2024-04-25T05:34:39.661612+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:34:39.661612+0000","last_undegraded":"2024-04-25T05:34:39.661612+0000","last_fullsized":"2024-04-25T05:34:39.661612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.038248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447963+0000","last_change":"2024-04-25T05:33:58.687406+0000","last_active":"2024-04-25T05:34:40.447963+0000","last_peered":"2024-04-25T05:34:40.447963+0000","last_clean":"2024-04-25T05:34:40.447963+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:34:40.447963+0000","last_undegraded":"2024-04-25T05:34:40.447963+0000","last_fullsized":"2024-04-25T05:34:40.447963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.844471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.718080+0000","last_change":"2024-04-25T05:34:00.689589+0000","last_active":"2024-04-25T05:34:39.718080+0000","last_peered":"2024-04-25T05:34:39.718080+0000","last_clean":"2024-04-25T05:34:39.718080+0000","last_became_active":"2024-04-25T05:34:00.688897+0000","last_became_peered":"2024-04-25T05:34:00.688897+0000","last_unstale":"2024-04-25T05:34:39.718080+0000","last_undegraded":"2024-04-25T05:34:39.718080+0000","last_fullsized":"2024-04-25T05:34:39.718080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661292+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:34:39.661292+0000","last_peered":"2024-04-25T05:34:39.661292+0000","last_clean":"2024-04-25T05:34:39.661292+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:34:39.661292+0000","last_undegraded":"2024-04-25T05:34:39.661292+0000","last_fullsized":"2024-04-25T05:34:39.661292+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.871648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661028+0000","last_change":"2024-04-25T05:34:02.699322+0000","last_active":"2024-04-25T05:34:39.661028+0000","last_peered":"2024-04-25T05:34:39.661028+0000","last_clean":"2024-04-25T05:34:39.661028+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:34:39.661028+0000","last_undegraded":"2024-04-25T05:34:39.661028+0000","last_fullsized":"2024-04-25T05:34:39.661028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.105773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661879+0000","last_change":"2024-04-25T05:33:58.680453+0000","last_active":"2024-04-25T05:34:39.661879+0000","last_peered":"2024-04-25T05:34:39.661879+0000","last_clean":"2024-04-25T05:34:39.661879+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:34:39.661879+0000","last_undegraded":"2024-04-25T05:34:39.661879+0000","last_fullsized":"2024-04-25T05:34:39.661879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.585516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661773+0000","last_change":"2024-04-25T05:34:00.693320+0000","last_active":"2024-04-25T05:34:39.661773+0000","last_peered":"2024-04-25T05:34:39.661773+0000","last_clean":"2024-04-25T05:34:39.661773+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:34:39.661773+0000","last_undegraded":"2024-04-25T05:34:39.661773+0000","last_fullsized":"2024-04-25T05:34:39.661773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.690705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446999+0000","last_change":"2024-04-25T05:34:04.724697+0000","last_active":"2024-04-25T05:34:40.446999+0000","last_peered":"2024-04-25T05:34:40.446999+0000","last_clean":"2024-04-25T05:34:40.446999+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:34:40.446999+0000","last_undegraded":"2024-04-25T05:34:40.446999+0000","last_fullsized":"2024-04-25T05:34:40.446999+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.282641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449259+0000","last_change":"2024-04-25T05:34:02.708587+0000","last_active":"2024-04-25T05:34:40.449259+0000","last_peered":"2024-04-25T05:34:40.449259+0000","last_clean":"2024-04-25T05:34:40.449259+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:34:40.449259+0000","last_undegraded":"2024-04-25T05:34:40.449259+0000","last_fullsized":"2024-04-25T05:34:40.449259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.470304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448433+0000","last_change":"2024-04-25T05:33:58.685869+0000","last_active":"2024-04-25T05:34:40.448433+0000","last_peered":"2024-04-25T05:34:40.448433+0000","last_clean":"2024-04-25T05:34:40.448433+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:34:40.448433+0000","last_undegraded":"2024-04-25T05:34:40.448433+0000","last_fullsized":"2024-04-25T05:34:40.448433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.260635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":74,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661528+0000","last_change":"2024-04-25T05:34:00.999737+0000","last_active":"2024-04-25T05:34:39.661528+0000","last_peered":"2024-04-25T05:34:39.661528+0000","last_clean":"2024-04-25T05:34:39.661528+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:34:39.661528+0000","last_undegraded":"2024-04-25T05:34:39.661528+0000","last_fullsized":"2024-04-25T05:34:39.661528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.305882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662507+0000","last_change":"2024-04-25T05:34:04.715688+0000","last_active":"2024-04-25T05:34:39.662507+0000","last_peered":"2024-04-25T05:34:39.662507+0000","last_clean":"2024-04-25T05:34:39.662507+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:34:39.662507+0000","last_undegraded":"2024-04-25T05:34:39.662507+0000","last_fullsized":"2024-04-25T05:34:39.662507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.959881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661991+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:34:39.661991+0000","last_peered":"2024-04-25T05:34:39.661991+0000","last_clean":"2024-04-25T05:34:39.661991+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:34:39.661991+0000","last_undegraded":"2024-04-25T05:34:39.661991+0000","last_fullsized":"2024-04-25T05:34:39.661991+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.077903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446793+0000","last_change":"2024-04-25T05:33:58.685955+0000","last_active":"2024-04-25T05:34:40.446793+0000","last_peered":"2024-04-25T05:34:40.446793+0000","last_clean":"2024-04-25T05:34:40.446793+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:34:40.446793+0000","last_undegraded":"2024-04-25T05:34:40.446793+0000","last_fullsized":"2024-04-25T05:34:40.446793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.649756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449405+0000","last_change":"2024-04-25T05:34:01.002652+0000","last_active":"2024-04-25T05:34:40.449405+0000","last_peered":"2024-04-25T05:34:40.449405+0000","last_clean":"2024-04-25T05:34:40.449405+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:34:40.449405+0000","last_undegraded":"2024-04-25T05:34:40.449405+0000","last_fullsized":"2024-04-25T05:34:40.449405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.164431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447041+0000","last_change":"2024-04-25T05:34:04.719603+0000","last_active":"2024-04-25T05:34:40.447041+0000","last_peered":"2024-04-25T05:34:40.447041+0000","last_clean":"2024-04-25T05:34:40.447041+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:34:40.447041+0000","last_undegraded":"2024-04-25T05:34:40.447041+0000","last_fullsized":"2024-04-25T05:34:40.447041+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.818750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662042+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:34:39.662042+0000","last_peered":"2024-04-25T05:34:39.662042+0000","last_clean":"2024-04-25T05:34:39.662042+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:34:39.662042+0000","last_undegraded":"2024-04-25T05:34:39.662042+0000","last_fullsized":"2024-04-25T05:34:39.662042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.004188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448684+0000","last_change":"2024-04-25T05:33:58.685792+0000","last_active":"2024-04-25T05:34:40.448684+0000","last_peered":"2024-04-25T05:34:40.448684+0000","last_clean":"2024-04-25T05:34:40.448684+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:34:40.448684+0000","last_undegraded":"2024-04-25T05:34:40.448684+0000","last_fullsized":"2024-04-25T05:34:40.448684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.497320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447321+0000","last_change":"2024-04-25T05:34:01.002399+0000","last_active":"2024-04-25T05:34:40.447321+0000","last_peered":"2024-04-25T05:34:40.447321+0000","last_clean":"2024-04-25T05:34:40.447321+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:34:40.447321+0000","last_undegraded":"2024-04-25T05:34:40.447321+0000","last_fullsized":"2024-04-25T05:34:40.447321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.611153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447838+0000","last_change":"2024-04-25T05:34:04.760885+0000","last_active":"2024-04-25T05:34:40.447838+0000","last_peered":"2024-04-25T05:34:40.447838+0000","last_clean":"2024-04-25T05:34:40.447838+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:34:40.447838+0000","last_undegraded":"2024-04-25T05:34:40.447838+0000","last_fullsized":"2024-04-25T05:34:40.447838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.397373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448289+0000","last_change":"2024-04-25T05:34:02.700150+0000","last_active":"2024-04-25T05:34:40.448289+0000","last_peered":"2024-04-25T05:34:40.448289+0000","last_clean":"2024-04-25T05:34:40.448289+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:34:40.448289+0000","last_undegraded":"2024-04-25T05:34:40.448289+0000","last_fullsized":"2024-04-25T05:34:40.448289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.726607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446917+0000","last_change":"2024-04-25T05:33:58.682812+0000","last_active":"2024-04-25T05:34:40.446917+0000","last_peered":"2024-04-25T05:34:40.446917+0000","last_clean":"2024-04-25T05:34:40.446917+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:34:40.446917+0000","last_undegraded":"2024-04-25T05:34:40.446917+0000","last_fullsized":"2024-04-25T05:34:40.446917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.737056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662237+0000","last_change":"2024-04-25T05:34:01.000303+0000","last_active":"2024-04-25T05:34:39.662237+0000","last_peered":"2024-04-25T05:34:39.662237+0000","last_clean":"2024-04-25T05:34:39.662237+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:34:39.662237+0000","last_undegraded":"2024-04-25T05:34:39.662237+0000","last_fullsized":"2024-04-25T05:34:39.662237+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.779729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447368+0000","last_change":"2024-04-25T05:34:04.759369+0000","last_active":"2024-04-25T05:34:40.447368+0000","last_peered":"2024-04-25T05:34:40.447368+0000","last_clean":"2024-04-25T05:34:40.447368+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:34:40.447368+0000","last_undegraded":"2024-04-25T05:34:40.447368+0000","last_fullsized":"2024-04-25T05:34:40.447368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.510841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448600+0000","last_change":"2024-04-25T05:33:58.689200+0000","last_active":"2024-04-25T05:34:40.448600+0000","last_peered":"2024-04-25T05:34:40.448600+0000","last_clean":"2024-04-25T05:34:40.448600+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:34:40.448600+0000","last_undegraded":"2024-04-25T05:34:40.448600+0000","last_fullsized":"2024-04-25T05:34:40.448600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.468188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":84,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446773+0000","last_change":"2024-04-25T05:34:01.000840+0000","last_active":"2024-04-25T05:34:40.446773+0000","last_peered":"2024-04-25T05:34:40.446773+0000","last_clean":"2024-04-25T05:34:40.446773+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:34:40.446773+0000","last_undegraded":"2024-04-25T05:34:40.446773+0000","last_fullsized":"2024-04-25T05:34:40.446773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.859671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448963+0000","last_change":"2024-04-25T05:34:02.701548+0000","last_active":"2024-04-25T05:34:40.448963+0000","last_peered":"2024-04-25T05:34:40.448963+0000","last_clean":"2024-04-25T05:34:40.448963+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:34:40.448963+0000","last_undegraded":"2024-04-25T05:34:40.448963+0000","last_fullsized":"2024-04-25T05:34:40.448963+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.747443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661401+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:34:39.661401+0000","last_peered":"2024-04-25T05:34:39.661401+0000","last_clean":"2024-04-25T05:34:39.661401+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:34:39.661401+0000","last_undegraded":"2024-04-25T05:34:39.661401+0000","last_fullsized":"2024-04-25T05:34:39.661401+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.987326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449186+0000","last_change":"2024-04-25T05:33:58.687364+0000","last_active":"2024-04-25T05:34:40.449186+0000","last_peered":"2024-04-25T05:34:40.449186+0000","last_clean":"2024-04-25T05:34:40.449186+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:34:40.449186+0000","last_undegraded":"2024-04-25T05:34:40.449186+0000","last_fullsized":"2024-04-25T05:34:40.449186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.749177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":82,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662016+0000","last_change":"2024-04-25T05:34:00.694001+0000","last_active":"2024-04-25T05:34:39.662016+0000","last_peered":"2024-04-25T05:34:39.662016+0000","last_clean":"2024-04-25T05:34:39.662016+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:34:39.662016+0000","last_undegraded":"2024-04-25T05:34:39.662016+0000","last_fullsized":"2024-04-25T05:34:39.662016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.293624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446249+0000","last_change":"2024-04-25T05:34:02.710703+0000","last_active":"2024-04-25T05:34:40.446249+0000","last_peered":"2024-04-25T05:34:40.446249+0000","last_clean":"2024-04-25T05:34:40.446249+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:34:40.446249+0000","last_undegraded":"2024-04-25T05:34:40.446249+0000","last_fullsized":"2024-04-25T05:34:40.446249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.457617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446897+0000","last_change":"2024-04-25T05:34:04.727176+0000","last_active":"2024-04-25T05:34:40.446897+0000","last_peered":"2024-04-25T05:34:40.446897+0000","last_clean":"2024-04-25T05:34:40.446897+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:34:40.446897+0000","last_undegraded":"2024-04-25T05:34:40.446897+0000","last_fullsized":"2024-04-25T05:34:40.446897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447398+0000","last_change":"2024-04-25T05:34:02.710395+0000","last_active":"2024-04-25T05:34:40.447398+0000","last_peered":"2024-04-25T05:34:40.447398+0000","last_clean":"2024-04-25T05:34:40.447398+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:34:40.447398+0000","last_undegraded":"2024-04-25T05:34:40.447398+0000","last_fullsized":"2024-04-25T05:34:40.447398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.749912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448240+0000","last_change":"2024-04-25T05:33:58.686463+0000","last_active":"2024-04-25T05:34:40.448240+0000","last_peered":"2024-04-25T05:34:40.448240+0000","last_clean":"2024-04-25T05:34:40.448240+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:34:40.448240+0000","last_undegraded":"2024-04-25T05:34:40.448240+0000","last_fullsized":"2024-04-25T05:34:40.448240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:59.578656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":78,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449252+0000","last_change":"2024-04-25T05:34:00.999892+0000","last_active":"2024-04-25T05:34:40.449252+0000","last_peered":"2024-04-25T05:34:40.449252+0000","last_clean":"2024-04-25T05:34:40.449252+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:34:40.449252+0000","last_undegraded":"2024-04-25T05:34:40.449252+0000","last_fullsized":"2024-04-25T05:34:40.449252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.466514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446357+0000","last_change":"2024-04-25T05:34:04.725833+0000","last_active":"2024-04-25T05:34:40.446357+0000","last_peered":"2024-04-25T05:34:40.446357+0000","last_clean":"2024-04-25T05:34:40.446357+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:34:40.446357+0000","last_undegraded":"2024-04-25T05:34:40.446357+0000","last_fullsized":"2024-04-25T05:34:40.446357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446510+0000","last_change":"2024-04-25T05:34:02.700901+0000","last_active":"2024-04-25T05:34:40.446510+0000","last_peered":"2024-04-25T05:34:40.446510+0000","last_clean":"2024-04-25T05:34:40.446510+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:34:40.446510+0000","last_undegraded":"2024-04-25T05:34:40.446510+0000","last_fullsized":"2024-04-25T05:34:40.446510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.464318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447656+0000","last_change":"2024-04-25T05:33:58.685937+0000","last_active":"2024-04-25T05:34:40.447656+0000","last_peered":"2024-04-25T05:34:40.447656+0000","last_clean":"2024-04-25T05:34:40.447656+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:34:40.447656+0000","last_undegraded":"2024-04-25T05:34:40.447656+0000","last_fullsized":"2024-04-25T05:34:40.447656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.691268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447677+0000","last_change":"2024-04-25T05:34:00.696605+0000","last_active":"2024-04-25T05:34:40.447677+0000","last_peered":"2024-04-25T05:34:40.447677+0000","last_clean":"2024-04-25T05:34:40.447677+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:34:40.447677+0000","last_undegraded":"2024-04-25T05:34:40.447677+0000","last_fullsized":"2024-04-25T05:34:40.447677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.993568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661325+0000","last_change":"2024-04-25T05:34:04.722629+0000","last_active":"2024-04-25T05:34:39.661325+0000","last_peered":"2024-04-25T05:34:39.661325+0000","last_clean":"2024-04-25T05:34:39.661325+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:34:39.661325+0000","last_undegraded":"2024-04-25T05:34:39.661325+0000","last_fullsized":"2024-04-25T05:34:39.661325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.965741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447137+0000","last_change":"2024-04-25T05:34:00.675901+0000","last_active":"2024-04-25T05:34:40.447137+0000","last_peered":"2024-04-25T05:34:40.447137+0000","last_clean":"2024-04-25T05:34:40.447137+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:34:40.447137+0000","last_undegraded":"2024-04-25T05:34:40.447137+0000","last_fullsized":"2024-04-25T05:34:40.447137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.111301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661284+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:34:39.661284+0000","last_peered":"2024-04-25T05:34:39.661284+0000","last_clean":"2024-04-25T05:34:39.661284+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:34:39.661284+0000","last_undegraded":"2024-04-25T05:34:39.661284+0000","last_fullsized":"2024-04-25T05:34:39.661284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.968115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'34","reported_seq":104,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660701+0000","last_change":"2024-04-25T05:34:00.688687+0000","last_active":"2024-04-25T05:34:39.660701+0000","last_peered":"2024-04-25T05:34:39.660701+0000","last_clean":"2024-04-25T05:34:39.660701+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:34:39.660701+0000","last_undegraded":"2024-04-25T05:34:39.660701+0000","last_fullsized":"2024-04-25T05:34:39.660701+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.582841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447110+0000","last_change":"2024-04-25T05:34:02.708391+0000","last_active":"2024-04-25T05:34:40.447110+0000","last_peered":"2024-04-25T05:34:40.447110+0000","last_clean":"2024-04-25T05:34:40.447110+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:34:40.447110+0000","last_undegraded":"2024-04-25T05:34:40.447110+0000","last_fullsized":"2024-04-25T05:34:40.447110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.712592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448448+0000","last_change":"2024-04-25T05:34:04.760767+0000","last_active":"2024-04-25T05:34:40.448448+0000","last_peered":"2024-04-25T05:34:40.448448+0000","last_clean":"2024-04-25T05:34:40.448448+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:34:40.448448+0000","last_undegraded":"2024-04-25T05:34:40.448448+0000","last_fullsized":"2024-04-25T05:34:40.448448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.164337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660546+0000","last_change":"2024-04-25T05:33:58.679884+0000","last_active":"2024-04-25T05:34:39.660546+0000","last_peered":"2024-04-25T05:34:39.660546+0000","last_clean":"2024-04-25T05:34:39.660546+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:34:39.660546+0000","last_undegraded":"2024-04-25T05:34:39.660546+0000","last_fullsized":"2024-04-25T05:34:39.660546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.742024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447555+0000","last_change":"2024-04-25T05:34:00.685025+0000","last_active":"2024-04-25T05:34:40.447555+0000","last_peered":"2024-04-25T05:34:40.447555+0000","last_clean":"2024-04-25T05:34:40.447555+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:34:40.447555+0000","last_undegraded":"2024-04-25T05:34:40.447555+0000","last_fullsized":"2024-04-25T05:34:40.447555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.669540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":74,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.718420+0000","last_change":"2024-04-25T05:34:00.694199+0000","last_active":"2024-04-25T05:34:39.718420+0000","last_peered":"2024-04-25T05:34:39.718420+0000","last_clean":"2024-04-25T05:34:39.718420+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:34:39.718420+0000","last_undegraded":"2024-04-25T05:34:39.718420+0000","last_fullsized":"2024-04-25T05:34:39.718420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.968271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446560+0000","last_change":"2024-04-25T05:34:02.702812+0000","last_active":"2024-04-25T05:34:40.446560+0000","last_peered":"2024-04-25T05:34:40.446560+0000","last_clean":"2024-04-25T05:34:40.446560+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:34:40.446560+0000","last_undegraded":"2024-04-25T05:34:40.446560+0000","last_fullsized":"2024-04-25T05:34:40.446560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.890640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448554+0000","last_change":"2024-04-25T05:34:04.722965+0000","last_active":"2024-04-25T05:34:40.448554+0000","last_peered":"2024-04-25T05:34:40.448554+0000","last_clean":"2024-04-25T05:34:40.448554+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:34:40.448554+0000","last_undegraded":"2024-04-25T05:34:40.448554+0000","last_fullsized":"2024-04-25T05:34:40.448554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":290,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447135+0000","last_change":"2024-04-25T05:34:20.908770+0000","last_active":"2024-04-25T05:34:40.447135+0000","last_peered":"2024-04-25T05:34:40.447135+0000","last_clean":"2024-04-25T05:34:40.447135+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:34:40.447135+0000","last_undegraded":"2024-04-25T05:34:40.447135+0000","last_fullsized":"2024-04-25T05:34:40.447135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-25T05:34:20.908611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:34:20.908611+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.546403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661272+0000","last_change":"2024-04-25T05:33:58.679230+0000","last_active":"2024-04-25T05:34:39.661272+0000","last_peered":"2024-04-25T05:34:39.661272+0000","last_clean":"2024-04-25T05:34:39.661272+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:34:39.661272+0000","last_undegraded":"2024-04-25T05:34:39.661272+0000","last_fullsized":"2024-04-25T05:34:39.661272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.518886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448051+0000","last_change":"2024-04-25T05:34:01.002201+0000","last_active":"2024-04-25T05:34:40.448051+0000","last_peered":"2024-04-25T05:34:40.448051+0000","last_clean":"2024-04-25T05:34:40.448051+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:34:40.448051+0000","last_undegraded":"2024-04-25T05:34:40.448051+0000","last_fullsized":"2024-04-25T05:34:40.448051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.821400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448569+0000","last_change":"2024-04-25T05:34:02.701047+0000","last_active":"2024-04-25T05:34:40.448569+0000","last_peered":"2024-04-25T05:34:40.448569+0000","last_clean":"2024-04-25T05:34:40.448569+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:34:40.448569+0000","last_undegraded":"2024-04-25T05:34:40.448569+0000","last_fullsized":"2024-04-25T05:34:40.448569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.440947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660282+0000","last_change":"2024-04-25T05:34:04.726534+0000","last_active":"2024-04-25T05:34:39.660282+0000","last_peered":"2024-04-25T05:34:39.660282+0000","last_clean":"2024-04-25T05:34:39.660282+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:34:39.660282+0000","last_undegraded":"2024-04-25T05:34:39.660282+0000","last_fullsized":"2024-04-25T05:34:39.660282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446588+0000","last_change":"2024-04-25T05:33:58.682492+0000","last_active":"2024-04-25T05:34:40.446588+0000","last_peered":"2024-04-25T05:34:40.446588+0000","last_clean":"2024-04-25T05:34:40.446588+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:34:40.446588+0000","last_undegraded":"2024-04-25T05:34:40.446588+0000","last_fullsized":"2024-04-25T05:34:40.446588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.900372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":88,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.777769+0000","last_change":"2024-04-25T05:34:01.000301+0000","last_active":"2024-04-25T05:34:40.777769+0000","last_peered":"2024-04-25T05:34:40.777769+0000","last_clean":"2024-04-25T05:34:40.777769+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:34:40.777769+0000","last_undegraded":"2024-04-25T05:34:40.777769+0000","last_fullsized":"2024-04-25T05:34:40.777769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.243918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447101+0000","last_change":"2024-04-25T05:34:00.694619+0000","last_active":"2024-04-25T05:34:40.447101+0000","last_peered":"2024-04-25T05:34:40.447101+0000","last_clean":"2024-04-25T05:34:40.447101+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:34:40.447101+0000","last_undegraded":"2024-04-25T05:34:40.447101+0000","last_fullsized":"2024-04-25T05:34:40.447101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:01.840274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449044+0000","last_change":"2024-04-25T05:34:02.709328+0000","last_active":"2024-04-25T05:34:40.449044+0000","last_peered":"2024-04-25T05:34:40.449044+0000","last_clean":"2024-04-25T05:34:40.449044+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:34:40.449044+0000","last_undegraded":"2024-04-25T05:34:40.449044+0000","last_fullsized":"2024-04-25T05:34:40.449044+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.085576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660243+0000","last_change":"2024-04-25T05:34:04.759135+0000","last_active":"2024-04-25T05:34:39.660243+0000","last_peered":"2024-04-25T05:34:39.660243+0000","last_clean":"2024-04-25T05:34:39.660243+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:34:39.660243+0000","last_undegraded":"2024-04-25T05:34:39.660243+0000","last_fullsized":"2024-04-25T05:34:39.660243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660811+0000","last_change":"2024-04-25T05:33:58.677172+0000","last_active":"2024-04-25T05:34:39.660811+0000","last_peered":"2024-04-25T05:34:39.660811+0000","last_clean":"2024-04-25T05:34:39.660811+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:34:39.660811+0000","last_undegraded":"2024-04-25T05:34:39.660811+0000","last_fullsized":"2024-04-25T05:34:39.660811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.326701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447490+0000","last_change":"2024-04-25T05:34:00.999939+0000","last_active":"2024-04-25T05:34:40.447490+0000","last_peered":"2024-04-25T05:34:40.447490+0000","last_clean":"2024-04-25T05:34:40.447490+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:34:40.447490+0000","last_undegraded":"2024-04-25T05:34:40.447490+0000","last_fullsized":"2024-04-25T05:34:40.447490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.684394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449256+0000","last_change":"2024-04-25T05:34:02.708443+0000","last_active":"2024-04-25T05:34:40.449256+0000","last_peered":"2024-04-25T05:34:40.449256+0000","last_clean":"2024-04-25T05:34:40.449256+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:34:40.449256+0000","last_undegraded":"2024-04-25T05:34:40.449256+0000","last_fullsized":"2024-04-25T05:34:40.449256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.195589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446348+0000","last_change":"2024-04-25T05:34:04.728674+0000","last_active":"2024-04-25T05:34:40.446348+0000","last_peered":"2024-04-25T05:34:40.446348+0000","last_clean":"2024-04-25T05:34:40.446348+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:34:40.446348+0000","last_undegraded":"2024-04-25T05:34:40.446348+0000","last_fullsized":"2024-04-25T05:34:40.446348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.707833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449295+0000","last_change":"2024-04-25T05:33:58.678744+0000","last_active":"2024-04-25T05:34:40.449295+0000","last_peered":"2024-04-25T05:34:40.449295+0000","last_clean":"2024-04-25T05:34:40.449295+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:34:40.449295+0000","last_undegraded":"2024-04-25T05:34:40.449295+0000","last_fullsized":"2024-04-25T05:34:40.449295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.579714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661114+0000","last_change":"2024-04-25T05:34:00.688978+0000","last_active":"2024-04-25T05:34:39.661114+0000","last_peered":"2024-04-25T05:34:39.661114+0000","last_clean":"2024-04-25T05:34:39.661114+0000","last_became_active":"2024-04-25T05:34:00.688453+0000","last_became_peered":"2024-04-25T05:34:00.688453+0000","last_unstale":"2024-04-25T05:34:39.661114+0000","last_undegraded":"2024-04-25T05:34:39.661114+0000","last_fullsized":"2024-04-25T05:34:39.661114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.622999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447712+0000","last_change":"2024-04-25T05:34:02.708185+0000","last_active":"2024-04-25T05:34:40.447712+0000","last_peered":"2024-04-25T05:34:40.447712+0000","last_clean":"2024-04-25T05:34:40.447712+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:34:40.447712+0000","last_undegraded":"2024-04-25T05:34:40.447712+0000","last_fullsized":"2024-04-25T05:34:40.447712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.047597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448887+0000","last_change":"2024-04-25T05:34:04.761218+0000","last_active":"2024-04-25T05:34:40.448887+0000","last_peered":"2024-04-25T05:34:40.448887+0000","last_clean":"2024-04-25T05:34:40.448887+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:34:40.448887+0000","last_undegraded":"2024-04-25T05:34:40.448887+0000","last_fullsized":"2024-04-25T05:34:40.448887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.789438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447217+0000","last_change":"2024-04-25T05:33:58.682409+0000","last_active":"2024-04-25T05:34:40.447217+0000","last_peered":"2024-04-25T05:34:40.447217+0000","last_clean":"2024-04-25T05:34:40.447217+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:34:40.447217+0000","last_undegraded":"2024-04-25T05:34:40.447217+0000","last_fullsized":"2024-04-25T05:34:40.447217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661628+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:34:39.661628+0000","last_peered":"2024-04-25T05:34:39.661628+0000","last_clean":"2024-04-25T05:34:39.661628+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:34:39.661628+0000","last_undegraded":"2024-04-25T05:34:39.661628+0000","last_fullsized":"2024-04-25T05:34:39.661628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.826192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661001+0000","last_change":"2024-04-25T05:34:02.701774+0000","last_active":"2024-04-25T05:34:39.661001+0000","last_peered":"2024-04-25T05:34:39.661001+0000","last_clean":"2024-04-25T05:34:39.661001+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:34:39.661001+0000","last_undegraded":"2024-04-25T05:34:39.661001+0000","last_fullsized":"2024-04-25T05:34:39.661001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.422624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661265+0000","last_change":"2024-04-25T05:34:04.718464+0000","last_active":"2024-04-25T05:34:39.661265+0000","last_peered":"2024-04-25T05:34:39.661265+0000","last_clean":"2024-04-25T05:34:39.661265+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:34:39.661265+0000","last_undegraded":"2024-04-25T05:34:39.661265+0000","last_fullsized":"2024-04-25T05:34:39.661265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661746+0000","last_change":"2024-04-25T05:33:58.676147+0000","last_active":"2024-04-25T05:34:39.661746+0000","last_peered":"2024-04-25T05:34:39.661746+0000","last_clean":"2024-04-25T05:34:39.661746+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:34:39.661746+0000","last_undegraded":"2024-04-25T05:34:39.661746+0000","last_fullsized":"2024-04-25T05:34:39.661746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.509783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":91,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660915+0000","last_change":"2024-04-25T05:34:00.999824+0000","last_active":"2024-04-25T05:34:39.660915+0000","last_peered":"2024-04-25T05:34:39.660915+0000","last_clean":"2024-04-25T05:34:39.660915+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:34:39.660915+0000","last_undegraded":"2024-04-25T05:34:39.660915+0000","last_fullsized":"2024-04-25T05:34:39.660915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.441893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661138+0000","last_change":"2024-04-25T05:34:02.699472+0000","last_active":"2024-04-25T05:34:39.661138+0000","last_peered":"2024-04-25T05:34:39.661138+0000","last_clean":"2024-04-25T05:34:39.661138+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:34:39.661138+0000","last_undegraded":"2024-04-25T05:34:39.661138+0000","last_fullsized":"2024-04-25T05:34:39.661138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.651777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447057+0000","last_change":"2024-04-25T05:34:04.724152+0000","last_active":"2024-04-25T05:34:40.447057+0000","last_peered":"2024-04-25T05:34:40.447057+0000","last_clean":"2024-04-25T05:34:40.447057+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:34:40.447057+0000","last_undegraded":"2024-04-25T05:34:40.447057+0000","last_fullsized":"2024-04-25T05:34:40.447057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.599745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662278+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:34:39.662278+0000","last_peered":"2024-04-25T05:34:39.662278+0000","last_clean":"2024-04-25T05:34:39.662278+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:34:39.662278+0000","last_undegraded":"2024-04-25T05:34:39.662278+0000","last_fullsized":"2024-04-25T05:34:39.662278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.390939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.660826+0000","last_change":"2024-04-25T05:33:58.681579+0000","last_active":"2024-04-25T05:34:39.660826+0000","last_peered":"2024-04-25T05:34:39.660826+0000","last_clean":"2024-04-25T05:34:39.660826+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:34:39.660826+0000","last_undegraded":"2024-04-25T05:34:39.660826+0000","last_fullsized":"2024-04-25T05:34:39.660826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.706716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.718415+0000","last_change":"2024-04-25T05:34:00.691497+0000","last_active":"2024-04-25T05:34:39.718415+0000","last_peered":"2024-04-25T05:34:39.718415+0000","last_clean":"2024-04-25T05:34:39.718415+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:34:39.718415+0000","last_undegraded":"2024-04-25T05:34:39.718415+0000","last_fullsized":"2024-04-25T05:34:39.718415+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.049287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661404+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:34:39.661404+0000","last_peered":"2024-04-25T05:34:39.661404+0000","last_clean":"2024-04-25T05:34:39.661404+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:34:39.661404+0000","last_undegraded":"2024-04-25T05:34:39.661404+0000","last_fullsized":"2024-04-25T05:34:39.661404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:31.872714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448922+0000","last_change":"2024-04-25T05:34:02.699262+0000","last_active":"2024-04-25T05:34:40.448922+0000","last_peered":"2024-04-25T05:34:40.448922+0000","last_clean":"2024-04-25T05:34:40.448922+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:34:40.448922+0000","last_undegraded":"2024-04-25T05:34:40.448922+0000","last_fullsized":"2024-04-25T05:34:40.448922+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.861356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661135+0000","last_change":"2024-04-25T05:33:58.675050+0000","last_active":"2024-04-25T05:34:39.661135+0000","last_peered":"2024-04-25T05:34:39.661135+0000","last_clean":"2024-04-25T05:34:39.661135+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:34:39.661135+0000","last_undegraded":"2024-04-25T05:34:39.661135+0000","last_fullsized":"2024-04-25T05:34:39.661135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.277179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662124+0000","last_change":"2024-04-25T05:34:00.692677+0000","last_active":"2024-04-25T05:34:39.662124+0000","last_peered":"2024-04-25T05:34:39.662124+0000","last_clean":"2024-04-25T05:34:39.662124+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:34:39.662124+0000","last_undegraded":"2024-04-25T05:34:39.662124+0000","last_fullsized":"2024-04-25T05:34:39.662124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446565+0000","last_change":"2024-04-25T05:34:04.723087+0000","last_active":"2024-04-25T05:34:40.446565+0000","last_peered":"2024-04-25T05:34:40.446565+0000","last_clean":"2024-04-25T05:34:40.446565+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:34:40.446565+0000","last_undegraded":"2024-04-25T05:34:40.446565+0000","last_fullsized":"2024-04-25T05:34:40.446565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.795993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.718025+0000","last_change":"2024-04-25T05:34:02.708656+0000","last_active":"2024-04-25T05:34:39.718025+0000","last_peered":"2024-04-25T05:34:39.718025+0000","last_clean":"2024-04-25T05:34:39.718025+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:34:39.718025+0000","last_undegraded":"2024-04-25T05:34:39.718025+0000","last_fullsized":"2024-04-25T05:34:39.718025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.620659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449090+0000","last_change":"2024-04-25T05:33:58.684035+0000","last_active":"2024-04-25T05:34:40.449090+0000","last_peered":"2024-04-25T05:34:40.449090+0000","last_clean":"2024-04-25T05:34:40.449090+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:34:40.449090+0000","last_undegraded":"2024-04-25T05:34:40.449090+0000","last_fullsized":"2024-04-25T05:34:40.449090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.247558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662290+0000","last_change":"2024-04-25T05:34:00.692475+0000","last_active":"2024-04-25T05:34:39.662290+0000","last_peered":"2024-04-25T05:34:39.662290+0000","last_clean":"2024-04-25T05:34:39.662290+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:34:39.662290+0000","last_undegraded":"2024-04-25T05:34:39.662290+0000","last_fullsized":"2024-04-25T05:34:39.662290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.725418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662341+0000","last_change":"2024-04-25T05:34:04.719644+0000","last_active":"2024-04-25T05:34:39.662341+0000","last_peered":"2024-04-25T05:34:39.662341+0000","last_clean":"2024-04-25T05:34:39.662341+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:34:39.662341+0000","last_undegraded":"2024-04-25T05:34:39.662341+0000","last_fullsized":"2024-04-25T05:34:39.662341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447246+0000","last_change":"2024-04-25T05:34:02.705065+0000","last_active":"2024-04-25T05:34:40.447246+0000","last_peered":"2024-04-25T05:34:40.447246+0000","last_clean":"2024-04-25T05:34:40.447246+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:34:40.447246+0000","last_undegraded":"2024-04-25T05:34:40.447246+0000","last_fullsized":"2024-04-25T05:34:40.447246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.937446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661733+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:34:39.661733+0000","last_peered":"2024-04-25T05:34:39.661733+0000","last_clean":"2024-04-25T05:34:39.661733+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:34:39.661733+0000","last_undegraded":"2024-04-25T05:34:39.661733+0000","last_fullsized":"2024-04-25T05:34:39.661733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.422993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661663+0000","last_change":"2024-04-25T05:34:00.686532+0000","last_active":"2024-04-25T05:34:39.661663+0000","last_peered":"2024-04-25T05:34:39.661663+0000","last_clean":"2024-04-25T05:34:39.661663+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:34:39.661663+0000","last_undegraded":"2024-04-25T05:34:39.661663+0000","last_fullsized":"2024-04-25T05:34:39.661663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.818346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070478,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":890688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364818,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32428,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712724,"statfs":{"total":95995035648,"available":95961829376,"internally_reserved":0,"allocated":1277952,"data_stored":889590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":741376,"data_stored":299578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1294336,"data_stored":888955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31840,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713312,"statfs":{"total":95995035648,"available":95962431488,"internally_reserved":0,"allocated":675840,"data_stored":297954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":298684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":299404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":298588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:34:48.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:34:48.374 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:48.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:48.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: Updating smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:34:49.180 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:49.182 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:34:49.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:49 smithi043 ceph-mon[27010]: Deploying daemon alertmanager.a on smithi043 2024-04-25T05:34:49.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:49 smithi043 ceph-mon[27010]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:49.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:49 smithi043 ceph-mon[37492]: Deploying daemon alertmanager.a on smithi043 2024-04-25T05:34:49.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:49 smithi043 ceph-mon[37492]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:49 smithi060 ceph-mon[30898]: Deploying daemon alertmanager.a on smithi043 2024-04-25T05:34:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:49 smithi060 ceph-mon[30898]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:34:49.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:34:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:34:50.057 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-25T05:34:47.656299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1260,"num_read_kb":1074,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":235928,"kb_used_data":7288,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725288,"statfs":{"total":767960285184,"available":767718694912,"internally_reserved":0,"allocated":7462912,"data_stored":4195301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002048"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661166+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:34:39.661166+0000","last_peered":"2024-04-25T05:34:39.661166+0000","last_clean":"2024-04-25T05:34:39.661166+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:34:39.661166+0000","last_undegraded":"2024-04-25T05:34:39.661166+0000","last_fullsized":"2024-04-25T05:34:39.661166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.921834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662251+0000","last_change":"2024-04-25T05:34:02.700825+0000","last_active":"2024-04-25T05:34:44.662251+0000","last_peered":"2024-04-25T05:34:44.662251+0000","last_clean":"2024-04-25T05:34:44.662251+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:34:44.662251+0000","last_undegraded":"2024-04-25T05:34:44.662251+0000","last_fullsized":"2024-04-25T05:34:44.662251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.901521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447130+0000","last_change":"2024-04-25T05:33:58.677608+0000","last_active":"2024-04-25T05:34:40.447130+0000","last_peered":"2024-04-25T05:34:40.447130+0000","last_clean":"2024-04-25T05:34:40.447130+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:34:40.447130+0000","last_undegraded":"2024-04-25T05:34:40.447130+0000","last_fullsized":"2024-04-25T05:34:40.447130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.073153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661088+0000","last_change":"2024-04-25T05:34:00.692628+0000","last_active":"2024-04-25T05:34:39.661088+0000","last_peered":"2024-04-25T05:34:39.661088+0000","last_clean":"2024-04-25T05:34:39.661088+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:34:39.661088+0000","last_undegraded":"2024-04-25T05:34:39.661088+0000","last_fullsized":"2024-04-25T05:34:39.661088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.003557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663768+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:34:44.663768+0000","last_peered":"2024-04-25T05:34:44.663768+0000","last_clean":"2024-04-25T05:34:44.663768+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:34:44.663768+0000","last_undegraded":"2024-04-25T05:34:44.663768+0000","last_fullsized":"2024-04-25T05:34:44.663768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.498112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661088+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:34:39.661088+0000","last_peered":"2024-04-25T05:34:39.661088+0000","last_clean":"2024-04-25T05:34:39.661088+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:34:39.661088+0000","last_undegraded":"2024-04-25T05:34:39.661088+0000","last_fullsized":"2024-04-25T05:34:39.661088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.555566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664284+0000","last_change":"2024-04-25T05:33:58.679726+0000","last_active":"2024-04-25T05:34:44.664284+0000","last_peered":"2024-04-25T05:34:44.664284+0000","last_clean":"2024-04-25T05:34:44.664284+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:34:44.664284+0000","last_undegraded":"2024-04-25T05:34:44.664284+0000","last_fullsized":"2024-04-25T05:34:44.664284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.377018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662666+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:34:44.662666+0000","last_peered":"2024-04-25T05:34:44.662666+0000","last_clean":"2024-04-25T05:34:44.662666+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:34:44.662666+0000","last_undegraded":"2024-04-25T05:34:44.662666+0000","last_fullsized":"2024-04-25T05:34:44.662666+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.111490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446558+0000","last_change":"2024-04-25T05:34:04.724350+0000","last_active":"2024-04-25T05:34:40.446558+0000","last_peered":"2024-04-25T05:34:40.446558+0000","last_clean":"2024-04-25T05:34:40.446558+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:34:40.446558+0000","last_undegraded":"2024-04-25T05:34:40.446558+0000","last_fullsized":"2024-04-25T05:34:40.446558+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.943468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063851+0000","last_change":"2024-04-25T05:34:02.709504+0000","last_active":"2024-04-25T05:34:45.063851+0000","last_peered":"2024-04-25T05:34:45.063851+0000","last_clean":"2024-04-25T05:34:45.063851+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:34:45.063851+0000","last_undegraded":"2024-04-25T05:34:45.063851+0000","last_fullsized":"2024-04-25T05:34:45.063851+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.151709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064001+0000","last_change":"2024-04-25T05:33:58.687700+0000","last_active":"2024-04-25T05:34:45.064001+0000","last_peered":"2024-04-25T05:34:45.064001+0000","last_clean":"2024-04-25T05:34:45.064001+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:34:45.064001+0000","last_undegraded":"2024-04-25T05:34:45.064001+0000","last_fullsized":"2024-04-25T05:34:45.064001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.471958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":76,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661416+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:34:39.661416+0000","last_peered":"2024-04-25T05:34:39.661416+0000","last_clean":"2024-04-25T05:34:39.661416+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:34:39.661416+0000","last_undegraded":"2024-04-25T05:34:39.661416+0000","last_fullsized":"2024-04-25T05:34:39.661416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.181248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062223+0000","last_change":"2024-04-25T05:34:04.722769+0000","last_active":"2024-04-25T05:34:45.062223+0000","last_peered":"2024-04-25T05:34:45.062223+0000","last_clean":"2024-04-25T05:34:45.062223+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:34:45.062223+0000","last_undegraded":"2024-04-25T05:34:45.062223+0000","last_fullsized":"2024-04-25T05:34:45.062223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.673388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448466+0000","last_change":"2024-04-25T05:34:02.701665+0000","last_active":"2024-04-25T05:34:40.448466+0000","last_peered":"2024-04-25T05:34:40.448466+0000","last_clean":"2024-04-25T05:34:40.448466+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:34:40.448466+0000","last_undegraded":"2024-04-25T05:34:40.448466+0000","last_fullsized":"2024-04-25T05:34:40.448466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.814291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063788+0000","last_change":"2024-04-25T05:33:58.688919+0000","last_active":"2024-04-25T05:34:45.063788+0000","last_peered":"2024-04-25T05:34:45.063788+0000","last_clean":"2024-04-25T05:34:45.063788+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:34:45.063788+0000","last_undegraded":"2024-04-25T05:34:45.063788+0000","last_fullsized":"2024-04-25T05:34:45.063788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.028583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661513+0000","last_change":"2024-04-25T05:34:00.693330+0000","last_active":"2024-04-25T05:34:39.661513+0000","last_peered":"2024-04-25T05:34:39.661513+0000","last_clean":"2024-04-25T05:34:39.661513+0000","last_became_active":"2024-04-25T05:34:00.693003+0000","last_became_peered":"2024-04-25T05:34:00.693003+0000","last_unstale":"2024-04-25T05:34:39.661513+0000","last_undegraded":"2024-04-25T05:34:39.661513+0000","last_fullsized":"2024-04-25T05:34:39.661513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.762449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662183+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:34:39.662183+0000","last_peered":"2024-04-25T05:34:39.662183+0000","last_clean":"2024-04-25T05:34:39.662183+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:34:39.662183+0000","last_undegraded":"2024-04-25T05:34:39.662183+0000","last_fullsized":"2024-04-25T05:34:39.662183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.388118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662766+0000","last_change":"2024-04-25T05:34:02.708603+0000","last_active":"2024-04-25T05:34:44.662766+0000","last_peered":"2024-04-25T05:34:44.662766+0000","last_clean":"2024-04-25T05:34:44.662766+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:34:44.662766+0000","last_undegraded":"2024-04-25T05:34:44.662766+0000","last_fullsized":"2024-04-25T05:34:44.662766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.908190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447588+0000","last_change":"2024-04-25T05:33:58.683934+0000","last_active":"2024-04-25T05:34:40.447588+0000","last_peered":"2024-04-25T05:34:40.447588+0000","last_clean":"2024-04-25T05:34:40.447588+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:34:40.447588+0000","last_undegraded":"2024-04-25T05:34:40.447588+0000","last_fullsized":"2024-04-25T05:34:40.447588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.927389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896223+0000","last_change":"2024-04-25T05:34:00.690831+0000","last_active":"2024-04-25T05:34:44.896223+0000","last_peered":"2024-04-25T05:34:44.896223+0000","last_clean":"2024-04-25T05:34:44.896223+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:34:44.896223+0000","last_undegraded":"2024-04-25T05:34:44.896223+0000","last_fullsized":"2024-04-25T05:34:44.896223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.940787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664451+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:34:44.664451+0000","last_peered":"2024-04-25T05:34:44.664451+0000","last_clean":"2024-04-25T05:34:44.664451+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:34:44.664451+0000","last_undegraded":"2024-04-25T05:34:44.664451+0000","last_fullsized":"2024-04-25T05:34:44.664451+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661601+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:34:39.661601+0000","last_peered":"2024-04-25T05:34:39.661601+0000","last_clean":"2024-04-25T05:34:39.661601+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:34:39.661601+0000","last_undegraded":"2024-04-25T05:34:39.661601+0000","last_fullsized":"2024-04-25T05:34:39.661601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.792766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661499+0000","last_change":"2024-04-25T05:33:58.677738+0000","last_active":"2024-04-25T05:34:39.661499+0000","last_peered":"2024-04-25T05:34:39.661499+0000","last_clean":"2024-04-25T05:34:39.661499+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:34:39.661499+0000","last_undegraded":"2024-04-25T05:34:39.661499+0000","last_fullsized":"2024-04-25T05:34:39.661499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.705248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":72,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664031+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:34:44.664031+0000","last_peered":"2024-04-25T05:34:44.664031+0000","last_clean":"2024-04-25T05:34:44.664031+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:34:44.664031+0000","last_undegraded":"2024-04-25T05:34:44.664031+0000","last_fullsized":"2024-04-25T05:34:44.664031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.537702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447918+0000","last_change":"2024-04-25T05:34:04.725886+0000","last_active":"2024-04-25T05:34:40.447918+0000","last_peered":"2024-04-25T05:34:40.447918+0000","last_clean":"2024-04-25T05:34:40.447918+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:34:40.447918+0000","last_undegraded":"2024-04-25T05:34:40.447918+0000","last_fullsized":"2024-04-25T05:34:40.447918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063892+0000","last_change":"2024-04-25T05:34:02.701003+0000","last_active":"2024-04-25T05:34:45.063892+0000","last_peered":"2024-04-25T05:34:45.063892+0000","last_clean":"2024-04-25T05:34:45.063892+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:34:45.063892+0000","last_undegraded":"2024-04-25T05:34:45.063892+0000","last_fullsized":"2024-04-25T05:34:45.063892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.795737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662279+0000","last_change":"2024-04-25T05:33:58.678204+0000","last_active":"2024-04-25T05:34:44.662279+0000","last_peered":"2024-04-25T05:34:44.662279+0000","last_clean":"2024-04-25T05:34:44.662279+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:34:44.662279+0000","last_undegraded":"2024-04-25T05:34:44.662279+0000","last_fullsized":"2024-04-25T05:34:44.662279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.231648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447222+0000","last_change":"2024-04-25T05:34:01.002306+0000","last_active":"2024-04-25T05:34:40.447222+0000","last_peered":"2024-04-25T05:34:40.447222+0000","last_clean":"2024-04-25T05:34:40.447222+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:34:40.447222+0000","last_undegraded":"2024-04-25T05:34:40.447222+0000","last_fullsized":"2024-04-25T05:34:40.447222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.875946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662473+0000","last_change":"2024-04-25T05:34:04.722159+0000","last_active":"2024-04-25T05:34:44.662473+0000","last_peered":"2024-04-25T05:34:44.662473+0000","last_clean":"2024-04-25T05:34:44.662473+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:34:44.662473+0000","last_undegraded":"2024-04-25T05:34:44.662473+0000","last_fullsized":"2024-04-25T05:34:44.662473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.853709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447421+0000","last_change":"2024-04-25T05:34:02.710777+0000","last_active":"2024-04-25T05:34:40.447421+0000","last_peered":"2024-04-25T05:34:40.447421+0000","last_clean":"2024-04-25T05:34:40.447421+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:34:40.447421+0000","last_undegraded":"2024-04-25T05:34:40.447421+0000","last_fullsized":"2024-04-25T05:34:40.447421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.845754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662927+0000","last_change":"2024-04-25T05:33:58.676498+0000","last_active":"2024-04-25T05:34:44.662927+0000","last_peered":"2024-04-25T05:34:44.662927+0000","last_clean":"2024-04-25T05:34:44.662927+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:34:44.662927+0000","last_undegraded":"2024-04-25T05:34:44.662927+0000","last_fullsized":"2024-04-25T05:34:44.662927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.714656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063777+0000","last_change":"2024-04-25T05:34:00.691266+0000","last_active":"2024-04-25T05:34:45.063777+0000","last_peered":"2024-04-25T05:34:45.063777+0000","last_clean":"2024-04-25T05:34:45.063777+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:34:45.063777+0000","last_undegraded":"2024-04-25T05:34:45.063777+0000","last_fullsized":"2024-04-25T05:34:45.063777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.730290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448060+0000","last_change":"2024-04-25T05:34:04.725684+0000","last_active":"2024-04-25T05:34:40.448060+0000","last_peered":"2024-04-25T05:34:40.448060+0000","last_clean":"2024-04-25T05:34:40.448060+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:34:40.448060+0000","last_undegraded":"2024-04-25T05:34:40.448060+0000","last_fullsized":"2024-04-25T05:34:40.448060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.515944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062203+0000","last_change":"2024-04-25T05:34:02.701292+0000","last_active":"2024-04-25T05:34:45.062203+0000","last_peered":"2024-04-25T05:34:45.062203+0000","last_clean":"2024-04-25T05:34:45.062203+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:34:45.062203+0000","last_undegraded":"2024-04-25T05:34:45.062203+0000","last_fullsized":"2024-04-25T05:34:45.062203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447095+0000","last_change":"2024-04-25T05:33:58.678452+0000","last_active":"2024-04-25T05:34:40.447095+0000","last_peered":"2024-04-25T05:34:40.447095+0000","last_clean":"2024-04-25T05:34:40.447095+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:34:40.447095+0000","last_undegraded":"2024-04-25T05:34:40.447095+0000","last_fullsized":"2024-04-25T05:34:40.447095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.043078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663040+0000","last_change":"2024-04-25T05:34:00.692546+0000","last_active":"2024-04-25T05:34:44.663040+0000","last_peered":"2024-04-25T05:34:44.663040+0000","last_clean":"2024-04-25T05:34:44.663040+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:34:44.663040+0000","last_undegraded":"2024-04-25T05:34:44.663040+0000","last_fullsized":"2024-04-25T05:34:44.663040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.114492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663636+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:34:44.663636+0000","last_peered":"2024-04-25T05:34:44.663636+0000","last_clean":"2024-04-25T05:34:44.663636+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:34:44.663636+0000","last_undegraded":"2024-04-25T05:34:44.663636+0000","last_fullsized":"2024-04-25T05:34:44.663636+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.872916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448376+0000","last_change":"2024-04-25T05:34:02.701784+0000","last_active":"2024-04-25T05:34:40.448376+0000","last_peered":"2024-04-25T05:34:40.448376+0000","last_clean":"2024-04-25T05:34:40.448376+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:34:40.448376+0000","last_undegraded":"2024-04-25T05:34:40.448376+0000","last_fullsized":"2024-04-25T05:34:40.448376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.034685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064131+0000","last_change":"2024-04-25T05:33:58.679682+0000","last_active":"2024-04-25T05:34:45.064131+0000","last_peered":"2024-04-25T05:34:45.064131+0000","last_clean":"2024-04-25T05:34:45.064131+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:34:45.064131+0000","last_undegraded":"2024-04-25T05:34:45.064131+0000","last_fullsized":"2024-04-25T05:34:45.064131+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662567+0000","last_change":"2024-04-25T05:34:00.999704+0000","last_active":"2024-04-25T05:34:44.662567+0000","last_peered":"2024-04-25T05:34:44.662567+0000","last_clean":"2024-04-25T05:34:44.662567+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:34:44.662567+0000","last_undegraded":"2024-04-25T05:34:44.662567+0000","last_fullsized":"2024-04-25T05:34:44.662567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.051072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446242+0000","last_change":"2024-04-25T05:34:04.727071+0000","last_active":"2024-04-25T05:34:40.446242+0000","last_peered":"2024-04-25T05:34:40.446242+0000","last_clean":"2024-04-25T05:34:40.446242+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:34:40.446242+0000","last_undegraded":"2024-04-25T05:34:40.446242+0000","last_fullsized":"2024-04-25T05:34:40.446242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.039265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663894+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:34:44.663894+0000","last_peered":"2024-04-25T05:34:44.663894+0000","last_clean":"2024-04-25T05:34:44.663894+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:34:44.663894+0000","last_undegraded":"2024-04-25T05:34:44.663894+0000","last_fullsized":"2024-04-25T05:34:44.663894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.038248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447963+0000","last_change":"2024-04-25T05:33:58.687406+0000","last_active":"2024-04-25T05:34:40.447963+0000","last_peered":"2024-04-25T05:34:40.447963+0000","last_clean":"2024-04-25T05:34:40.447963+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:34:40.447963+0000","last_undegraded":"2024-04-25T05:34:40.447963+0000","last_fullsized":"2024-04-25T05:34:40.447963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.844471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663076+0000","last_change":"2024-04-25T05:34:00.689589+0000","last_active":"2024-04-25T05:34:44.663076+0000","last_peered":"2024-04-25T05:34:44.663076+0000","last_clean":"2024-04-25T05:34:44.663076+0000","last_became_active":"2024-04-25T05:34:00.688897+0000","last_became_peered":"2024-04-25T05:34:00.688897+0000","last_unstale":"2024-04-25T05:34:44.663076+0000","last_undegraded":"2024-04-25T05:34:44.663076+0000","last_fullsized":"2024-04-25T05:34:44.663076+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662249+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:34:44.662249+0000","last_peered":"2024-04-25T05:34:44.662249+0000","last_clean":"2024-04-25T05:34:44.662249+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:34:44.662249+0000","last_undegraded":"2024-04-25T05:34:44.662249+0000","last_fullsized":"2024-04-25T05:34:44.662249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.871648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662797+0000","last_change":"2024-04-25T05:34:02.699322+0000","last_active":"2024-04-25T05:34:44.662797+0000","last_peered":"2024-04-25T05:34:44.662797+0000","last_clean":"2024-04-25T05:34:44.662797+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:34:44.662797+0000","last_undegraded":"2024-04-25T05:34:44.662797+0000","last_fullsized":"2024-04-25T05:34:44.662797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.105773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661879+0000","last_change":"2024-04-25T05:33:58.680453+0000","last_active":"2024-04-25T05:34:39.661879+0000","last_peered":"2024-04-25T05:34:39.661879+0000","last_clean":"2024-04-25T05:34:39.661879+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:34:39.661879+0000","last_undegraded":"2024-04-25T05:34:39.661879+0000","last_fullsized":"2024-04-25T05:34:39.661879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.585516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661773+0000","last_change":"2024-04-25T05:34:00.693320+0000","last_active":"2024-04-25T05:34:39.661773+0000","last_peered":"2024-04-25T05:34:39.661773+0000","last_clean":"2024-04-25T05:34:39.661773+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:34:39.661773+0000","last_undegraded":"2024-04-25T05:34:39.661773+0000","last_fullsized":"2024-04-25T05:34:39.661773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.690705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446999+0000","last_change":"2024-04-25T05:34:04.724697+0000","last_active":"2024-04-25T05:34:40.446999+0000","last_peered":"2024-04-25T05:34:40.446999+0000","last_clean":"2024-04-25T05:34:40.446999+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:34:40.446999+0000","last_undegraded":"2024-04-25T05:34:40.446999+0000","last_fullsized":"2024-04-25T05:34:40.446999+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.282641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063891+0000","last_change":"2024-04-25T05:34:02.708587+0000","last_active":"2024-04-25T05:34:45.063891+0000","last_peered":"2024-04-25T05:34:45.063891+0000","last_clean":"2024-04-25T05:34:45.063891+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:34:45.063891+0000","last_undegraded":"2024-04-25T05:34:45.063891+0000","last_fullsized":"2024-04-25T05:34:45.063891+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.470304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448433+0000","last_change":"2024-04-25T05:33:58.685869+0000","last_active":"2024-04-25T05:34:40.448433+0000","last_peered":"2024-04-25T05:34:40.448433+0000","last_clean":"2024-04-25T05:34:40.448433+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:34:40.448433+0000","last_undegraded":"2024-04-25T05:34:40.448433+0000","last_fullsized":"2024-04-25T05:34:40.448433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.260635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663931+0000","last_change":"2024-04-25T05:34:00.999737+0000","last_active":"2024-04-25T05:34:44.663931+0000","last_peered":"2024-04-25T05:34:44.663931+0000","last_clean":"2024-04-25T05:34:44.663931+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:34:44.663931+0000","last_undegraded":"2024-04-25T05:34:44.663931+0000","last_fullsized":"2024-04-25T05:34:44.663931+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.305882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662507+0000","last_change":"2024-04-25T05:34:04.715688+0000","last_active":"2024-04-25T05:34:39.662507+0000","last_peered":"2024-04-25T05:34:39.662507+0000","last_clean":"2024-04-25T05:34:39.662507+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:34:39.662507+0000","last_undegraded":"2024-04-25T05:34:39.662507+0000","last_fullsized":"2024-04-25T05:34:39.662507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.959881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664365+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:34:44.664365+0000","last_peered":"2024-04-25T05:34:44.664365+0000","last_clean":"2024-04-25T05:34:44.664365+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:34:44.664365+0000","last_undegraded":"2024-04-25T05:34:44.664365+0000","last_fullsized":"2024-04-25T05:34:44.664365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.077903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446793+0000","last_change":"2024-04-25T05:33:58.685955+0000","last_active":"2024-04-25T05:34:40.446793+0000","last_peered":"2024-04-25T05:34:40.446793+0000","last_clean":"2024-04-25T05:34:40.446793+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:34:40.446793+0000","last_undegraded":"2024-04-25T05:34:40.446793+0000","last_fullsized":"2024-04-25T05:34:40.446793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.649756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064280+0000","last_change":"2024-04-25T05:34:01.002652+0000","last_active":"2024-04-25T05:34:45.064280+0000","last_peered":"2024-04-25T05:34:45.064280+0000","last_clean":"2024-04-25T05:34:45.064280+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:34:45.064280+0000","last_undegraded":"2024-04-25T05:34:45.064280+0000","last_fullsized":"2024-04-25T05:34:45.064280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.164431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.895731+0000","last_change":"2024-04-25T05:34:04.719603+0000","last_active":"2024-04-25T05:34:44.895731+0000","last_peered":"2024-04-25T05:34:44.895731+0000","last_clean":"2024-04-25T05:34:44.895731+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:34:44.895731+0000","last_undegraded":"2024-04-25T05:34:44.895731+0000","last_fullsized":"2024-04-25T05:34:44.895731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.818750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664420+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:34:44.664420+0000","last_peered":"2024-04-25T05:34:44.664420+0000","last_clean":"2024-04-25T05:34:44.664420+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:34:44.664420+0000","last_undegraded":"2024-04-25T05:34:44.664420+0000","last_fullsized":"2024-04-25T05:34:44.664420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.004188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448684+0000","last_change":"2024-04-25T05:33:58.685792+0000","last_active":"2024-04-25T05:34:40.448684+0000","last_peered":"2024-04-25T05:34:40.448684+0000","last_clean":"2024-04-25T05:34:40.448684+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:34:40.448684+0000","last_undegraded":"2024-04-25T05:34:40.448684+0000","last_fullsized":"2024-04-25T05:34:40.448684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.497320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447321+0000","last_change":"2024-04-25T05:34:01.002399+0000","last_active":"2024-04-25T05:34:40.447321+0000","last_peered":"2024-04-25T05:34:40.447321+0000","last_clean":"2024-04-25T05:34:40.447321+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:34:40.447321+0000","last_undegraded":"2024-04-25T05:34:40.447321+0000","last_fullsized":"2024-04-25T05:34:40.447321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.611153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447838+0000","last_change":"2024-04-25T05:34:04.760885+0000","last_active":"2024-04-25T05:34:40.447838+0000","last_peered":"2024-04-25T05:34:40.447838+0000","last_clean":"2024-04-25T05:34:40.447838+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:34:40.447838+0000","last_undegraded":"2024-04-25T05:34:40.447838+0000","last_fullsized":"2024-04-25T05:34:40.447838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.397373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064132+0000","last_change":"2024-04-25T05:34:02.700150+0000","last_active":"2024-04-25T05:34:45.064132+0000","last_peered":"2024-04-25T05:34:45.064132+0000","last_clean":"2024-04-25T05:34:45.064132+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:34:45.064132+0000","last_undegraded":"2024-04-25T05:34:45.064132+0000","last_fullsized":"2024-04-25T05:34:45.064132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.726607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062179+0000","last_change":"2024-04-25T05:33:58.682812+0000","last_active":"2024-04-25T05:34:45.062179+0000","last_peered":"2024-04-25T05:34:45.062179+0000","last_clean":"2024-04-25T05:34:45.062179+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:34:45.062179+0000","last_undegraded":"2024-04-25T05:34:45.062179+0000","last_fullsized":"2024-04-25T05:34:45.062179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.737056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664526+0000","last_change":"2024-04-25T05:34:01.000303+0000","last_active":"2024-04-25T05:34:44.664526+0000","last_peered":"2024-04-25T05:34:44.664526+0000","last_clean":"2024-04-25T05:34:44.664526+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:34:44.664526+0000","last_undegraded":"2024-04-25T05:34:44.664526+0000","last_fullsized":"2024-04-25T05:34:44.664526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.779729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063712+0000","last_change":"2024-04-25T05:34:04.759369+0000","last_active":"2024-04-25T05:34:45.063712+0000","last_peered":"2024-04-25T05:34:45.063712+0000","last_clean":"2024-04-25T05:34:45.063712+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:34:45.063712+0000","last_undegraded":"2024-04-25T05:34:45.063712+0000","last_fullsized":"2024-04-25T05:34:45.063712+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.510841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448600+0000","last_change":"2024-04-25T05:33:58.689200+0000","last_active":"2024-04-25T05:34:40.448600+0000","last_peered":"2024-04-25T05:34:40.448600+0000","last_clean":"2024-04-25T05:34:40.448600+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:34:40.448600+0000","last_undegraded":"2024-04-25T05:34:40.448600+0000","last_fullsized":"2024-04-25T05:34:40.448600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.468188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":86,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063231+0000","last_change":"2024-04-25T05:34:01.000840+0000","last_active":"2024-04-25T05:34:45.063231+0000","last_peered":"2024-04-25T05:34:45.063231+0000","last_clean":"2024-04-25T05:34:45.063231+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:34:45.063231+0000","last_undegraded":"2024-04-25T05:34:45.063231+0000","last_fullsized":"2024-04-25T05:34:45.063231+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.859671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448963+0000","last_change":"2024-04-25T05:34:02.701548+0000","last_active":"2024-04-25T05:34:40.448963+0000","last_peered":"2024-04-25T05:34:40.448963+0000","last_clean":"2024-04-25T05:34:40.448963+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:34:40.448963+0000","last_undegraded":"2024-04-25T05:34:40.448963+0000","last_fullsized":"2024-04-25T05:34:40.448963+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.747443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663758+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:34:44.663758+0000","last_peered":"2024-04-25T05:34:44.663758+0000","last_clean":"2024-04-25T05:34:44.663758+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:34:44.663758+0000","last_undegraded":"2024-04-25T05:34:44.663758+0000","last_fullsized":"2024-04-25T05:34:44.663758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.987326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.449186+0000","last_change":"2024-04-25T05:33:58.687364+0000","last_active":"2024-04-25T05:34:40.449186+0000","last_peered":"2024-04-25T05:34:40.449186+0000","last_clean":"2024-04-25T05:34:40.449186+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:34:40.449186+0000","last_undegraded":"2024-04-25T05:34:40.449186+0000","last_fullsized":"2024-04-25T05:34:40.449186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.749177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":82,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662016+0000","last_change":"2024-04-25T05:34:00.694001+0000","last_active":"2024-04-25T05:34:39.662016+0000","last_peered":"2024-04-25T05:34:39.662016+0000","last_clean":"2024-04-25T05:34:39.662016+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:34:39.662016+0000","last_undegraded":"2024-04-25T05:34:39.662016+0000","last_fullsized":"2024-04-25T05:34:39.662016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.293624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.895977+0000","last_change":"2024-04-25T05:34:02.710703+0000","last_active":"2024-04-25T05:34:44.895977+0000","last_peered":"2024-04-25T05:34:44.895977+0000","last_clean":"2024-04-25T05:34:44.895977+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:34:44.895977+0000","last_undegraded":"2024-04-25T05:34:44.895977+0000","last_fullsized":"2024-04-25T05:34:44.895977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.457617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446897+0000","last_change":"2024-04-25T05:34:04.727176+0000","last_active":"2024-04-25T05:34:40.446897+0000","last_peered":"2024-04-25T05:34:40.446897+0000","last_clean":"2024-04-25T05:34:40.446897+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:34:40.446897+0000","last_undegraded":"2024-04-25T05:34:40.446897+0000","last_fullsized":"2024-04-25T05:34:40.446897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896191+0000","last_change":"2024-04-25T05:34:02.710395+0000","last_active":"2024-04-25T05:34:44.896191+0000","last_peered":"2024-04-25T05:34:44.896191+0000","last_clean":"2024-04-25T05:34:44.896191+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:34:44.896191+0000","last_undegraded":"2024-04-25T05:34:44.896191+0000","last_fullsized":"2024-04-25T05:34:44.896191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.749912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448240+0000","last_change":"2024-04-25T05:33:58.686463+0000","last_active":"2024-04-25T05:34:40.448240+0000","last_peered":"2024-04-25T05:34:40.448240+0000","last_clean":"2024-04-25T05:34:40.448240+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:34:40.448240+0000","last_undegraded":"2024-04-25T05:34:40.448240+0000","last_fullsized":"2024-04-25T05:34:40.448240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:59.578656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":80,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062714+0000","last_change":"2024-04-25T05:34:00.999892+0000","last_active":"2024-04-25T05:34:45.062714+0000","last_peered":"2024-04-25T05:34:45.062714+0000","last_clean":"2024-04-25T05:34:45.062714+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:34:45.062714+0000","last_undegraded":"2024-04-25T05:34:45.062714+0000","last_fullsized":"2024-04-25T05:34:45.062714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.466514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446357+0000","last_change":"2024-04-25T05:34:04.725833+0000","last_active":"2024-04-25T05:34:40.446357+0000","last_peered":"2024-04-25T05:34:40.446357+0000","last_clean":"2024-04-25T05:34:40.446357+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:34:40.446357+0000","last_undegraded":"2024-04-25T05:34:40.446357+0000","last_fullsized":"2024-04-25T05:34:40.446357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896169+0000","last_change":"2024-04-25T05:34:02.700901+0000","last_active":"2024-04-25T05:34:44.896169+0000","last_peered":"2024-04-25T05:34:44.896169+0000","last_clean":"2024-04-25T05:34:44.896169+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:34:44.896169+0000","last_undegraded":"2024-04-25T05:34:44.896169+0000","last_fullsized":"2024-04-25T05:34:44.896169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.464318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063893+0000","last_change":"2024-04-25T05:33:58.685937+0000","last_active":"2024-04-25T05:34:45.063893+0000","last_peered":"2024-04-25T05:34:45.063893+0000","last_clean":"2024-04-25T05:34:45.063893+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:34:45.063893+0000","last_undegraded":"2024-04-25T05:34:45.063893+0000","last_fullsized":"2024-04-25T05:34:45.063893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.691268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447677+0000","last_change":"2024-04-25T05:34:00.696605+0000","last_active":"2024-04-25T05:34:40.447677+0000","last_peered":"2024-04-25T05:34:40.447677+0000","last_clean":"2024-04-25T05:34:40.447677+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:34:40.447677+0000","last_undegraded":"2024-04-25T05:34:40.447677+0000","last_fullsized":"2024-04-25T05:34:40.447677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.993568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664104+0000","last_change":"2024-04-25T05:34:04.722629+0000","last_active":"2024-04-25T05:34:44.664104+0000","last_peered":"2024-04-25T05:34:44.664104+0000","last_clean":"2024-04-25T05:34:44.664104+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:34:44.664104+0000","last_undegraded":"2024-04-25T05:34:44.664104+0000","last_fullsized":"2024-04-25T05:34:44.664104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.965741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063467+0000","last_change":"2024-04-25T05:34:00.675901+0000","last_active":"2024-04-25T05:34:45.063467+0000","last_peered":"2024-04-25T05:34:45.063467+0000","last_clean":"2024-04-25T05:34:45.063467+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:34:45.063467+0000","last_undegraded":"2024-04-25T05:34:45.063467+0000","last_fullsized":"2024-04-25T05:34:45.063467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.111301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661284+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:34:39.661284+0000","last_peered":"2024-04-25T05:34:39.661284+0000","last_clean":"2024-04-25T05:34:39.661284+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:34:39.661284+0000","last_undegraded":"2024-04-25T05:34:39.661284+0000","last_fullsized":"2024-04-25T05:34:39.661284+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.968115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'34","reported_seq":107,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662626+0000","last_change":"2024-04-25T05:34:00.688687+0000","last_active":"2024-04-25T05:34:44.662626+0000","last_peered":"2024-04-25T05:34:44.662626+0000","last_clean":"2024-04-25T05:34:44.662626+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:34:44.662626+0000","last_undegraded":"2024-04-25T05:34:44.662626+0000","last_fullsized":"2024-04-25T05:34:44.662626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.582841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447110+0000","last_change":"2024-04-25T05:34:02.708391+0000","last_active":"2024-04-25T05:34:40.447110+0000","last_peered":"2024-04-25T05:34:40.447110+0000","last_clean":"2024-04-25T05:34:40.447110+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:34:40.447110+0000","last_undegraded":"2024-04-25T05:34:40.447110+0000","last_fullsized":"2024-04-25T05:34:40.447110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.712592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448448+0000","last_change":"2024-04-25T05:34:04.760767+0000","last_active":"2024-04-25T05:34:40.448448+0000","last_peered":"2024-04-25T05:34:40.448448+0000","last_clean":"2024-04-25T05:34:40.448448+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:34:40.448448+0000","last_undegraded":"2024-04-25T05:34:40.448448+0000","last_fullsized":"2024-04-25T05:34:40.448448+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.164337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662108+0000","last_change":"2024-04-25T05:33:58.679884+0000","last_active":"2024-04-25T05:34:44.662108+0000","last_peered":"2024-04-25T05:34:44.662108+0000","last_clean":"2024-04-25T05:34:44.662108+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:34:44.662108+0000","last_undegraded":"2024-04-25T05:34:44.662108+0000","last_fullsized":"2024-04-25T05:34:44.662108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.742024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896026+0000","last_change":"2024-04-25T05:34:00.685025+0000","last_active":"2024-04-25T05:34:44.896026+0000","last_peered":"2024-04-25T05:34:44.896026+0000","last_clean":"2024-04-25T05:34:44.896026+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:34:44.896026+0000","last_undegraded":"2024-04-25T05:34:44.896026+0000","last_fullsized":"2024-04-25T05:34:44.896026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.669540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663289+0000","last_change":"2024-04-25T05:34:00.694199+0000","last_active":"2024-04-25T05:34:44.663289+0000","last_peered":"2024-04-25T05:34:44.663289+0000","last_clean":"2024-04-25T05:34:44.663289+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:34:44.663289+0000","last_undegraded":"2024-04-25T05:34:44.663289+0000","last_fullsized":"2024-04-25T05:34:44.663289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.968271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896572+0000","last_change":"2024-04-25T05:34:02.702812+0000","last_active":"2024-04-25T05:34:44.896572+0000","last_peered":"2024-04-25T05:34:44.896572+0000","last_clean":"2024-04-25T05:34:44.896572+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:34:44.896572+0000","last_undegraded":"2024-04-25T05:34:44.896572+0000","last_fullsized":"2024-04-25T05:34:44.896572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.890640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062882+0000","last_change":"2024-04-25T05:34:04.722965+0000","last_active":"2024-04-25T05:34:45.062882+0000","last_peered":"2024-04-25T05:34:45.062882+0000","last_clean":"2024-04-25T05:34:45.062882+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:34:45.062882+0000","last_undegraded":"2024-04-25T05:34:45.062882+0000","last_fullsized":"2024-04-25T05:34:45.062882+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":290,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447135+0000","last_change":"2024-04-25T05:34:20.908770+0000","last_active":"2024-04-25T05:34:40.447135+0000","last_peered":"2024-04-25T05:34:40.447135+0000","last_clean":"2024-04-25T05:34:40.447135+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:34:40.447135+0000","last_undegraded":"2024-04-25T05:34:40.447135+0000","last_fullsized":"2024-04-25T05:34:40.447135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-25T05:34:20.908611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:34:20.908611+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.546403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663663+0000","last_change":"2024-04-25T05:33:58.679230+0000","last_active":"2024-04-25T05:34:44.663663+0000","last_peered":"2024-04-25T05:34:44.663663+0000","last_clean":"2024-04-25T05:34:44.663663+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:34:44.663663+0000","last_undegraded":"2024-04-25T05:34:44.663663+0000","last_fullsized":"2024-04-25T05:34:44.663663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.518886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063160+0000","last_change":"2024-04-25T05:34:01.002201+0000","last_active":"2024-04-25T05:34:45.063160+0000","last_peered":"2024-04-25T05:34:45.063160+0000","last_clean":"2024-04-25T05:34:45.063160+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:34:45.063160+0000","last_undegraded":"2024-04-25T05:34:45.063160+0000","last_fullsized":"2024-04-25T05:34:45.063160+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.821400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448569+0000","last_change":"2024-04-25T05:34:02.701047+0000","last_active":"2024-04-25T05:34:40.448569+0000","last_peered":"2024-04-25T05:34:40.448569+0000","last_clean":"2024-04-25T05:34:40.448569+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:34:40.448569+0000","last_undegraded":"2024-04-25T05:34:40.448569+0000","last_fullsized":"2024-04-25T05:34:40.448569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.440947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662302+0000","last_change":"2024-04-25T05:34:04.726534+0000","last_active":"2024-04-25T05:34:44.662302+0000","last_peered":"2024-04-25T05:34:44.662302+0000","last_clean":"2024-04-25T05:34:44.662302+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:34:44.662302+0000","last_undegraded":"2024-04-25T05:34:44.662302+0000","last_fullsized":"2024-04-25T05:34:44.662302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446588+0000","last_change":"2024-04-25T05:33:58.682492+0000","last_active":"2024-04-25T05:34:40.446588+0000","last_peered":"2024-04-25T05:34:40.446588+0000","last_clean":"2024-04-25T05:34:40.446588+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:34:40.446588+0000","last_undegraded":"2024-04-25T05:34:40.446588+0000","last_fullsized":"2024-04-25T05:34:40.446588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.900372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":88,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.777769+0000","last_change":"2024-04-25T05:34:01.000301+0000","last_active":"2024-04-25T05:34:40.777769+0000","last_peered":"2024-04-25T05:34:40.777769+0000","last_clean":"2024-04-25T05:34:40.777769+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:34:40.777769+0000","last_undegraded":"2024-04-25T05:34:40.777769+0000","last_fullsized":"2024-04-25T05:34:40.777769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.243918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447101+0000","last_change":"2024-04-25T05:34:00.694619+0000","last_active":"2024-04-25T05:34:40.447101+0000","last_peered":"2024-04-25T05:34:40.447101+0000","last_clean":"2024-04-25T05:34:40.447101+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:34:40.447101+0000","last_undegraded":"2024-04-25T05:34:40.447101+0000","last_fullsized":"2024-04-25T05:34:40.447101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:01.840274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064398+0000","last_change":"2024-04-25T05:34:02.709328+0000","last_active":"2024-04-25T05:34:45.064398+0000","last_peered":"2024-04-25T05:34:45.064398+0000","last_clean":"2024-04-25T05:34:45.064398+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:34:45.064398+0000","last_undegraded":"2024-04-25T05:34:45.064398+0000","last_fullsized":"2024-04-25T05:34:45.064398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.085576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662093+0000","last_change":"2024-04-25T05:34:04.759135+0000","last_active":"2024-04-25T05:34:44.662093+0000","last_peered":"2024-04-25T05:34:44.662093+0000","last_clean":"2024-04-25T05:34:44.662093+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:34:44.662093+0000","last_undegraded":"2024-04-25T05:34:44.662093+0000","last_fullsized":"2024-04-25T05:34:44.662093+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662526+0000","last_change":"2024-04-25T05:33:58.677172+0000","last_active":"2024-04-25T05:34:44.662526+0000","last_peered":"2024-04-25T05:34:44.662526+0000","last_clean":"2024-04-25T05:34:44.662526+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:34:44.662526+0000","last_undegraded":"2024-04-25T05:34:44.662526+0000","last_fullsized":"2024-04-25T05:34:44.662526+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.326701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447490+0000","last_change":"2024-04-25T05:34:00.999939+0000","last_active":"2024-04-25T05:34:40.447490+0000","last_peered":"2024-04-25T05:34:40.447490+0000","last_clean":"2024-04-25T05:34:40.447490+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:34:40.447490+0000","last_undegraded":"2024-04-25T05:34:40.447490+0000","last_fullsized":"2024-04-25T05:34:40.447490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.684394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062720+0000","last_change":"2024-04-25T05:34:02.708443+0000","last_active":"2024-04-25T05:34:45.062720+0000","last_peered":"2024-04-25T05:34:45.062720+0000","last_clean":"2024-04-25T05:34:45.062720+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:34:45.062720+0000","last_undegraded":"2024-04-25T05:34:45.062720+0000","last_fullsized":"2024-04-25T05:34:45.062720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.195589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.446348+0000","last_change":"2024-04-25T05:34:04.728674+0000","last_active":"2024-04-25T05:34:40.446348+0000","last_peered":"2024-04-25T05:34:40.446348+0000","last_clean":"2024-04-25T05:34:40.446348+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:34:40.446348+0000","last_undegraded":"2024-04-25T05:34:40.446348+0000","last_fullsized":"2024-04-25T05:34:40.446348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.707833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064105+0000","last_change":"2024-04-25T05:33:58.678744+0000","last_active":"2024-04-25T05:34:45.064105+0000","last_peered":"2024-04-25T05:34:45.064105+0000","last_clean":"2024-04-25T05:34:45.064105+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:34:45.064105+0000","last_undegraded":"2024-04-25T05:34:45.064105+0000","last_fullsized":"2024-04-25T05:34:45.064105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.579714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662971+0000","last_change":"2024-04-25T05:34:00.688978+0000","last_active":"2024-04-25T05:34:44.662971+0000","last_peered":"2024-04-25T05:34:44.662971+0000","last_clean":"2024-04-25T05:34:44.662971+0000","last_became_active":"2024-04-25T05:34:00.688453+0000","last_became_peered":"2024-04-25T05:34:00.688453+0000","last_unstale":"2024-04-25T05:34:44.662971+0000","last_undegraded":"2024-04-25T05:34:44.662971+0000","last_fullsized":"2024-04-25T05:34:44.662971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.622999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447712+0000","last_change":"2024-04-25T05:34:02.708185+0000","last_active":"2024-04-25T05:34:40.447712+0000","last_peered":"2024-04-25T05:34:40.447712+0000","last_clean":"2024-04-25T05:34:40.447712+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:34:40.447712+0000","last_undegraded":"2024-04-25T05:34:40.447712+0000","last_fullsized":"2024-04-25T05:34:40.447712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.047597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.448887+0000","last_change":"2024-04-25T05:34:04.761218+0000","last_active":"2024-04-25T05:34:40.448887+0000","last_peered":"2024-04-25T05:34:40.448887+0000","last_clean":"2024-04-25T05:34:40.448887+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:34:40.448887+0000","last_undegraded":"2024-04-25T05:34:40.448887+0000","last_fullsized":"2024-04-25T05:34:40.448887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.789438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:40.447217+0000","last_change":"2024-04-25T05:33:58.682409+0000","last_active":"2024-04-25T05:34:40.447217+0000","last_peered":"2024-04-25T05:34:40.447217+0000","last_clean":"2024-04-25T05:34:40.447217+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:34:40.447217+0000","last_undegraded":"2024-04-25T05:34:40.447217+0000","last_fullsized":"2024-04-25T05:34:40.447217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661628+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:34:39.661628+0000","last_peered":"2024-04-25T05:34:39.661628+0000","last_clean":"2024-04-25T05:34:39.661628+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:34:39.661628+0000","last_undegraded":"2024-04-25T05:34:39.661628+0000","last_fullsized":"2024-04-25T05:34:39.661628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.826192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663379+0000","last_change":"2024-04-25T05:34:02.701774+0000","last_active":"2024-04-25T05:34:44.663379+0000","last_peered":"2024-04-25T05:34:44.663379+0000","last_clean":"2024-04-25T05:34:44.663379+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:34:44.663379+0000","last_undegraded":"2024-04-25T05:34:44.663379+0000","last_fullsized":"2024-04-25T05:34:44.663379+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.422624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661265+0000","last_change":"2024-04-25T05:34:04.718464+0000","last_active":"2024-04-25T05:34:39.661265+0000","last_peered":"2024-04-25T05:34:39.661265+0000","last_clean":"2024-04-25T05:34:39.661265+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:34:39.661265+0000","last_undegraded":"2024-04-25T05:34:39.661265+0000","last_fullsized":"2024-04-25T05:34:39.661265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.664208+0000","last_change":"2024-04-25T05:33:58.676147+0000","last_active":"2024-04-25T05:34:44.664208+0000","last_peered":"2024-04-25T05:34:44.664208+0000","last_clean":"2024-04-25T05:34:44.664208+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:34:44.664208+0000","last_undegraded":"2024-04-25T05:34:44.664208+0000","last_fullsized":"2024-04-25T05:34:44.664208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.509783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":93,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663268+0000","last_change":"2024-04-25T05:34:00.999824+0000","last_active":"2024-04-25T05:34:44.663268+0000","last_peered":"2024-04-25T05:34:44.663268+0000","last_clean":"2024-04-25T05:34:44.663268+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:34:44.663268+0000","last_undegraded":"2024-04-25T05:34:44.663268+0000","last_fullsized":"2024-04-25T05:34:44.663268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.441893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661138+0000","last_change":"2024-04-25T05:34:02.699472+0000","last_active":"2024-04-25T05:34:39.661138+0000","last_peered":"2024-04-25T05:34:39.661138+0000","last_clean":"2024-04-25T05:34:39.661138+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:34:39.661138+0000","last_undegraded":"2024-04-25T05:34:39.661138+0000","last_fullsized":"2024-04-25T05:34:39.661138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.651777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.063028+0000","last_change":"2024-04-25T05:34:04.724152+0000","last_active":"2024-04-25T05:34:45.063028+0000","last_peered":"2024-04-25T05:34:45.063028+0000","last_clean":"2024-04-25T05:34:45.063028+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:34:45.063028+0000","last_undegraded":"2024-04-25T05:34:45.063028+0000","last_fullsized":"2024-04-25T05:34:45.063028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.599745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662278+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:34:39.662278+0000","last_peered":"2024-04-25T05:34:39.662278+0000","last_clean":"2024-04-25T05:34:39.662278+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:34:39.662278+0000","last_undegraded":"2024-04-25T05:34:39.662278+0000","last_fullsized":"2024-04-25T05:34:39.662278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.390939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663169+0000","last_change":"2024-04-25T05:33:58.681579+0000","last_active":"2024-04-25T05:34:44.663169+0000","last_peered":"2024-04-25T05:34:44.663169+0000","last_clean":"2024-04-25T05:34:44.663169+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:34:44.663169+0000","last_undegraded":"2024-04-25T05:34:44.663169+0000","last_fullsized":"2024-04-25T05:34:44.663169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.706716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663192+0000","last_change":"2024-04-25T05:34:00.691497+0000","last_active":"2024-04-25T05:34:44.663192+0000","last_peered":"2024-04-25T05:34:44.663192+0000","last_clean":"2024-04-25T05:34:44.663192+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:34:44.663192+0000","last_undegraded":"2024-04-25T05:34:44.663192+0000","last_fullsized":"2024-04-25T05:34:44.663192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.049287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663907+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:34:44.663907+0000","last_peered":"2024-04-25T05:34:44.663907+0000","last_clean":"2024-04-25T05:34:44.663907+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:34:44.663907+0000","last_undegraded":"2024-04-25T05:34:44.663907+0000","last_fullsized":"2024-04-25T05:34:44.663907+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:31.872714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.064354+0000","last_change":"2024-04-25T05:34:02.699262+0000","last_active":"2024-04-25T05:34:45.064354+0000","last_peered":"2024-04-25T05:34:45.064354+0000","last_clean":"2024-04-25T05:34:45.064354+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:34:45.064354+0000","last_undegraded":"2024-04-25T05:34:45.064354+0000","last_fullsized":"2024-04-25T05:34:45.064354+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.861356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661135+0000","last_change":"2024-04-25T05:33:58.675050+0000","last_active":"2024-04-25T05:34:39.661135+0000","last_peered":"2024-04-25T05:34:39.661135+0000","last_clean":"2024-04-25T05:34:39.661135+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:34:39.661135+0000","last_undegraded":"2024-04-25T05:34:39.661135+0000","last_fullsized":"2024-04-25T05:34:39.661135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.277179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662124+0000","last_change":"2024-04-25T05:34:00.692677+0000","last_active":"2024-04-25T05:34:39.662124+0000","last_peered":"2024-04-25T05:34:39.662124+0000","last_clean":"2024-04-25T05:34:39.662124+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:34:39.662124+0000","last_undegraded":"2024-04-25T05:34:39.662124+0000","last_fullsized":"2024-04-25T05:34:39.662124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.895770+0000","last_change":"2024-04-25T05:34:04.723087+0000","last_active":"2024-04-25T05:34:44.895770+0000","last_peered":"2024-04-25T05:34:44.895770+0000","last_clean":"2024-04-25T05:34:44.895770+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:34:44.895770+0000","last_undegraded":"2024-04-25T05:34:44.895770+0000","last_fullsized":"2024-04-25T05:34:44.895770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.795993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.662802+0000","last_change":"2024-04-25T05:34:02.708656+0000","last_active":"2024-04-25T05:34:44.662802+0000","last_peered":"2024-04-25T05:34:44.662802+0000","last_clean":"2024-04-25T05:34:44.662802+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:34:44.662802+0000","last_undegraded":"2024-04-25T05:34:44.662802+0000","last_fullsized":"2024-04-25T05:34:44.662802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.620659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:45.062471+0000","last_change":"2024-04-25T05:33:58.684035+0000","last_active":"2024-04-25T05:34:45.062471+0000","last_peered":"2024-04-25T05:34:45.062471+0000","last_clean":"2024-04-25T05:34:45.062471+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:34:45.062471+0000","last_undegraded":"2024-04-25T05:34:45.062471+0000","last_fullsized":"2024-04-25T05:34:45.062471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.247558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662290+0000","last_change":"2024-04-25T05:34:00.692475+0000","last_active":"2024-04-25T05:34:39.662290+0000","last_peered":"2024-04-25T05:34:39.662290+0000","last_clean":"2024-04-25T05:34:39.662290+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:34:39.662290+0000","last_undegraded":"2024-04-25T05:34:39.662290+0000","last_fullsized":"2024-04-25T05:34:39.662290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.725418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.662341+0000","last_change":"2024-04-25T05:34:04.719644+0000","last_active":"2024-04-25T05:34:39.662341+0000","last_peered":"2024-04-25T05:34:39.662341+0000","last_clean":"2024-04-25T05:34:39.662341+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:34:39.662341+0000","last_undegraded":"2024-04-25T05:34:39.662341+0000","last_fullsized":"2024-04-25T05:34:39.662341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":54,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.896472+0000","last_change":"2024-04-25T05:34:02.705065+0000","last_active":"2024-04-25T05:34:44.896472+0000","last_peered":"2024-04-25T05:34:44.896472+0000","last_clean":"2024-04-25T05:34:44.896472+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:34:44.896472+0000","last_undegraded":"2024-04-25T05:34:44.896472+0000","last_fullsized":"2024-04-25T05:34:44.896472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.937446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T05:34:44.663988+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:34:44.663988+0000","last_peered":"2024-04-25T05:34:44.663988+0000","last_clean":"2024-04-25T05:34:44.663988+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:34:44.663988+0000","last_undegraded":"2024-04-25T05:34:44.663988+0000","last_fullsized":"2024-04-25T05:34:44.663988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.422993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T05:34:39.661663+0000","last_change":"2024-04-25T05:34:00.686532+0000","last_active":"2024-04-25T05:34:39.661663+0000","last_peered":"2024-04-25T05:34:39.661663+0000","last_clean":"2024-04-25T05:34:39.661663+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:34:39.661663+0000","last_undegraded":"2024-04-25T05:34:39.661663+0000","last_fullsized":"2024-04-25T05:34:39.661663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.818346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070478,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":890688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364819,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1290240,"data_stored":895962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":305950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28284,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716868,"statfs":{"total":95995035648,"available":95966072832,"internally_reserved":0,"allocated":1294336,"data_stored":888955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713300,"statfs":{"total":95995035648,"available":95962419200,"internally_reserved":0,"allocated":688128,"data_stored":304326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":724992,"data_stored":305056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986203,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":712704,"data_stored":299404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691864,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:34:50.062 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T05:34:50.062 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T05:34:50.062 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T05:34:50.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T05:34:50.313 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:50.712 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:50 smithi043 ceph-mon[27010]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-25T05:34:50.713 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:50 smithi043 ceph-mon[27010]: osdmap e83: 8 total, 8 up, 8 in 2024-04-25T05:34:50.713 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:50 smithi043 ceph-mon[37492]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-25T05:34:50.713 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:50 smithi043 ceph-mon[37492]: osdmap e83: 8 total, 8 up, 8 in 2024-04-25T05:34:50.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:50 smithi060 ceph-mon[30898]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-25T05:34:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:50 smithi060 ceph-mon[30898]: osdmap e83: 8 total, 8 up, 8 in 2024-04-25T05:34:51.294 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:51.294 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T05:34:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2711407418' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T05:34:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2711407418' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T05:34:52.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2711407418' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T05:34:52.780 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T05:34:52.780 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T05:34:52.780 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-25T05:34:52.791 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-25T05:34:52.791 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-adk-testing-2024-04-24-1625'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-04-24-1625', 'sha1': 'd6f5f4e5ec08db21b2896cb71b4cb907b7fb4737'}} 2024-04-25T05:34:52.791 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-25T05:34:52.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-25T05:34:52.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:52 smithi060 ceph-mon[30898]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:52.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:34:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:34:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:52 smithi043 ceph-mon[27010]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:53.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:52 smithi043 ceph-mon[37492]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:34:53.318 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:53.985 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:53.986 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:53.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:34:54.275 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE Bus STOPPING 2024-04-25T05:34:54.275 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:34:54.275 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE Bus STOPPED 2024-04-25T05:34:54.275 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE Bus STARTING 2024-04-25T05:34:54.275 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE Serving on http://:::9283 2024-04-25T05:34:54.642 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:54] ENGINE Bus STARTED 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[37492]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[37492]: osdmap e84: 8 total, 8 up, 8 in 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[27010]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' 2024-04-25T05:34:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:55 smithi043 ceph-mon[27010]: osdmap e84: 8 total, 8 up, 8 in 2024-04-25T05:34:55.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:55 smithi060 ceph-mon[30898]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:34:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' 2024-04-25T05:34:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:55 smithi060 ceph-mon[30898]: osdmap e84: 8 total, 8 up, 8 in 2024-04-25T05:34:56.317 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-25T05:34:56.318 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T05:34:56.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:34:56.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:56 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:34:56.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:56 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:34:56.427 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:56 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:34:56.427 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:56 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:34:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:56 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:34:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:56 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:34:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:34:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:57 smithi043 ceph-mon[27010]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:57 smithi043 ceph-mon[27010]: osdmap e85: 8 total, 8 up, 8 in 2024-04-25T05:34:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:57 smithi043 ceph-mon[37492]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:57 smithi043 ceph-mon[37492]: osdmap e85: 8 total, 8 up, 8 in 2024-04-25T05:34:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:57 smithi060 ceph-mon[30898]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:57 smithi060 ceph-mon[30898]: osdmap e85: 8 total, 8 up, 8 in 2024-04-25T05:34:58.379 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:58 smithi043 systemd[1]: Starting Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:34:58.674 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:34:58.676 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:34:58.979 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:58 smithi043 podman[72003]: 2024-04-25 05:34:58.689621397 +0000 UTC m=+0.119104903 volume create 08a8265d3f42bb1b1fd42f845c8317f1d0812a7d76166ed3124d525d04315e2a 2024-04-25T05:34:58.979 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:58 smithi043 podman[72003]: 2024-04-25 05:34:58.593429761 +0000 UTC m=+0.022913279 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-25T05:34:58.979 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:58 smithi043 podman[72003]: 2024-04-25 05:34:58.798663573 +0000 UTC m=+0.228147078 container create 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:59.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-mon[27010]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-mon[37492]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 podman[72003]: 2024-04-25 05:34:59.101537216 +0000 UTC m=+0.531020724 container init 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 podman[72003]: 2024-04-25 05:34:59.104268786 +0000 UTC m=+0.533752294 container start 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.117Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.117Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.118Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.43 port=9094 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.119Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.142Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.143Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.145Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:34:59.145Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-25T05:34:59.392 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 bash[72003]: 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab 2024-04-25T05:34:59.393 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:34:59 smithi043 systemd[1]: Started Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:34:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:34:59 smithi060 ceph-mon[30898]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:34:59.801 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE Bus STOPPING 2024-04-25T05:34:59.801 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:34:59.802 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE Bus STOPPED 2024-04-25T05:34:59.802 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE Bus STARTING 2024-04-25T05:34:59.802 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE Serving on http://:::9283 2024-04-25T05:34:59.802 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:34:59] ENGINE Bus STARTED 2024-04-25T05:34:59.802 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:34:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:34:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:34:59.924 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-25T05:34:57.659064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1275,"num_read_kb":1089,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236212,"kb_used_data":7564,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725004,"statfs":{"total":767960285184,"available":767718404096,"internally_reserved":0,"allocated":7745536,"data_stored":4342057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.996694"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164073+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:34:56.164073+0000","last_peered":"2024-04-25T05:34:56.164073+0000","last_clean":"2024-04-25T05:34:56.164073+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:34:56.164073+0000","last_undegraded":"2024-04-25T05:34:56.164073+0000","last_fullsized":"2024-04-25T05:34:56.164073+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.921834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148652+0000","last_change":"2024-04-25T05:34:02.700825+0000","last_active":"2024-04-25T05:34:57.148652+0000","last_peered":"2024-04-25T05:34:57.148652+0000","last_clean":"2024-04-25T05:34:57.148652+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:34:57.148652+0000","last_undegraded":"2024-04-25T05:34:57.148652+0000","last_fullsized":"2024-04-25T05:34:57.148652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.901521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.203102+0000","last_change":"2024-04-25T05:33:58.677608+0000","last_active":"2024-04-25T05:34:57.203102+0000","last_peered":"2024-04-25T05:34:57.203102+0000","last_clean":"2024-04-25T05:34:57.203102+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:34:57.203102+0000","last_undegraded":"2024-04-25T05:34:57.203102+0000","last_fullsized":"2024-04-25T05:34:57.203102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.073153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.163918+0000","last_change":"2024-04-25T05:34:00.692628+0000","last_active":"2024-04-25T05:34:56.163918+0000","last_peered":"2024-04-25T05:34:56.163918+0000","last_clean":"2024-04-25T05:34:56.163918+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:34:56.163918+0000","last_undegraded":"2024-04-25T05:34:56.163918+0000","last_fullsized":"2024-04-25T05:34:56.163918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.003557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165182+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:34:56.165182+0000","last_peered":"2024-04-25T05:34:56.165182+0000","last_clean":"2024-04-25T05:34:56.165182+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:34:56.165182+0000","last_undegraded":"2024-04-25T05:34:56.165182+0000","last_fullsized":"2024-04-25T05:34:56.165182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.498112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.163943+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:34:56.163943+0000","last_peered":"2024-04-25T05:34:56.163943+0000","last_clean":"2024-04-25T05:34:56.163943+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:34:56.163943+0000","last_undegraded":"2024-04-25T05:34:56.163943+0000","last_fullsized":"2024-04-25T05:34:56.163943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.555566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165373+0000","last_change":"2024-04-25T05:33:58.679726+0000","last_active":"2024-04-25T05:34:56.165373+0000","last_peered":"2024-04-25T05:34:56.165373+0000","last_clean":"2024-04-25T05:34:56.165373+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:34:56.165373+0000","last_undegraded":"2024-04-25T05:34:56.165373+0000","last_fullsized":"2024-04-25T05:34:56.165373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.377018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":116,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165908+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:34:56.165908+0000","last_peered":"2024-04-25T05:34:56.165908+0000","last_clean":"2024-04-25T05:34:56.165908+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:34:56.165908+0000","last_undegraded":"2024-04-25T05:34:56.165908+0000","last_fullsized":"2024-04-25T05:34:56.165908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.111490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202123+0000","last_change":"2024-04-25T05:34:04.724350+0000","last_active":"2024-04-25T05:34:57.202123+0000","last_peered":"2024-04-25T05:34:57.202123+0000","last_clean":"2024-04-25T05:34:57.202123+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:34:57.202123+0000","last_undegraded":"2024-04-25T05:34:57.202123+0000","last_fullsized":"2024-04-25T05:34:57.202123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.943468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449490+0000","last_change":"2024-04-25T05:34:02.709504+0000","last_active":"2024-04-25T05:34:57.449490+0000","last_peered":"2024-04-25T05:34:57.449490+0000","last_clean":"2024-04-25T05:34:57.449490+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:34:57.449490+0000","last_undegraded":"2024-04-25T05:34:57.449490+0000","last_fullsized":"2024-04-25T05:34:57.449490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.151709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450234+0000","last_change":"2024-04-25T05:33:58.687700+0000","last_active":"2024-04-25T05:34:57.450234+0000","last_peered":"2024-04-25T05:34:57.450234+0000","last_clean":"2024-04-25T05:34:57.450234+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:34:57.450234+0000","last_undegraded":"2024-04-25T05:34:57.450234+0000","last_fullsized":"2024-04-25T05:34:57.450234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.471958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164328+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:34:56.164328+0000","last_peered":"2024-04-25T05:34:56.164328+0000","last_clean":"2024-04-25T05:34:56.164328+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:34:56.164328+0000","last_undegraded":"2024-04-25T05:34:56.164328+0000","last_fullsized":"2024-04-25T05:34:56.164328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.181248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448783+0000","last_change":"2024-04-25T05:34:04.722769+0000","last_active":"2024-04-25T05:34:57.448783+0000","last_peered":"2024-04-25T05:34:57.448783+0000","last_clean":"2024-04-25T05:34:57.448783+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:34:57.448783+0000","last_undegraded":"2024-04-25T05:34:57.448783+0000","last_fullsized":"2024-04-25T05:34:57.448783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.673388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448778+0000","last_change":"2024-04-25T05:34:02.701665+0000","last_active":"2024-04-25T05:34:57.448778+0000","last_peered":"2024-04-25T05:34:57.448778+0000","last_clean":"2024-04-25T05:34:57.448778+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:34:57.448778+0000","last_undegraded":"2024-04-25T05:34:57.448778+0000","last_fullsized":"2024-04-25T05:34:57.448778+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.814291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449882+0000","last_change":"2024-04-25T05:33:58.688919+0000","last_active":"2024-04-25T05:34:57.449882+0000","last_peered":"2024-04-25T05:34:57.449882+0000","last_clean":"2024-04-25T05:34:57.449882+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:34:57.449882+0000","last_undegraded":"2024-04-25T05:34:57.449882+0000","last_fullsized":"2024-04-25T05:34:57.449882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.028583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164455+0000","last_change":"2024-04-25T05:34:00.693330+0000","last_active":"2024-04-25T05:34:56.164455+0000","last_peered":"2024-04-25T05:34:56.164455+0000","last_clean":"2024-04-25T05:34:56.164455+0000","last_became_active":"2024-04-25T05:34:00.693003+0000","last_became_peered":"2024-04-25T05:34:00.693003+0000","last_unstale":"2024-04-25T05:34:56.164455+0000","last_undegraded":"2024-04-25T05:34:56.164455+0000","last_fullsized":"2024-04-25T05:34:56.164455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.762449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165128+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:34:56.165128+0000","last_peered":"2024-04-25T05:34:56.165128+0000","last_clean":"2024-04-25T05:34:56.165128+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:34:56.165128+0000","last_undegraded":"2024-04-25T05:34:56.165128+0000","last_fullsized":"2024-04-25T05:34:56.165128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.388118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149081+0000","last_change":"2024-04-25T05:34:02.708603+0000","last_active":"2024-04-25T05:34:57.149081+0000","last_peered":"2024-04-25T05:34:57.149081+0000","last_clean":"2024-04-25T05:34:57.149081+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:34:57.149081+0000","last_undegraded":"2024-04-25T05:34:57.149081+0000","last_fullsized":"2024-04-25T05:34:57.149081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.908190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202569+0000","last_change":"2024-04-25T05:33:58.683934+0000","last_active":"2024-04-25T05:34:57.202569+0000","last_peered":"2024-04-25T05:34:57.202569+0000","last_clean":"2024-04-25T05:34:57.202569+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:34:57.202569+0000","last_undegraded":"2024-04-25T05:34:57.202569+0000","last_fullsized":"2024-04-25T05:34:57.202569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.927389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164448+0000","last_change":"2024-04-25T05:34:00.690831+0000","last_active":"2024-04-25T05:34:56.164448+0000","last_peered":"2024-04-25T05:34:56.164448+0000","last_clean":"2024-04-25T05:34:56.164448+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:34:56.164448+0000","last_undegraded":"2024-04-25T05:34:56.164448+0000","last_fullsized":"2024-04-25T05:34:56.164448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.940787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166455+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:34:56.166455+0000","last_peered":"2024-04-25T05:34:56.166455+0000","last_clean":"2024-04-25T05:34:56.166455+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:34:56.166455+0000","last_undegraded":"2024-04-25T05:34:56.166455+0000","last_fullsized":"2024-04-25T05:34:56.166455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164666+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:34:56.164666+0000","last_peered":"2024-04-25T05:34:56.164666+0000","last_clean":"2024-04-25T05:34:56.164666+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:34:56.164666+0000","last_undegraded":"2024-04-25T05:34:56.164666+0000","last_fullsized":"2024-04-25T05:34:56.164666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.792766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164332+0000","last_change":"2024-04-25T05:33:58.677738+0000","last_active":"2024-04-25T05:34:56.164332+0000","last_peered":"2024-04-25T05:34:56.164332+0000","last_clean":"2024-04-25T05:34:56.164332+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:34:56.164332+0000","last_undegraded":"2024-04-25T05:34:56.164332+0000","last_fullsized":"2024-04-25T05:34:56.164332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.705248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164690+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:34:56.164690+0000","last_peered":"2024-04-25T05:34:56.164690+0000","last_clean":"2024-04-25T05:34:56.164690+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:34:56.164690+0000","last_undegraded":"2024-04-25T05:34:56.164690+0000","last_fullsized":"2024-04-25T05:34:56.164690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.537702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449031+0000","last_change":"2024-04-25T05:34:04.725886+0000","last_active":"2024-04-25T05:34:57.449031+0000","last_peered":"2024-04-25T05:34:57.449031+0000","last_clean":"2024-04-25T05:34:57.449031+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:34:57.449031+0000","last_undegraded":"2024-04-25T05:34:57.449031+0000","last_fullsized":"2024-04-25T05:34:57.449031+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412842+0000","last_change":"2024-04-25T05:34:02.701003+0000","last_active":"2024-04-25T05:34:56.412842+0000","last_peered":"2024-04-25T05:34:56.412842+0000","last_clean":"2024-04-25T05:34:56.412842+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:34:56.412842+0000","last_undegraded":"2024-04-25T05:34:56.412842+0000","last_fullsized":"2024-04-25T05:34:56.412842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.795737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149927+0000","last_change":"2024-04-25T05:33:58.678204+0000","last_active":"2024-04-25T05:34:57.149927+0000","last_peered":"2024-04-25T05:34:57.149927+0000","last_clean":"2024-04-25T05:34:57.149927+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:34:57.149927+0000","last_undegraded":"2024-04-25T05:34:57.149927+0000","last_fullsized":"2024-04-25T05:34:57.149927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.231648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201870+0000","last_change":"2024-04-25T05:34:01.002306+0000","last_active":"2024-04-25T05:34:57.201870+0000","last_peered":"2024-04-25T05:34:57.201870+0000","last_clean":"2024-04-25T05:34:57.201870+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:34:57.201870+0000","last_undegraded":"2024-04-25T05:34:57.201870+0000","last_fullsized":"2024-04-25T05:34:57.201870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.875946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148890+0000","last_change":"2024-04-25T05:34:04.722159+0000","last_active":"2024-04-25T05:34:57.148890+0000","last_peered":"2024-04-25T05:34:57.148890+0000","last_clean":"2024-04-25T05:34:57.148890+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:34:57.148890+0000","last_undegraded":"2024-04-25T05:34:57.148890+0000","last_fullsized":"2024-04-25T05:34:57.148890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.853709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202633+0000","last_change":"2024-04-25T05:34:02.710777+0000","last_active":"2024-04-25T05:34:57.202633+0000","last_peered":"2024-04-25T05:34:57.202633+0000","last_clean":"2024-04-25T05:34:57.202633+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:34:57.202633+0000","last_undegraded":"2024-04-25T05:34:57.202633+0000","last_fullsized":"2024-04-25T05:34:57.202633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.845754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164199+0000","last_change":"2024-04-25T05:33:58.676498+0000","last_active":"2024-04-25T05:34:56.164199+0000","last_peered":"2024-04-25T05:34:56.164199+0000","last_clean":"2024-04-25T05:34:56.164199+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:34:56.164199+0000","last_undegraded":"2024-04-25T05:34:56.164199+0000","last_fullsized":"2024-04-25T05:34:56.164199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.714656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412815+0000","last_change":"2024-04-25T05:34:00.691266+0000","last_active":"2024-04-25T05:34:56.412815+0000","last_peered":"2024-04-25T05:34:56.412815+0000","last_clean":"2024-04-25T05:34:56.412815+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:34:56.412815+0000","last_undegraded":"2024-04-25T05:34:56.412815+0000","last_fullsized":"2024-04-25T05:34:56.412815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.730290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448588+0000","last_change":"2024-04-25T05:34:04.725684+0000","last_active":"2024-04-25T05:34:57.448588+0000","last_peered":"2024-04-25T05:34:57.448588+0000","last_clean":"2024-04-25T05:34:57.448588+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:34:57.448588+0000","last_undegraded":"2024-04-25T05:34:57.448588+0000","last_fullsized":"2024-04-25T05:34:57.448588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.515944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412053+0000","last_change":"2024-04-25T05:34:02.701292+0000","last_active":"2024-04-25T05:34:56.412053+0000","last_peered":"2024-04-25T05:34:56.412053+0000","last_clean":"2024-04-25T05:34:56.412053+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:34:56.412053+0000","last_undegraded":"2024-04-25T05:34:56.412053+0000","last_fullsized":"2024-04-25T05:34:56.412053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202631+0000","last_change":"2024-04-25T05:33:58.678452+0000","last_active":"2024-04-25T05:34:57.202631+0000","last_peered":"2024-04-25T05:34:57.202631+0000","last_clean":"2024-04-25T05:34:57.202631+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:34:57.202631+0000","last_undegraded":"2024-04-25T05:34:57.202631+0000","last_fullsized":"2024-04-25T05:34:57.202631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.043078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164255+0000","last_change":"2024-04-25T05:34:00.692546+0000","last_active":"2024-04-25T05:34:56.164255+0000","last_peered":"2024-04-25T05:34:56.164255+0000","last_clean":"2024-04-25T05:34:56.164255+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:34:56.164255+0000","last_undegraded":"2024-04-25T05:34:56.164255+0000","last_fullsized":"2024-04-25T05:34:56.164255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.114492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165040+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:34:56.165040+0000","last_peered":"2024-04-25T05:34:56.165040+0000","last_clean":"2024-04-25T05:34:56.165040+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:34:56.165040+0000","last_undegraded":"2024-04-25T05:34:56.165040+0000","last_fullsized":"2024-04-25T05:34:56.165040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.872916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450336+0000","last_change":"2024-04-25T05:34:02.701784+0000","last_active":"2024-04-25T05:34:57.450336+0000","last_peered":"2024-04-25T05:34:57.450336+0000","last_clean":"2024-04-25T05:34:57.450336+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:34:57.450336+0000","last_undegraded":"2024-04-25T05:34:57.450336+0000","last_fullsized":"2024-04-25T05:34:57.450336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.034685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450594+0000","last_change":"2024-04-25T05:33:58.679682+0000","last_active":"2024-04-25T05:34:57.450594+0000","last_peered":"2024-04-25T05:34:57.450594+0000","last_clean":"2024-04-25T05:34:57.450594+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:34:57.450594+0000","last_undegraded":"2024-04-25T05:34:57.450594+0000","last_fullsized":"2024-04-25T05:34:57.450594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164904+0000","last_change":"2024-04-25T05:34:00.999704+0000","last_active":"2024-04-25T05:34:56.164904+0000","last_peered":"2024-04-25T05:34:56.164904+0000","last_clean":"2024-04-25T05:34:56.164904+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:34:56.164904+0000","last_undegraded":"2024-04-25T05:34:56.164904+0000","last_fullsized":"2024-04-25T05:34:56.164904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.051072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201928+0000","last_change":"2024-04-25T05:34:04.727071+0000","last_active":"2024-04-25T05:34:57.201928+0000","last_peered":"2024-04-25T05:34:57.201928+0000","last_clean":"2024-04-25T05:34:57.201928+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:34:57.201928+0000","last_undegraded":"2024-04-25T05:34:57.201928+0000","last_fullsized":"2024-04-25T05:34:57.201928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.039265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165288+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:34:56.165288+0000","last_peered":"2024-04-25T05:34:56.165288+0000","last_clean":"2024-04-25T05:34:56.165288+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:34:56.165288+0000","last_undegraded":"2024-04-25T05:34:56.165288+0000","last_fullsized":"2024-04-25T05:34:56.165288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.038248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449403+0000","last_change":"2024-04-25T05:33:58.687406+0000","last_active":"2024-04-25T05:34:57.449403+0000","last_peered":"2024-04-25T05:34:57.449403+0000","last_clean":"2024-04-25T05:34:57.449403+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:34:57.449403+0000","last_undegraded":"2024-04-25T05:34:57.449403+0000","last_fullsized":"2024-04-25T05:34:57.449403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.844471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148255+0000","last_change":"2024-04-25T05:34:00.689589+0000","last_active":"2024-04-25T05:34:57.148255+0000","last_peered":"2024-04-25T05:34:57.148255+0000","last_clean":"2024-04-25T05:34:57.148255+0000","last_became_active":"2024-04-25T05:34:00.688897+0000","last_became_peered":"2024-04-25T05:34:00.688897+0000","last_unstale":"2024-04-25T05:34:57.148255+0000","last_undegraded":"2024-04-25T05:34:57.148255+0000","last_fullsized":"2024-04-25T05:34:57.148255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164404+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:34:56.164404+0000","last_peered":"2024-04-25T05:34:56.164404+0000","last_clean":"2024-04-25T05:34:56.164404+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:34:56.164404+0000","last_undegraded":"2024-04-25T05:34:56.164404+0000","last_fullsized":"2024-04-25T05:34:56.164404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.871648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149997+0000","last_change":"2024-04-25T05:34:02.699322+0000","last_active":"2024-04-25T05:34:57.149997+0000","last_peered":"2024-04-25T05:34:57.149997+0000","last_clean":"2024-04-25T05:34:57.149997+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:34:57.149997+0000","last_undegraded":"2024-04-25T05:34:57.149997+0000","last_fullsized":"2024-04-25T05:34:57.149997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.105773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164821+0000","last_change":"2024-04-25T05:33:58.680453+0000","last_active":"2024-04-25T05:34:56.164821+0000","last_peered":"2024-04-25T05:34:56.164821+0000","last_clean":"2024-04-25T05:34:56.164821+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:34:56.164821+0000","last_undegraded":"2024-04-25T05:34:56.164821+0000","last_fullsized":"2024-04-25T05:34:56.164821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.585516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164558+0000","last_change":"2024-04-25T05:34:00.693320+0000","last_active":"2024-04-25T05:34:56.164558+0000","last_peered":"2024-04-25T05:34:56.164558+0000","last_clean":"2024-04-25T05:34:56.164558+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:34:56.164558+0000","last_undegraded":"2024-04-25T05:34:56.164558+0000","last_fullsized":"2024-04-25T05:34:56.164558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.690705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202562+0000","last_change":"2024-04-25T05:34:04.724697+0000","last_active":"2024-04-25T05:34:57.202562+0000","last_peered":"2024-04-25T05:34:57.202562+0000","last_clean":"2024-04-25T05:34:57.202562+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:34:57.202562+0000","last_undegraded":"2024-04-25T05:34:57.202562+0000","last_fullsized":"2024-04-25T05:34:57.202562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.282641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450340+0000","last_change":"2024-04-25T05:34:02.708587+0000","last_active":"2024-04-25T05:34:57.450340+0000","last_peered":"2024-04-25T05:34:57.450340+0000","last_clean":"2024-04-25T05:34:57.450340+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:34:57.450340+0000","last_undegraded":"2024-04-25T05:34:57.450340+0000","last_fullsized":"2024-04-25T05:34:57.450340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.470304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448963+0000","last_change":"2024-04-25T05:33:58.685869+0000","last_active":"2024-04-25T05:34:57.448963+0000","last_peered":"2024-04-25T05:34:57.448963+0000","last_clean":"2024-04-25T05:34:57.448963+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:34:57.448963+0000","last_undegraded":"2024-04-25T05:34:57.448963+0000","last_fullsized":"2024-04-25T05:34:57.448963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.260635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165186+0000","last_change":"2024-04-25T05:34:00.999737+0000","last_active":"2024-04-25T05:34:56.165186+0000","last_peered":"2024-04-25T05:34:56.165186+0000","last_clean":"2024-04-25T05:34:56.165186+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:34:56.165186+0000","last_undegraded":"2024-04-25T05:34:56.165186+0000","last_fullsized":"2024-04-25T05:34:56.165186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.305882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165280+0000","last_change":"2024-04-25T05:34:04.715688+0000","last_active":"2024-04-25T05:34:56.165280+0000","last_peered":"2024-04-25T05:34:56.165280+0000","last_clean":"2024-04-25T05:34:56.165280+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:34:56.165280+0000","last_undegraded":"2024-04-25T05:34:56.165280+0000","last_fullsized":"2024-04-25T05:34:56.165280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.959881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166431+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:34:56.166431+0000","last_peered":"2024-04-25T05:34:56.166431+0000","last_clean":"2024-04-25T05:34:56.166431+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:34:56.166431+0000","last_undegraded":"2024-04-25T05:34:56.166431+0000","last_fullsized":"2024-04-25T05:34:56.166431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.077903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202231+0000","last_change":"2024-04-25T05:33:58.685955+0000","last_active":"2024-04-25T05:34:57.202231+0000","last_peered":"2024-04-25T05:34:57.202231+0000","last_clean":"2024-04-25T05:34:57.202231+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:34:57.202231+0000","last_undegraded":"2024-04-25T05:34:57.202231+0000","last_fullsized":"2024-04-25T05:34:57.202231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.649756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448747+0000","last_change":"2024-04-25T05:34:01.002652+0000","last_active":"2024-04-25T05:34:57.448747+0000","last_peered":"2024-04-25T05:34:57.448747+0000","last_clean":"2024-04-25T05:34:57.448747+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:34:57.448747+0000","last_undegraded":"2024-04-25T05:34:57.448747+0000","last_fullsized":"2024-04-25T05:34:57.448747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.164431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164778+0000","last_change":"2024-04-25T05:34:04.719603+0000","last_active":"2024-04-25T05:34:56.164778+0000","last_peered":"2024-04-25T05:34:56.164778+0000","last_clean":"2024-04-25T05:34:56.164778+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:34:56.164778+0000","last_undegraded":"2024-04-25T05:34:56.164778+0000","last_fullsized":"2024-04-25T05:34:56.164778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.818750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166430+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:34:56.166430+0000","last_peered":"2024-04-25T05:34:56.166430+0000","last_clean":"2024-04-25T05:34:56.166430+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:34:56.166430+0000","last_undegraded":"2024-04-25T05:34:56.166430+0000","last_fullsized":"2024-04-25T05:34:56.166430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.004188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448679+0000","last_change":"2024-04-25T05:33:58.685792+0000","last_active":"2024-04-25T05:34:57.448679+0000","last_peered":"2024-04-25T05:34:57.448679+0000","last_clean":"2024-04-25T05:34:57.448679+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:34:57.448679+0000","last_undegraded":"2024-04-25T05:34:57.448679+0000","last_fullsized":"2024-04-25T05:34:57.448679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.497320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201838+0000","last_change":"2024-04-25T05:34:01.002399+0000","last_active":"2024-04-25T05:34:57.201838+0000","last_peered":"2024-04-25T05:34:57.201838+0000","last_clean":"2024-04-25T05:34:57.201838+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:34:57.201838+0000","last_undegraded":"2024-04-25T05:34:57.201838+0000","last_fullsized":"2024-04-25T05:34:57.201838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.611153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449281+0000","last_change":"2024-04-25T05:34:04.760885+0000","last_active":"2024-04-25T05:34:57.449281+0000","last_peered":"2024-04-25T05:34:57.449281+0000","last_clean":"2024-04-25T05:34:57.449281+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:34:57.449281+0000","last_undegraded":"2024-04-25T05:34:57.449281+0000","last_fullsized":"2024-04-25T05:34:57.449281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.397373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450595+0000","last_change":"2024-04-25T05:34:02.700150+0000","last_active":"2024-04-25T05:34:57.450595+0000","last_peered":"2024-04-25T05:34:57.450595+0000","last_clean":"2024-04-25T05:34:57.450595+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:34:57.450595+0000","last_undegraded":"2024-04-25T05:34:57.450595+0000","last_fullsized":"2024-04-25T05:34:57.450595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.726607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.411961+0000","last_change":"2024-04-25T05:33:58.682812+0000","last_active":"2024-04-25T05:34:56.411961+0000","last_peered":"2024-04-25T05:34:56.411961+0000","last_clean":"2024-04-25T05:34:56.411961+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:34:56.411961+0000","last_undegraded":"2024-04-25T05:34:56.411961+0000","last_fullsized":"2024-04-25T05:34:56.411961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.737056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166548+0000","last_change":"2024-04-25T05:34:01.000303+0000","last_active":"2024-04-25T05:34:56.166548+0000","last_peered":"2024-04-25T05:34:56.166548+0000","last_clean":"2024-04-25T05:34:56.166548+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:34:56.166548+0000","last_undegraded":"2024-04-25T05:34:56.166548+0000","last_fullsized":"2024-04-25T05:34:56.166548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.779729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449287+0000","last_change":"2024-04-25T05:34:04.759369+0000","last_active":"2024-04-25T05:34:57.449287+0000","last_peered":"2024-04-25T05:34:57.449287+0000","last_clean":"2024-04-25T05:34:57.449287+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:34:57.449287+0000","last_undegraded":"2024-04-25T05:34:57.449287+0000","last_fullsized":"2024-04-25T05:34:57.449287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.510841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449983+0000","last_change":"2024-04-25T05:33:58.689200+0000","last_active":"2024-04-25T05:34:57.449983+0000","last_peered":"2024-04-25T05:34:57.449983+0000","last_clean":"2024-04-25T05:34:57.449983+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:34:57.449983+0000","last_undegraded":"2024-04-25T05:34:57.449983+0000","last_fullsized":"2024-04-25T05:34:57.449983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.468188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":92,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448403+0000","last_change":"2024-04-25T05:34:01.000840+0000","last_active":"2024-04-25T05:34:57.448403+0000","last_peered":"2024-04-25T05:34:57.448403+0000","last_clean":"2024-04-25T05:34:57.448403+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:34:57.448403+0000","last_undegraded":"2024-04-25T05:34:57.448403+0000","last_fullsized":"2024-04-25T05:34:57.448403+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.859671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449637+0000","last_change":"2024-04-25T05:34:02.701548+0000","last_active":"2024-04-25T05:34:57.449637+0000","last_peered":"2024-04-25T05:34:57.449637+0000","last_clean":"2024-04-25T05:34:57.449637+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:34:57.449637+0000","last_undegraded":"2024-04-25T05:34:57.449637+0000","last_fullsized":"2024-04-25T05:34:57.449637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.747443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165060+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:34:56.165060+0000","last_peered":"2024-04-25T05:34:56.165060+0000","last_clean":"2024-04-25T05:34:56.165060+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:34:56.165060+0000","last_undegraded":"2024-04-25T05:34:56.165060+0000","last_fullsized":"2024-04-25T05:34:56.165060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.987326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449772+0000","last_change":"2024-04-25T05:33:58.687364+0000","last_active":"2024-04-25T05:34:57.449772+0000","last_peered":"2024-04-25T05:34:57.449772+0000","last_clean":"2024-04-25T05:34:57.449772+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:34:57.449772+0000","last_undegraded":"2024-04-25T05:34:57.449772+0000","last_fullsized":"2024-04-25T05:34:57.449772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.749177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":90,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164422+0000","last_change":"2024-04-25T05:34:00.694001+0000","last_active":"2024-04-25T05:34:56.164422+0000","last_peered":"2024-04-25T05:34:56.164422+0000","last_clean":"2024-04-25T05:34:56.164422+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:34:56.164422+0000","last_undegraded":"2024-04-25T05:34:56.164422+0000","last_fullsized":"2024-04-25T05:34:56.164422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.293624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165138+0000","last_change":"2024-04-25T05:34:02.710703+0000","last_active":"2024-04-25T05:34:56.165138+0000","last_peered":"2024-04-25T05:34:56.165138+0000","last_clean":"2024-04-25T05:34:56.165138+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:34:56.165138+0000","last_undegraded":"2024-04-25T05:34:56.165138+0000","last_fullsized":"2024-04-25T05:34:56.165138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.457617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450099+0000","last_change":"2024-04-25T05:34:04.727176+0000","last_active":"2024-04-25T05:34:57.450099+0000","last_peered":"2024-04-25T05:34:57.450099+0000","last_clean":"2024-04-25T05:34:57.450099+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:34:57.450099+0000","last_undegraded":"2024-04-25T05:34:57.450099+0000","last_fullsized":"2024-04-25T05:34:57.450099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165412+0000","last_change":"2024-04-25T05:34:02.710395+0000","last_active":"2024-04-25T05:34:56.165412+0000","last_peered":"2024-04-25T05:34:56.165412+0000","last_clean":"2024-04-25T05:34:56.165412+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:34:56.165412+0000","last_undegraded":"2024-04-25T05:34:56.165412+0000","last_fullsized":"2024-04-25T05:34:56.165412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.749912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449750+0000","last_change":"2024-04-25T05:33:58.686463+0000","last_active":"2024-04-25T05:34:57.449750+0000","last_peered":"2024-04-25T05:34:57.449750+0000","last_clean":"2024-04-25T05:34:57.449750+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:34:57.449750+0000","last_undegraded":"2024-04-25T05:34:57.449750+0000","last_fullsized":"2024-04-25T05:34:57.449750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:59.578656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412421+0000","last_change":"2024-04-25T05:34:00.999892+0000","last_active":"2024-04-25T05:34:56.412421+0000","last_peered":"2024-04-25T05:34:56.412421+0000","last_clean":"2024-04-25T05:34:56.412421+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:34:56.412421+0000","last_undegraded":"2024-04-25T05:34:56.412421+0000","last_fullsized":"2024-04-25T05:34:56.412421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.466514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202039+0000","last_change":"2024-04-25T05:34:04.725833+0000","last_active":"2024-04-25T05:34:57.202039+0000","last_peered":"2024-04-25T05:34:57.202039+0000","last_clean":"2024-04-25T05:34:57.202039+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:34:57.202039+0000","last_undegraded":"2024-04-25T05:34:57.202039+0000","last_fullsized":"2024-04-25T05:34:57.202039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165080+0000","last_change":"2024-04-25T05:34:02.700901+0000","last_active":"2024-04-25T05:34:56.165080+0000","last_peered":"2024-04-25T05:34:56.165080+0000","last_clean":"2024-04-25T05:34:56.165080+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:34:56.165080+0000","last_undegraded":"2024-04-25T05:34:56.165080+0000","last_fullsized":"2024-04-25T05:34:56.165080+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.464318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450003+0000","last_change":"2024-04-25T05:33:58.685937+0000","last_active":"2024-04-25T05:34:57.450003+0000","last_peered":"2024-04-25T05:34:57.450003+0000","last_clean":"2024-04-25T05:34:57.450003+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:34:57.450003+0000","last_undegraded":"2024-04-25T05:34:57.450003+0000","last_fullsized":"2024-04-25T05:34:57.450003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.691268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201804+0000","last_change":"2024-04-25T05:34:00.696605+0000","last_active":"2024-04-25T05:34:57.201804+0000","last_peered":"2024-04-25T05:34:57.201804+0000","last_clean":"2024-04-25T05:34:57.201804+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:34:57.201804+0000","last_undegraded":"2024-04-25T05:34:57.201804+0000","last_fullsized":"2024-04-25T05:34:57.201804+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.993568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165030+0000","last_change":"2024-04-25T05:34:04.722629+0000","last_active":"2024-04-25T05:34:56.165030+0000","last_peered":"2024-04-25T05:34:56.165030+0000","last_clean":"2024-04-25T05:34:56.165030+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:34:56.165030+0000","last_undegraded":"2024-04-25T05:34:56.165030+0000","last_fullsized":"2024-04-25T05:34:56.165030+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.965741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449140+0000","last_change":"2024-04-25T05:34:00.675901+0000","last_active":"2024-04-25T05:34:57.449140+0000","last_peered":"2024-04-25T05:34:57.449140+0000","last_clean":"2024-04-25T05:34:57.449140+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:34:57.449140+0000","last_undegraded":"2024-04-25T05:34:57.449140+0000","last_fullsized":"2024-04-25T05:34:57.449140+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.111301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164255+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:34:56.164255+0000","last_peered":"2024-04-25T05:34:56.164255+0000","last_clean":"2024-04-25T05:34:56.164255+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:34:56.164255+0000","last_undegraded":"2024-04-25T05:34:56.164255+0000","last_fullsized":"2024-04-25T05:34:56.164255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.968115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'34","reported_seq":115,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148429+0000","last_change":"2024-04-25T05:34:00.688687+0000","last_active":"2024-04-25T05:34:57.148429+0000","last_peered":"2024-04-25T05:34:57.148429+0000","last_clean":"2024-04-25T05:34:57.148429+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:34:57.148429+0000","last_undegraded":"2024-04-25T05:34:57.148429+0000","last_fullsized":"2024-04-25T05:34:57.148429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.582841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202481+0000","last_change":"2024-04-25T05:34:02.708391+0000","last_active":"2024-04-25T05:34:57.202481+0000","last_peered":"2024-04-25T05:34:57.202481+0000","last_clean":"2024-04-25T05:34:57.202481+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:34:57.202481+0000","last_undegraded":"2024-04-25T05:34:57.202481+0000","last_fullsized":"2024-04-25T05:34:57.202481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.712592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450125+0000","last_change":"2024-04-25T05:34:04.760767+0000","last_active":"2024-04-25T05:34:57.450125+0000","last_peered":"2024-04-25T05:34:57.450125+0000","last_clean":"2024-04-25T05:34:57.450125+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:34:57.450125+0000","last_undegraded":"2024-04-25T05:34:57.450125+0000","last_fullsized":"2024-04-25T05:34:57.450125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.164337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150208+0000","last_change":"2024-04-25T05:33:58.679884+0000","last_active":"2024-04-25T05:34:57.150208+0000","last_peered":"2024-04-25T05:34:57.150208+0000","last_clean":"2024-04-25T05:34:57.150208+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:34:57.150208+0000","last_undegraded":"2024-04-25T05:34:57.150208+0000","last_fullsized":"2024-04-25T05:34:57.150208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.742024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164642+0000","last_change":"2024-04-25T05:34:00.685025+0000","last_active":"2024-04-25T05:34:56.164642+0000","last_peered":"2024-04-25T05:34:56.164642+0000","last_clean":"2024-04-25T05:34:56.164642+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:34:56.164642+0000","last_undegraded":"2024-04-25T05:34:56.164642+0000","last_fullsized":"2024-04-25T05:34:56.164642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.669540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150018+0000","last_change":"2024-04-25T05:34:00.694199+0000","last_active":"2024-04-25T05:34:57.150018+0000","last_peered":"2024-04-25T05:34:57.150018+0000","last_clean":"2024-04-25T05:34:57.150018+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:34:57.150018+0000","last_undegraded":"2024-04-25T05:34:57.150018+0000","last_fullsized":"2024-04-25T05:34:57.150018+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.968271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165588+0000","last_change":"2024-04-25T05:34:02.702812+0000","last_active":"2024-04-25T05:34:56.165588+0000","last_peered":"2024-04-25T05:34:56.165588+0000","last_clean":"2024-04-25T05:34:56.165588+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:34:56.165588+0000","last_undegraded":"2024-04-25T05:34:56.165588+0000","last_fullsized":"2024-04-25T05:34:56.165588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.890640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450722+0000","last_change":"2024-04-25T05:34:04.722965+0000","last_active":"2024-04-25T05:34:57.450722+0000","last_peered":"2024-04-25T05:34:57.450722+0000","last_clean":"2024-04-25T05:34:57.450722+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:34:57.450722+0000","last_undegraded":"2024-04-25T05:34:57.450722+0000","last_fullsized":"2024-04-25T05:34:57.450722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":298,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450212+0000","last_change":"2024-04-25T05:34:20.908770+0000","last_active":"2024-04-25T05:34:57.450212+0000","last_peered":"2024-04-25T05:34:57.450212+0000","last_clean":"2024-04-25T05:34:57.450212+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:34:57.450212+0000","last_undegraded":"2024-04-25T05:34:57.450212+0000","last_fullsized":"2024-04-25T05:34:57.450212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-25T05:34:20.908611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:34:20.908611+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.546403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164954+0000","last_change":"2024-04-25T05:33:58.679230+0000","last_active":"2024-04-25T05:34:56.164954+0000","last_peered":"2024-04-25T05:34:56.164954+0000","last_clean":"2024-04-25T05:34:56.164954+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:34:56.164954+0000","last_undegraded":"2024-04-25T05:34:56.164954+0000","last_fullsized":"2024-04-25T05:34:56.164954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.518886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412600+0000","last_change":"2024-04-25T05:34:01.002201+0000","last_active":"2024-04-25T05:34:56.412600+0000","last_peered":"2024-04-25T05:34:56.412600+0000","last_clean":"2024-04-25T05:34:56.412600+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:34:56.412600+0000","last_undegraded":"2024-04-25T05:34:56.412600+0000","last_fullsized":"2024-04-25T05:34:56.412600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.821400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449936+0000","last_change":"2024-04-25T05:34:02.701047+0000","last_active":"2024-04-25T05:34:57.449936+0000","last_peered":"2024-04-25T05:34:57.449936+0000","last_clean":"2024-04-25T05:34:57.449936+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:34:57.449936+0000","last_undegraded":"2024-04-25T05:34:57.449936+0000","last_fullsized":"2024-04-25T05:34:57.449936+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.440947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149254+0000","last_change":"2024-04-25T05:34:04.726534+0000","last_active":"2024-04-25T05:34:57.149254+0000","last_peered":"2024-04-25T05:34:57.149254+0000","last_clean":"2024-04-25T05:34:57.149254+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:34:57.149254+0000","last_undegraded":"2024-04-25T05:34:57.149254+0000","last_fullsized":"2024-04-25T05:34:57.149254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202234+0000","last_change":"2024-04-25T05:33:58.682492+0000","last_active":"2024-04-25T05:34:57.202234+0000","last_peered":"2024-04-25T05:34:57.202234+0000","last_clean":"2024-04-25T05:34:57.202234+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:34:57.202234+0000","last_undegraded":"2024-04-25T05:34:57.202234+0000","last_fullsized":"2024-04-25T05:34:57.202234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.900372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":111,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450104+0000","last_change":"2024-04-25T05:34:01.000301+0000","last_active":"2024-04-25T05:34:57.450104+0000","last_peered":"2024-04-25T05:34:57.450104+0000","last_clean":"2024-04-25T05:34:57.450104+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:34:57.450104+0000","last_undegraded":"2024-04-25T05:34:57.450104+0000","last_fullsized":"2024-04-25T05:34:57.450104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.243918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201985+0000","last_change":"2024-04-25T05:34:00.694619+0000","last_active":"2024-04-25T05:34:57.201985+0000","last_peered":"2024-04-25T05:34:57.201985+0000","last_clean":"2024-04-25T05:34:57.201985+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:34:57.201985+0000","last_undegraded":"2024-04-25T05:34:57.201985+0000","last_fullsized":"2024-04-25T05:34:57.201985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:01.840274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.451127+0000","last_change":"2024-04-25T05:34:02.709328+0000","last_active":"2024-04-25T05:34:57.451127+0000","last_peered":"2024-04-25T05:34:57.451127+0000","last_clean":"2024-04-25T05:34:57.451127+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:34:57.451127+0000","last_undegraded":"2024-04-25T05:34:57.451127+0000","last_fullsized":"2024-04-25T05:34:57.451127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.085576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148612+0000","last_change":"2024-04-25T05:34:04.759135+0000","last_active":"2024-04-25T05:34:57.148612+0000","last_peered":"2024-04-25T05:34:57.148612+0000","last_clean":"2024-04-25T05:34:57.148612+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:34:57.148612+0000","last_undegraded":"2024-04-25T05:34:57.148612+0000","last_fullsized":"2024-04-25T05:34:57.148612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150090+0000","last_change":"2024-04-25T05:33:58.677172+0000","last_active":"2024-04-25T05:34:57.150090+0000","last_peered":"2024-04-25T05:34:57.150090+0000","last_clean":"2024-04-25T05:34:57.150090+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:34:57.150090+0000","last_undegraded":"2024-04-25T05:34:57.150090+0000","last_fullsized":"2024-04-25T05:34:57.150090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.326701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202454+0000","last_change":"2024-04-25T05:34:00.999939+0000","last_active":"2024-04-25T05:34:57.202454+0000","last_peered":"2024-04-25T05:34:57.202454+0000","last_clean":"2024-04-25T05:34:57.202454+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:34:57.202454+0000","last_undegraded":"2024-04-25T05:34:57.202454+0000","last_fullsized":"2024-04-25T05:34:57.202454+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.684394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412458+0000","last_change":"2024-04-25T05:34:02.708443+0000","last_active":"2024-04-25T05:34:56.412458+0000","last_peered":"2024-04-25T05:34:56.412458+0000","last_clean":"2024-04-25T05:34:56.412458+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:34:56.412458+0000","last_undegraded":"2024-04-25T05:34:56.412458+0000","last_fullsized":"2024-04-25T05:34:56.412458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.195589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201940+0000","last_change":"2024-04-25T05:34:04.728674+0000","last_active":"2024-04-25T05:34:57.201940+0000","last_peered":"2024-04-25T05:34:57.201940+0000","last_clean":"2024-04-25T05:34:57.201940+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:34:57.201940+0000","last_undegraded":"2024-04-25T05:34:57.201940+0000","last_fullsized":"2024-04-25T05:34:57.201940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.707833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450455+0000","last_change":"2024-04-25T05:33:58.678744+0000","last_active":"2024-04-25T05:34:57.450455+0000","last_peered":"2024-04-25T05:34:57.450455+0000","last_clean":"2024-04-25T05:34:57.450455+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:34:57.450455+0000","last_undegraded":"2024-04-25T05:34:57.450455+0000","last_fullsized":"2024-04-25T05:34:57.450455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.579714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148413+0000","last_change":"2024-04-25T05:34:00.688978+0000","last_active":"2024-04-25T05:34:57.148413+0000","last_peered":"2024-04-25T05:34:57.148413+0000","last_clean":"2024-04-25T05:34:57.148413+0000","last_became_active":"2024-04-25T05:34:00.688453+0000","last_became_peered":"2024-04-25T05:34:00.688453+0000","last_unstale":"2024-04-25T05:34:57.148413+0000","last_undegraded":"2024-04-25T05:34:57.148413+0000","last_fullsized":"2024-04-25T05:34:57.148413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.622999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202680+0000","last_change":"2024-04-25T05:34:02.708185+0000","last_active":"2024-04-25T05:34:57.202680+0000","last_peered":"2024-04-25T05:34:57.202680+0000","last_clean":"2024-04-25T05:34:57.202680+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:34:57.202680+0000","last_undegraded":"2024-04-25T05:34:57.202680+0000","last_fullsized":"2024-04-25T05:34:57.202680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.047597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449373+0000","last_change":"2024-04-25T05:34:04.761218+0000","last_active":"2024-04-25T05:34:57.449373+0000","last_peered":"2024-04-25T05:34:57.449373+0000","last_clean":"2024-04-25T05:34:57.449373+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:34:57.449373+0000","last_undegraded":"2024-04-25T05:34:57.449373+0000","last_fullsized":"2024-04-25T05:34:57.449373+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.789438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202392+0000","last_change":"2024-04-25T05:33:58.682409+0000","last_active":"2024-04-25T05:34:57.202392+0000","last_peered":"2024-04-25T05:34:57.202392+0000","last_clean":"2024-04-25T05:34:57.202392+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:34:57.202392+0000","last_undegraded":"2024-04-25T05:34:57.202392+0000","last_fullsized":"2024-04-25T05:34:57.202392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164311+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:34:56.164311+0000","last_peered":"2024-04-25T05:34:56.164311+0000","last_clean":"2024-04-25T05:34:56.164311+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:34:56.164311+0000","last_undegraded":"2024-04-25T05:34:56.164311+0000","last_fullsized":"2024-04-25T05:34:56.164311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.826192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164597+0000","last_change":"2024-04-25T05:34:02.701774+0000","last_active":"2024-04-25T05:34:56.164597+0000","last_peered":"2024-04-25T05:34:56.164597+0000","last_clean":"2024-04-25T05:34:56.164597+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:34:56.164597+0000","last_undegraded":"2024-04-25T05:34:56.164597+0000","last_fullsized":"2024-04-25T05:34:56.164597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.422624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164139+0000","last_change":"2024-04-25T05:34:04.718464+0000","last_active":"2024-04-25T05:34:56.164139+0000","last_peered":"2024-04-25T05:34:56.164139+0000","last_clean":"2024-04-25T05:34:56.164139+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:34:56.164139+0000","last_undegraded":"2024-04-25T05:34:56.164139+0000","last_fullsized":"2024-04-25T05:34:56.164139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166256+0000","last_change":"2024-04-25T05:33:58.676147+0000","last_active":"2024-04-25T05:34:56.166256+0000","last_peered":"2024-04-25T05:34:56.166256+0000","last_clean":"2024-04-25T05:34:56.166256+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:34:56.166256+0000","last_undegraded":"2024-04-25T05:34:56.166256+0000","last_fullsized":"2024-04-25T05:34:56.166256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.509783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":99,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164458+0000","last_change":"2024-04-25T05:34:00.999824+0000","last_active":"2024-04-25T05:34:56.164458+0000","last_peered":"2024-04-25T05:34:56.164458+0000","last_clean":"2024-04-25T05:34:56.164458+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:34:56.164458+0000","last_undegraded":"2024-04-25T05:34:56.164458+0000","last_fullsized":"2024-04-25T05:34:56.164458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.441893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164046+0000","last_change":"2024-04-25T05:34:02.699472+0000","last_active":"2024-04-25T05:34:56.164046+0000","last_peered":"2024-04-25T05:34:56.164046+0000","last_clean":"2024-04-25T05:34:56.164046+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:34:56.164046+0000","last_undegraded":"2024-04-25T05:34:56.164046+0000","last_fullsized":"2024-04-25T05:34:56.164046+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.651777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450121+0000","last_change":"2024-04-25T05:34:04.724152+0000","last_active":"2024-04-25T05:34:57.450121+0000","last_peered":"2024-04-25T05:34:57.450121+0000","last_clean":"2024-04-25T05:34:57.450121+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:34:57.450121+0000","last_undegraded":"2024-04-25T05:34:57.450121+0000","last_fullsized":"2024-04-25T05:34:57.450121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.599745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165221+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:34:56.165221+0000","last_peered":"2024-04-25T05:34:56.165221+0000","last_clean":"2024-04-25T05:34:56.165221+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:34:56.165221+0000","last_undegraded":"2024-04-25T05:34:56.165221+0000","last_fullsized":"2024-04-25T05:34:56.165221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.390939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164334+0000","last_change":"2024-04-25T05:33:58.681579+0000","last_active":"2024-04-25T05:34:56.164334+0000","last_peered":"2024-04-25T05:34:56.164334+0000","last_clean":"2024-04-25T05:34:56.164334+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:34:56.164334+0000","last_undegraded":"2024-04-25T05:34:56.164334+0000","last_fullsized":"2024-04-25T05:34:56.164334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.706716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148511+0000","last_change":"2024-04-25T05:34:00.691497+0000","last_active":"2024-04-25T05:34:57.148511+0000","last_peered":"2024-04-25T05:34:57.148511+0000","last_clean":"2024-04-25T05:34:57.148511+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:34:57.148511+0000","last_undegraded":"2024-04-25T05:34:57.148511+0000","last_fullsized":"2024-04-25T05:34:57.148511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.049287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165051+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:34:56.165051+0000","last_peered":"2024-04-25T05:34:56.165051+0000","last_clean":"2024-04-25T05:34:56.165051+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:34:56.165051+0000","last_undegraded":"2024-04-25T05:34:56.165051+0000","last_fullsized":"2024-04-25T05:34:56.165051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:31.872714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.451031+0000","last_change":"2024-04-25T05:34:02.699262+0000","last_active":"2024-04-25T05:34:57.451031+0000","last_peered":"2024-04-25T05:34:57.451031+0000","last_clean":"2024-04-25T05:34:57.451031+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:34:57.451031+0000","last_undegraded":"2024-04-25T05:34:57.451031+0000","last_fullsized":"2024-04-25T05:34:57.451031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.861356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164103+0000","last_change":"2024-04-25T05:33:58.675050+0000","last_active":"2024-04-25T05:34:56.164103+0000","last_peered":"2024-04-25T05:34:56.164103+0000","last_clean":"2024-04-25T05:34:56.164103+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:34:56.164103+0000","last_undegraded":"2024-04-25T05:34:56.164103+0000","last_fullsized":"2024-04-25T05:34:56.164103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.277179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165094+0000","last_change":"2024-04-25T05:34:00.692677+0000","last_active":"2024-04-25T05:34:56.165094+0000","last_peered":"2024-04-25T05:34:56.165094+0000","last_clean":"2024-04-25T05:34:56.165094+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:34:56.165094+0000","last_undegraded":"2024-04-25T05:34:56.165094+0000","last_fullsized":"2024-04-25T05:34:56.165094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165793+0000","last_change":"2024-04-25T05:34:04.723087+0000","last_active":"2024-04-25T05:34:56.165793+0000","last_peered":"2024-04-25T05:34:56.165793+0000","last_clean":"2024-04-25T05:34:56.165793+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:34:56.165793+0000","last_undegraded":"2024-04-25T05:34:56.165793+0000","last_fullsized":"2024-04-25T05:34:56.165793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.795993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150189+0000","last_change":"2024-04-25T05:34:02.708656+0000","last_active":"2024-04-25T05:34:57.150189+0000","last_peered":"2024-04-25T05:34:57.150189+0000","last_clean":"2024-04-25T05:34:57.150189+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:34:57.150189+0000","last_undegraded":"2024-04-25T05:34:57.150189+0000","last_fullsized":"2024-04-25T05:34:57.150189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.620659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412253+0000","last_change":"2024-04-25T05:33:58.684035+0000","last_active":"2024-04-25T05:34:56.412253+0000","last_peered":"2024-04-25T05:34:56.412253+0000","last_clean":"2024-04-25T05:34:56.412253+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:34:56.412253+0000","last_undegraded":"2024-04-25T05:34:56.412253+0000","last_fullsized":"2024-04-25T05:34:56.412253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.247558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164959+0000","last_change":"2024-04-25T05:34:00.692475+0000","last_active":"2024-04-25T05:34:56.164959+0000","last_peered":"2024-04-25T05:34:56.164959+0000","last_clean":"2024-04-25T05:34:56.164959+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:34:56.164959+0000","last_undegraded":"2024-04-25T05:34:56.164959+0000","last_fullsized":"2024-04-25T05:34:56.164959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.725418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165261+0000","last_change":"2024-04-25T05:34:04.719644+0000","last_active":"2024-04-25T05:34:56.165261+0000","last_peered":"2024-04-25T05:34:56.165261+0000","last_clean":"2024-04-25T05:34:56.165261+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:34:56.165261+0000","last_undegraded":"2024-04-25T05:34:56.165261+0000","last_fullsized":"2024-04-25T05:34:56.165261+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165498+0000","last_change":"2024-04-25T05:34:02.705065+0000","last_active":"2024-04-25T05:34:56.165498+0000","last_peered":"2024-04-25T05:34:56.165498+0000","last_clean":"2024-04-25T05:34:56.165498+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:34:56.165498+0000","last_undegraded":"2024-04-25T05:34:56.165498+0000","last_fullsized":"2024-04-25T05:34:56.165498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.937446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165317+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:34:56.165317+0000","last_peered":"2024-04-25T05:34:56.165317+0000","last_clean":"2024-04-25T05:34:56.165317+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:34:56.165317+0000","last_undegraded":"2024-04-25T05:34:56.165317+0000","last_fullsized":"2024-04-25T05:34:56.165317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.422993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164684+0000","last_change":"2024-04-25T05:34:00.686532+0000","last_active":"2024-04-25T05:34:56.164684+0000","last_peered":"2024-04-25T05:34:56.164684+0000","last_clean":"2024-04-25T05:34:56.164684+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:34:56.164684+0000","last_undegraded":"2024-04-25T05:34:56.164684+0000","last_fullsized":"2024-04-25T05:34:56.164684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.818346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070481,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28328,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716824,"statfs":{"total":95995035648,"available":95966027776,"internally_reserved":0,"allocated":1339392,"data_stored":909804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364821,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1327104,"data_stored":915128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":778240,"data_stored":318694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28320,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716832,"statfs":{"total":95995035648,"available":95966035968,"internally_reserved":0,"allocated":1331200,"data_stored":908071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":724992,"data_stored":323492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":324222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":761856,"data_stored":324942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691866,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":317704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:34:59.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:35:00.179 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:00.429 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:00.430 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:00.880 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:35:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:35:01.141 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:35:01.120Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000313324s 2024-04-25T05:35:01.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-25T05:35:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: Deploying daemon grafana.a on smithi060 2024-04-25T05:35:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:35:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: 3.1d deep-scrub starts 2024-04-25T05:35:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:01 smithi060 ceph-mon[30898]: 3.1d deep-scrub ok 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: Deploying daemon grafana.a on smithi060 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: 3.1d deep-scrub starts 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[27010]: 3.1d deep-scrub ok 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-25T05:35:01.579 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-25T05:35:01.580 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: Deploying daemon grafana.a on smithi060 2024-04-25T05:35:01.580 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:35:01.580 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: 3.1d deep-scrub starts 2024-04-25T05:35:01.580 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:01 smithi043 ceph-mon[37492]: 3.1d deep-scrub ok 2024-04-25T05:35:01.612 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-25T05:34:59.659585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1275,"num_read_kb":1089,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236212,"kb_used_data":7564,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749725004,"statfs":{"total":767960285184,"available":767718404096,"internally_reserved":0,"allocated":7745536,"data_stored":4342057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002198"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164073+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:34:56.164073+0000","last_peered":"2024-04-25T05:34:56.164073+0000","last_clean":"2024-04-25T05:34:56.164073+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:34:56.164073+0000","last_undegraded":"2024-04-25T05:34:56.164073+0000","last_fullsized":"2024-04-25T05:34:56.164073+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.921834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148652+0000","last_change":"2024-04-25T05:34:02.700825+0000","last_active":"2024-04-25T05:34:57.148652+0000","last_peered":"2024-04-25T05:34:57.148652+0000","last_clean":"2024-04-25T05:34:57.148652+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:34:57.148652+0000","last_undegraded":"2024-04-25T05:34:57.148652+0000","last_fullsized":"2024-04-25T05:34:57.148652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.901521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.203102+0000","last_change":"2024-04-25T05:33:58.677608+0000","last_active":"2024-04-25T05:34:57.203102+0000","last_peered":"2024-04-25T05:34:57.203102+0000","last_clean":"2024-04-25T05:34:57.203102+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:34:57.203102+0000","last_undegraded":"2024-04-25T05:34:57.203102+0000","last_fullsized":"2024-04-25T05:34:57.203102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.073153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.163918+0000","last_change":"2024-04-25T05:34:00.692628+0000","last_active":"2024-04-25T05:34:56.163918+0000","last_peered":"2024-04-25T05:34:56.163918+0000","last_clean":"2024-04-25T05:34:56.163918+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:34:56.163918+0000","last_undegraded":"2024-04-25T05:34:56.163918+0000","last_fullsized":"2024-04-25T05:34:56.163918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.003557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165182+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:34:56.165182+0000","last_peered":"2024-04-25T05:34:56.165182+0000","last_clean":"2024-04-25T05:34:56.165182+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:34:56.165182+0000","last_undegraded":"2024-04-25T05:34:56.165182+0000","last_fullsized":"2024-04-25T05:34:56.165182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.498112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.163943+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:34:56.163943+0000","last_peered":"2024-04-25T05:34:56.163943+0000","last_clean":"2024-04-25T05:34:56.163943+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:34:56.163943+0000","last_undegraded":"2024-04-25T05:34:56.163943+0000","last_fullsized":"2024-04-25T05:34:56.163943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.555566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165373+0000","last_change":"2024-04-25T05:33:58.679726+0000","last_active":"2024-04-25T05:34:56.165373+0000","last_peered":"2024-04-25T05:34:56.165373+0000","last_clean":"2024-04-25T05:34:56.165373+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:34:56.165373+0000","last_undegraded":"2024-04-25T05:34:56.165373+0000","last_fullsized":"2024-04-25T05:34:56.165373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.377018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":116,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165908+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:34:56.165908+0000","last_peered":"2024-04-25T05:34:56.165908+0000","last_clean":"2024-04-25T05:34:56.165908+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:34:56.165908+0000","last_undegraded":"2024-04-25T05:34:56.165908+0000","last_fullsized":"2024-04-25T05:34:56.165908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.111490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202123+0000","last_change":"2024-04-25T05:34:04.724350+0000","last_active":"2024-04-25T05:34:57.202123+0000","last_peered":"2024-04-25T05:34:57.202123+0000","last_clean":"2024-04-25T05:34:57.202123+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:34:57.202123+0000","last_undegraded":"2024-04-25T05:34:57.202123+0000","last_fullsized":"2024-04-25T05:34:57.202123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.943468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449490+0000","last_change":"2024-04-25T05:34:02.709504+0000","last_active":"2024-04-25T05:34:57.449490+0000","last_peered":"2024-04-25T05:34:57.449490+0000","last_clean":"2024-04-25T05:34:57.449490+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:34:57.449490+0000","last_undegraded":"2024-04-25T05:34:57.449490+0000","last_fullsized":"2024-04-25T05:34:57.449490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.151709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450234+0000","last_change":"2024-04-25T05:33:58.687700+0000","last_active":"2024-04-25T05:34:57.450234+0000","last_peered":"2024-04-25T05:34:57.450234+0000","last_clean":"2024-04-25T05:34:57.450234+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:34:57.450234+0000","last_undegraded":"2024-04-25T05:34:57.450234+0000","last_fullsized":"2024-04-25T05:34:57.450234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.471958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164328+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:34:56.164328+0000","last_peered":"2024-04-25T05:34:56.164328+0000","last_clean":"2024-04-25T05:34:56.164328+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:34:56.164328+0000","last_undegraded":"2024-04-25T05:34:56.164328+0000","last_fullsized":"2024-04-25T05:34:56.164328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.181248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448783+0000","last_change":"2024-04-25T05:34:04.722769+0000","last_active":"2024-04-25T05:34:57.448783+0000","last_peered":"2024-04-25T05:34:57.448783+0000","last_clean":"2024-04-25T05:34:57.448783+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:34:57.448783+0000","last_undegraded":"2024-04-25T05:34:57.448783+0000","last_fullsized":"2024-04-25T05:34:57.448783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.673388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448778+0000","last_change":"2024-04-25T05:34:02.701665+0000","last_active":"2024-04-25T05:34:57.448778+0000","last_peered":"2024-04-25T05:34:57.448778+0000","last_clean":"2024-04-25T05:34:57.448778+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:34:57.448778+0000","last_undegraded":"2024-04-25T05:34:57.448778+0000","last_fullsized":"2024-04-25T05:34:57.448778+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.814291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449882+0000","last_change":"2024-04-25T05:33:58.688919+0000","last_active":"2024-04-25T05:34:57.449882+0000","last_peered":"2024-04-25T05:34:57.449882+0000","last_clean":"2024-04-25T05:34:57.449882+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:34:57.449882+0000","last_undegraded":"2024-04-25T05:34:57.449882+0000","last_fullsized":"2024-04-25T05:34:57.449882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.028583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164455+0000","last_change":"2024-04-25T05:34:00.693330+0000","last_active":"2024-04-25T05:34:56.164455+0000","last_peered":"2024-04-25T05:34:56.164455+0000","last_clean":"2024-04-25T05:34:56.164455+0000","last_became_active":"2024-04-25T05:34:00.693003+0000","last_became_peered":"2024-04-25T05:34:00.693003+0000","last_unstale":"2024-04-25T05:34:56.164455+0000","last_undegraded":"2024-04-25T05:34:56.164455+0000","last_fullsized":"2024-04-25T05:34:56.164455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.762449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165128+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:34:56.165128+0000","last_peered":"2024-04-25T05:34:56.165128+0000","last_clean":"2024-04-25T05:34:56.165128+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:34:56.165128+0000","last_undegraded":"2024-04-25T05:34:56.165128+0000","last_fullsized":"2024-04-25T05:34:56.165128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.388118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149081+0000","last_change":"2024-04-25T05:34:02.708603+0000","last_active":"2024-04-25T05:34:57.149081+0000","last_peered":"2024-04-25T05:34:57.149081+0000","last_clean":"2024-04-25T05:34:57.149081+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:34:57.149081+0000","last_undegraded":"2024-04-25T05:34:57.149081+0000","last_fullsized":"2024-04-25T05:34:57.149081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.908190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202569+0000","last_change":"2024-04-25T05:33:58.683934+0000","last_active":"2024-04-25T05:34:57.202569+0000","last_peered":"2024-04-25T05:34:57.202569+0000","last_clean":"2024-04-25T05:34:57.202569+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:34:57.202569+0000","last_undegraded":"2024-04-25T05:34:57.202569+0000","last_fullsized":"2024-04-25T05:34:57.202569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.927389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164448+0000","last_change":"2024-04-25T05:34:00.690831+0000","last_active":"2024-04-25T05:34:56.164448+0000","last_peered":"2024-04-25T05:34:56.164448+0000","last_clean":"2024-04-25T05:34:56.164448+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:34:56.164448+0000","last_undegraded":"2024-04-25T05:34:56.164448+0000","last_fullsized":"2024-04-25T05:34:56.164448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.940787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166455+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:34:56.166455+0000","last_peered":"2024-04-25T05:34:56.166455+0000","last_clean":"2024-04-25T05:34:56.166455+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:34:56.166455+0000","last_undegraded":"2024-04-25T05:34:56.166455+0000","last_fullsized":"2024-04-25T05:34:56.166455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164666+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:34:56.164666+0000","last_peered":"2024-04-25T05:34:56.164666+0000","last_clean":"2024-04-25T05:34:56.164666+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:34:56.164666+0000","last_undegraded":"2024-04-25T05:34:56.164666+0000","last_fullsized":"2024-04-25T05:34:56.164666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.792766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164332+0000","last_change":"2024-04-25T05:33:58.677738+0000","last_active":"2024-04-25T05:34:56.164332+0000","last_peered":"2024-04-25T05:34:56.164332+0000","last_clean":"2024-04-25T05:34:56.164332+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:34:56.164332+0000","last_undegraded":"2024-04-25T05:34:56.164332+0000","last_fullsized":"2024-04-25T05:34:56.164332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.705248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164690+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:34:56.164690+0000","last_peered":"2024-04-25T05:34:56.164690+0000","last_clean":"2024-04-25T05:34:56.164690+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:34:56.164690+0000","last_undegraded":"2024-04-25T05:34:56.164690+0000","last_fullsized":"2024-04-25T05:34:56.164690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.537702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449031+0000","last_change":"2024-04-25T05:34:04.725886+0000","last_active":"2024-04-25T05:34:57.449031+0000","last_peered":"2024-04-25T05:34:57.449031+0000","last_clean":"2024-04-25T05:34:57.449031+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:34:57.449031+0000","last_undegraded":"2024-04-25T05:34:57.449031+0000","last_fullsized":"2024-04-25T05:34:57.449031+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412842+0000","last_change":"2024-04-25T05:34:02.701003+0000","last_active":"2024-04-25T05:34:56.412842+0000","last_peered":"2024-04-25T05:34:56.412842+0000","last_clean":"2024-04-25T05:34:56.412842+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:34:56.412842+0000","last_undegraded":"2024-04-25T05:34:56.412842+0000","last_fullsized":"2024-04-25T05:34:56.412842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.795737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149927+0000","last_change":"2024-04-25T05:33:58.678204+0000","last_active":"2024-04-25T05:34:57.149927+0000","last_peered":"2024-04-25T05:34:57.149927+0000","last_clean":"2024-04-25T05:34:57.149927+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:34:57.149927+0000","last_undegraded":"2024-04-25T05:34:57.149927+0000","last_fullsized":"2024-04-25T05:34:57.149927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.231648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201870+0000","last_change":"2024-04-25T05:34:01.002306+0000","last_active":"2024-04-25T05:34:57.201870+0000","last_peered":"2024-04-25T05:34:57.201870+0000","last_clean":"2024-04-25T05:34:57.201870+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:34:57.201870+0000","last_undegraded":"2024-04-25T05:34:57.201870+0000","last_fullsized":"2024-04-25T05:34:57.201870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.875946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148890+0000","last_change":"2024-04-25T05:34:04.722159+0000","last_active":"2024-04-25T05:34:57.148890+0000","last_peered":"2024-04-25T05:34:57.148890+0000","last_clean":"2024-04-25T05:34:57.148890+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:34:57.148890+0000","last_undegraded":"2024-04-25T05:34:57.148890+0000","last_fullsized":"2024-04-25T05:34:57.148890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.853709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202633+0000","last_change":"2024-04-25T05:34:02.710777+0000","last_active":"2024-04-25T05:34:57.202633+0000","last_peered":"2024-04-25T05:34:57.202633+0000","last_clean":"2024-04-25T05:34:57.202633+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:34:57.202633+0000","last_undegraded":"2024-04-25T05:34:57.202633+0000","last_fullsized":"2024-04-25T05:34:57.202633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.845754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164199+0000","last_change":"2024-04-25T05:33:58.676498+0000","last_active":"2024-04-25T05:34:56.164199+0000","last_peered":"2024-04-25T05:34:56.164199+0000","last_clean":"2024-04-25T05:34:56.164199+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:34:56.164199+0000","last_undegraded":"2024-04-25T05:34:56.164199+0000","last_fullsized":"2024-04-25T05:34:56.164199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.714656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412815+0000","last_change":"2024-04-25T05:34:00.691266+0000","last_active":"2024-04-25T05:34:56.412815+0000","last_peered":"2024-04-25T05:34:56.412815+0000","last_clean":"2024-04-25T05:34:56.412815+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:34:56.412815+0000","last_undegraded":"2024-04-25T05:34:56.412815+0000","last_fullsized":"2024-04-25T05:34:56.412815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.730290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448588+0000","last_change":"2024-04-25T05:34:04.725684+0000","last_active":"2024-04-25T05:34:57.448588+0000","last_peered":"2024-04-25T05:34:57.448588+0000","last_clean":"2024-04-25T05:34:57.448588+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:34:57.448588+0000","last_undegraded":"2024-04-25T05:34:57.448588+0000","last_fullsized":"2024-04-25T05:34:57.448588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.515944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412053+0000","last_change":"2024-04-25T05:34:02.701292+0000","last_active":"2024-04-25T05:34:56.412053+0000","last_peered":"2024-04-25T05:34:56.412053+0000","last_clean":"2024-04-25T05:34:56.412053+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:34:56.412053+0000","last_undegraded":"2024-04-25T05:34:56.412053+0000","last_fullsized":"2024-04-25T05:34:56.412053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.466742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202631+0000","last_change":"2024-04-25T05:33:58.678452+0000","last_active":"2024-04-25T05:34:57.202631+0000","last_peered":"2024-04-25T05:34:57.202631+0000","last_clean":"2024-04-25T05:34:57.202631+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:34:57.202631+0000","last_undegraded":"2024-04-25T05:34:57.202631+0000","last_fullsized":"2024-04-25T05:34:57.202631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.043078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164255+0000","last_change":"2024-04-25T05:34:00.692546+0000","last_active":"2024-04-25T05:34:56.164255+0000","last_peered":"2024-04-25T05:34:56.164255+0000","last_clean":"2024-04-25T05:34:56.164255+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:34:56.164255+0000","last_undegraded":"2024-04-25T05:34:56.164255+0000","last_fullsized":"2024-04-25T05:34:56.164255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.114492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165040+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:34:56.165040+0000","last_peered":"2024-04-25T05:34:56.165040+0000","last_clean":"2024-04-25T05:34:56.165040+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:34:56.165040+0000","last_undegraded":"2024-04-25T05:34:56.165040+0000","last_fullsized":"2024-04-25T05:34:56.165040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.872916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450336+0000","last_change":"2024-04-25T05:34:02.701784+0000","last_active":"2024-04-25T05:34:57.450336+0000","last_peered":"2024-04-25T05:34:57.450336+0000","last_clean":"2024-04-25T05:34:57.450336+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:34:57.450336+0000","last_undegraded":"2024-04-25T05:34:57.450336+0000","last_fullsized":"2024-04-25T05:34:57.450336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.034685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450594+0000","last_change":"2024-04-25T05:33:58.679682+0000","last_active":"2024-04-25T05:34:57.450594+0000","last_peered":"2024-04-25T05:34:57.450594+0000","last_clean":"2024-04-25T05:34:57.450594+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:34:57.450594+0000","last_undegraded":"2024-04-25T05:34:57.450594+0000","last_fullsized":"2024-04-25T05:34:57.450594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164904+0000","last_change":"2024-04-25T05:34:00.999704+0000","last_active":"2024-04-25T05:34:56.164904+0000","last_peered":"2024-04-25T05:34:56.164904+0000","last_clean":"2024-04-25T05:34:56.164904+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:34:56.164904+0000","last_undegraded":"2024-04-25T05:34:56.164904+0000","last_fullsized":"2024-04-25T05:34:56.164904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.051072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201928+0000","last_change":"2024-04-25T05:34:04.727071+0000","last_active":"2024-04-25T05:34:57.201928+0000","last_peered":"2024-04-25T05:34:57.201928+0000","last_clean":"2024-04-25T05:34:57.201928+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:34:57.201928+0000","last_undegraded":"2024-04-25T05:34:57.201928+0000","last_fullsized":"2024-04-25T05:34:57.201928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.039265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165288+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:34:56.165288+0000","last_peered":"2024-04-25T05:34:56.165288+0000","last_clean":"2024-04-25T05:34:56.165288+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:34:56.165288+0000","last_undegraded":"2024-04-25T05:34:56.165288+0000","last_fullsized":"2024-04-25T05:34:56.165288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.038248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449403+0000","last_change":"2024-04-25T05:33:58.687406+0000","last_active":"2024-04-25T05:34:57.449403+0000","last_peered":"2024-04-25T05:34:57.449403+0000","last_clean":"2024-04-25T05:34:57.449403+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:34:57.449403+0000","last_undegraded":"2024-04-25T05:34:57.449403+0000","last_fullsized":"2024-04-25T05:34:57.449403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.844471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148255+0000","last_change":"2024-04-25T05:34:00.689589+0000","last_active":"2024-04-25T05:34:57.148255+0000","last_peered":"2024-04-25T05:34:57.148255+0000","last_clean":"2024-04-25T05:34:57.148255+0000","last_became_active":"2024-04-25T05:34:00.688897+0000","last_became_peered":"2024-04-25T05:34:00.688897+0000","last_unstale":"2024-04-25T05:34:57.148255+0000","last_undegraded":"2024-04-25T05:34:57.148255+0000","last_fullsized":"2024-04-25T05:34:57.148255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164404+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:34:56.164404+0000","last_peered":"2024-04-25T05:34:56.164404+0000","last_clean":"2024-04-25T05:34:56.164404+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:34:56.164404+0000","last_undegraded":"2024-04-25T05:34:56.164404+0000","last_fullsized":"2024-04-25T05:34:56.164404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.871648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149997+0000","last_change":"2024-04-25T05:34:02.699322+0000","last_active":"2024-04-25T05:34:57.149997+0000","last_peered":"2024-04-25T05:34:57.149997+0000","last_clean":"2024-04-25T05:34:57.149997+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:34:57.149997+0000","last_undegraded":"2024-04-25T05:34:57.149997+0000","last_fullsized":"2024-04-25T05:34:57.149997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.105773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164821+0000","last_change":"2024-04-25T05:33:58.680453+0000","last_active":"2024-04-25T05:34:56.164821+0000","last_peered":"2024-04-25T05:34:56.164821+0000","last_clean":"2024-04-25T05:34:56.164821+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:34:56.164821+0000","last_undegraded":"2024-04-25T05:34:56.164821+0000","last_fullsized":"2024-04-25T05:34:56.164821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.585516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164558+0000","last_change":"2024-04-25T05:34:00.693320+0000","last_active":"2024-04-25T05:34:56.164558+0000","last_peered":"2024-04-25T05:34:56.164558+0000","last_clean":"2024-04-25T05:34:56.164558+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:34:56.164558+0000","last_undegraded":"2024-04-25T05:34:56.164558+0000","last_fullsized":"2024-04-25T05:34:56.164558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.690705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202562+0000","last_change":"2024-04-25T05:34:04.724697+0000","last_active":"2024-04-25T05:34:57.202562+0000","last_peered":"2024-04-25T05:34:57.202562+0000","last_clean":"2024-04-25T05:34:57.202562+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:34:57.202562+0000","last_undegraded":"2024-04-25T05:34:57.202562+0000","last_fullsized":"2024-04-25T05:34:57.202562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.282641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450340+0000","last_change":"2024-04-25T05:34:02.708587+0000","last_active":"2024-04-25T05:34:57.450340+0000","last_peered":"2024-04-25T05:34:57.450340+0000","last_clean":"2024-04-25T05:34:57.450340+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:34:57.450340+0000","last_undegraded":"2024-04-25T05:34:57.450340+0000","last_fullsized":"2024-04-25T05:34:57.450340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.470304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448963+0000","last_change":"2024-04-25T05:33:58.685869+0000","last_active":"2024-04-25T05:34:57.448963+0000","last_peered":"2024-04-25T05:34:57.448963+0000","last_clean":"2024-04-25T05:34:57.448963+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:34:57.448963+0000","last_undegraded":"2024-04-25T05:34:57.448963+0000","last_fullsized":"2024-04-25T05:34:57.448963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.260635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165186+0000","last_change":"2024-04-25T05:34:00.999737+0000","last_active":"2024-04-25T05:34:56.165186+0000","last_peered":"2024-04-25T05:34:56.165186+0000","last_clean":"2024-04-25T05:34:56.165186+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:34:56.165186+0000","last_undegraded":"2024-04-25T05:34:56.165186+0000","last_fullsized":"2024-04-25T05:34:56.165186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.305882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165280+0000","last_change":"2024-04-25T05:34:04.715688+0000","last_active":"2024-04-25T05:34:56.165280+0000","last_peered":"2024-04-25T05:34:56.165280+0000","last_clean":"2024-04-25T05:34:56.165280+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:34:56.165280+0000","last_undegraded":"2024-04-25T05:34:56.165280+0000","last_fullsized":"2024-04-25T05:34:56.165280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.959881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166431+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:34:56.166431+0000","last_peered":"2024-04-25T05:34:56.166431+0000","last_clean":"2024-04-25T05:34:56.166431+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:34:56.166431+0000","last_undegraded":"2024-04-25T05:34:56.166431+0000","last_fullsized":"2024-04-25T05:34:56.166431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.077903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202231+0000","last_change":"2024-04-25T05:33:58.685955+0000","last_active":"2024-04-25T05:34:57.202231+0000","last_peered":"2024-04-25T05:34:57.202231+0000","last_clean":"2024-04-25T05:34:57.202231+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:34:57.202231+0000","last_undegraded":"2024-04-25T05:34:57.202231+0000","last_fullsized":"2024-04-25T05:34:57.202231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.649756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448747+0000","last_change":"2024-04-25T05:34:01.002652+0000","last_active":"2024-04-25T05:34:57.448747+0000","last_peered":"2024-04-25T05:34:57.448747+0000","last_clean":"2024-04-25T05:34:57.448747+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:34:57.448747+0000","last_undegraded":"2024-04-25T05:34:57.448747+0000","last_fullsized":"2024-04-25T05:34:57.448747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.164431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164778+0000","last_change":"2024-04-25T05:34:04.719603+0000","last_active":"2024-04-25T05:34:56.164778+0000","last_peered":"2024-04-25T05:34:56.164778+0000","last_clean":"2024-04-25T05:34:56.164778+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:34:56.164778+0000","last_undegraded":"2024-04-25T05:34:56.164778+0000","last_fullsized":"2024-04-25T05:34:56.164778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.818750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166430+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:34:56.166430+0000","last_peered":"2024-04-25T05:34:56.166430+0000","last_clean":"2024-04-25T05:34:56.166430+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:34:56.166430+0000","last_undegraded":"2024-04-25T05:34:56.166430+0000","last_fullsized":"2024-04-25T05:34:56.166430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.004188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448679+0000","last_change":"2024-04-25T05:33:58.685792+0000","last_active":"2024-04-25T05:34:57.448679+0000","last_peered":"2024-04-25T05:34:57.448679+0000","last_clean":"2024-04-25T05:34:57.448679+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:34:57.448679+0000","last_undegraded":"2024-04-25T05:34:57.448679+0000","last_fullsized":"2024-04-25T05:34:57.448679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.497320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201838+0000","last_change":"2024-04-25T05:34:01.002399+0000","last_active":"2024-04-25T05:34:57.201838+0000","last_peered":"2024-04-25T05:34:57.201838+0000","last_clean":"2024-04-25T05:34:57.201838+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:34:57.201838+0000","last_undegraded":"2024-04-25T05:34:57.201838+0000","last_fullsized":"2024-04-25T05:34:57.201838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.611153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449281+0000","last_change":"2024-04-25T05:34:04.760885+0000","last_active":"2024-04-25T05:34:57.449281+0000","last_peered":"2024-04-25T05:34:57.449281+0000","last_clean":"2024-04-25T05:34:57.449281+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:34:57.449281+0000","last_undegraded":"2024-04-25T05:34:57.449281+0000","last_fullsized":"2024-04-25T05:34:57.449281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.397373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450595+0000","last_change":"2024-04-25T05:34:02.700150+0000","last_active":"2024-04-25T05:34:57.450595+0000","last_peered":"2024-04-25T05:34:57.450595+0000","last_clean":"2024-04-25T05:34:57.450595+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:34:57.450595+0000","last_undegraded":"2024-04-25T05:34:57.450595+0000","last_fullsized":"2024-04-25T05:34:57.450595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.726607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.411961+0000","last_change":"2024-04-25T05:33:58.682812+0000","last_active":"2024-04-25T05:34:56.411961+0000","last_peered":"2024-04-25T05:34:56.411961+0000","last_clean":"2024-04-25T05:34:56.411961+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:34:56.411961+0000","last_undegraded":"2024-04-25T05:34:56.411961+0000","last_fullsized":"2024-04-25T05:34:56.411961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.737056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166548+0000","last_change":"2024-04-25T05:34:01.000303+0000","last_active":"2024-04-25T05:34:56.166548+0000","last_peered":"2024-04-25T05:34:56.166548+0000","last_clean":"2024-04-25T05:34:56.166548+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:34:56.166548+0000","last_undegraded":"2024-04-25T05:34:56.166548+0000","last_fullsized":"2024-04-25T05:34:56.166548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.779729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449287+0000","last_change":"2024-04-25T05:34:04.759369+0000","last_active":"2024-04-25T05:34:57.449287+0000","last_peered":"2024-04-25T05:34:57.449287+0000","last_clean":"2024-04-25T05:34:57.449287+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:34:57.449287+0000","last_undegraded":"2024-04-25T05:34:57.449287+0000","last_fullsized":"2024-04-25T05:34:57.449287+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.510841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449983+0000","last_change":"2024-04-25T05:33:58.689200+0000","last_active":"2024-04-25T05:34:57.449983+0000","last_peered":"2024-04-25T05:34:57.449983+0000","last_clean":"2024-04-25T05:34:57.449983+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:34:57.449983+0000","last_undegraded":"2024-04-25T05:34:57.449983+0000","last_fullsized":"2024-04-25T05:34:57.449983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.468188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":92,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.448403+0000","last_change":"2024-04-25T05:34:01.000840+0000","last_active":"2024-04-25T05:34:57.448403+0000","last_peered":"2024-04-25T05:34:57.448403+0000","last_clean":"2024-04-25T05:34:57.448403+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:34:57.448403+0000","last_undegraded":"2024-04-25T05:34:57.448403+0000","last_fullsized":"2024-04-25T05:34:57.448403+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.859671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449637+0000","last_change":"2024-04-25T05:34:02.701548+0000","last_active":"2024-04-25T05:34:57.449637+0000","last_peered":"2024-04-25T05:34:57.449637+0000","last_clean":"2024-04-25T05:34:57.449637+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:34:57.449637+0000","last_undegraded":"2024-04-25T05:34:57.449637+0000","last_fullsized":"2024-04-25T05:34:57.449637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:02.747443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165060+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:34:56.165060+0000","last_peered":"2024-04-25T05:34:56.165060+0000","last_clean":"2024-04-25T05:34:56.165060+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:34:56.165060+0000","last_undegraded":"2024-04-25T05:34:56.165060+0000","last_fullsized":"2024-04-25T05:34:56.165060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.987326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449772+0000","last_change":"2024-04-25T05:33:58.687364+0000","last_active":"2024-04-25T05:34:57.449772+0000","last_peered":"2024-04-25T05:34:57.449772+0000","last_clean":"2024-04-25T05:34:57.449772+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:34:57.449772+0000","last_undegraded":"2024-04-25T05:34:57.449772+0000","last_fullsized":"2024-04-25T05:34:57.449772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.749177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":90,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164422+0000","last_change":"2024-04-25T05:34:00.694001+0000","last_active":"2024-04-25T05:34:56.164422+0000","last_peered":"2024-04-25T05:34:56.164422+0000","last_clean":"2024-04-25T05:34:56.164422+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:34:56.164422+0000","last_undegraded":"2024-04-25T05:34:56.164422+0000","last_fullsized":"2024-04-25T05:34:56.164422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.293624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165138+0000","last_change":"2024-04-25T05:34:02.710703+0000","last_active":"2024-04-25T05:34:56.165138+0000","last_peered":"2024-04-25T05:34:56.165138+0000","last_clean":"2024-04-25T05:34:56.165138+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:34:56.165138+0000","last_undegraded":"2024-04-25T05:34:56.165138+0000","last_fullsized":"2024-04-25T05:34:56.165138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.457617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450099+0000","last_change":"2024-04-25T05:34:04.727176+0000","last_active":"2024-04-25T05:34:57.450099+0000","last_peered":"2024-04-25T05:34:57.450099+0000","last_clean":"2024-04-25T05:34:57.450099+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:34:57.450099+0000","last_undegraded":"2024-04-25T05:34:57.450099+0000","last_fullsized":"2024-04-25T05:34:57.450099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165412+0000","last_change":"2024-04-25T05:34:02.710395+0000","last_active":"2024-04-25T05:34:56.165412+0000","last_peered":"2024-04-25T05:34:56.165412+0000","last_clean":"2024-04-25T05:34:56.165412+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:34:56.165412+0000","last_undegraded":"2024-04-25T05:34:56.165412+0000","last_fullsized":"2024-04-25T05:34:56.165412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.749912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449750+0000","last_change":"2024-04-25T05:33:58.686463+0000","last_active":"2024-04-25T05:34:57.449750+0000","last_peered":"2024-04-25T05:34:57.449750+0000","last_clean":"2024-04-25T05:34:57.449750+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:34:57.449750+0000","last_undegraded":"2024-04-25T05:34:57.449750+0000","last_fullsized":"2024-04-25T05:34:57.449750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:59.578656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412421+0000","last_change":"2024-04-25T05:34:00.999892+0000","last_active":"2024-04-25T05:34:56.412421+0000","last_peered":"2024-04-25T05:34:56.412421+0000","last_clean":"2024-04-25T05:34:56.412421+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:34:56.412421+0000","last_undegraded":"2024-04-25T05:34:56.412421+0000","last_fullsized":"2024-04-25T05:34:56.412421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.466514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202039+0000","last_change":"2024-04-25T05:34:04.725833+0000","last_active":"2024-04-25T05:34:57.202039+0000","last_peered":"2024-04-25T05:34:57.202039+0000","last_clean":"2024-04-25T05:34:57.202039+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:34:57.202039+0000","last_undegraded":"2024-04-25T05:34:57.202039+0000","last_fullsized":"2024-04-25T05:34:57.202039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165080+0000","last_change":"2024-04-25T05:34:02.700901+0000","last_active":"2024-04-25T05:34:56.165080+0000","last_peered":"2024-04-25T05:34:56.165080+0000","last_clean":"2024-04-25T05:34:56.165080+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:34:56.165080+0000","last_undegraded":"2024-04-25T05:34:56.165080+0000","last_fullsized":"2024-04-25T05:34:56.165080+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.464318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450003+0000","last_change":"2024-04-25T05:33:58.685937+0000","last_active":"2024-04-25T05:34:57.450003+0000","last_peered":"2024-04-25T05:34:57.450003+0000","last_clean":"2024-04-25T05:34:57.450003+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:34:57.450003+0000","last_undegraded":"2024-04-25T05:34:57.450003+0000","last_fullsized":"2024-04-25T05:34:57.450003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.691268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201804+0000","last_change":"2024-04-25T05:34:00.696605+0000","last_active":"2024-04-25T05:34:57.201804+0000","last_peered":"2024-04-25T05:34:57.201804+0000","last_clean":"2024-04-25T05:34:57.201804+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:34:57.201804+0000","last_undegraded":"2024-04-25T05:34:57.201804+0000","last_fullsized":"2024-04-25T05:34:57.201804+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.993568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165030+0000","last_change":"2024-04-25T05:34:04.722629+0000","last_active":"2024-04-25T05:34:56.165030+0000","last_peered":"2024-04-25T05:34:56.165030+0000","last_clean":"2024-04-25T05:34:56.165030+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:34:56.165030+0000","last_undegraded":"2024-04-25T05:34:56.165030+0000","last_fullsized":"2024-04-25T05:34:56.165030+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:11.965741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449140+0000","last_change":"2024-04-25T05:34:00.675901+0000","last_active":"2024-04-25T05:34:57.449140+0000","last_peered":"2024-04-25T05:34:57.449140+0000","last_clean":"2024-04-25T05:34:57.449140+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:34:57.449140+0000","last_undegraded":"2024-04-25T05:34:57.449140+0000","last_fullsized":"2024-04-25T05:34:57.449140+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.111301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164255+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:34:56.164255+0000","last_peered":"2024-04-25T05:34:56.164255+0000","last_clean":"2024-04-25T05:34:56.164255+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:34:56.164255+0000","last_undegraded":"2024-04-25T05:34:56.164255+0000","last_fullsized":"2024-04-25T05:34:56.164255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.968115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'34","reported_seq":115,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148429+0000","last_change":"2024-04-25T05:34:00.688687+0000","last_active":"2024-04-25T05:34:57.148429+0000","last_peered":"2024-04-25T05:34:57.148429+0000","last_clean":"2024-04-25T05:34:57.148429+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:34:57.148429+0000","last_undegraded":"2024-04-25T05:34:57.148429+0000","last_fullsized":"2024-04-25T05:34:57.148429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.582841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202481+0000","last_change":"2024-04-25T05:34:02.708391+0000","last_active":"2024-04-25T05:34:57.202481+0000","last_peered":"2024-04-25T05:34:57.202481+0000","last_clean":"2024-04-25T05:34:57.202481+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:34:57.202481+0000","last_undegraded":"2024-04-25T05:34:57.202481+0000","last_fullsized":"2024-04-25T05:34:57.202481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.712592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450125+0000","last_change":"2024-04-25T05:34:04.760767+0000","last_active":"2024-04-25T05:34:57.450125+0000","last_peered":"2024-04-25T05:34:57.450125+0000","last_clean":"2024-04-25T05:34:57.450125+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:34:57.450125+0000","last_undegraded":"2024-04-25T05:34:57.450125+0000","last_fullsized":"2024-04-25T05:34:57.450125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.164337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150208+0000","last_change":"2024-04-25T05:33:58.679884+0000","last_active":"2024-04-25T05:34:57.150208+0000","last_peered":"2024-04-25T05:34:57.150208+0000","last_clean":"2024-04-25T05:34:57.150208+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:34:57.150208+0000","last_undegraded":"2024-04-25T05:34:57.150208+0000","last_fullsized":"2024-04-25T05:34:57.150208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.742024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164642+0000","last_change":"2024-04-25T05:34:00.685025+0000","last_active":"2024-04-25T05:34:56.164642+0000","last_peered":"2024-04-25T05:34:56.164642+0000","last_clean":"2024-04-25T05:34:56.164642+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:34:56.164642+0000","last_undegraded":"2024-04-25T05:34:56.164642+0000","last_fullsized":"2024-04-25T05:34:56.164642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:00.669540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150018+0000","last_change":"2024-04-25T05:34:00.694199+0000","last_active":"2024-04-25T05:34:57.150018+0000","last_peered":"2024-04-25T05:34:57.150018+0000","last_clean":"2024-04-25T05:34:57.150018+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:34:57.150018+0000","last_undegraded":"2024-04-25T05:34:57.150018+0000","last_fullsized":"2024-04-25T05:34:57.150018+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.968271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165588+0000","last_change":"2024-04-25T05:34:02.702812+0000","last_active":"2024-04-25T05:34:56.165588+0000","last_peered":"2024-04-25T05:34:56.165588+0000","last_clean":"2024-04-25T05:34:56.165588+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:34:56.165588+0000","last_undegraded":"2024-04-25T05:34:56.165588+0000","last_fullsized":"2024-04-25T05:34:56.165588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.890640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450722+0000","last_change":"2024-04-25T05:34:04.722965+0000","last_active":"2024-04-25T05:34:57.450722+0000","last_peered":"2024-04-25T05:34:57.450722+0000","last_clean":"2024-04-25T05:34:57.450722+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:34:57.450722+0000","last_undegraded":"2024-04-25T05:34:57.450722+0000","last_fullsized":"2024-04-25T05:34:57.450722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":298,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450212+0000","last_change":"2024-04-25T05:34:20.908770+0000","last_active":"2024-04-25T05:34:57.450212+0000","last_peered":"2024-04-25T05:34:57.450212+0000","last_clean":"2024-04-25T05:34:57.450212+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:34:57.450212+0000","last_undegraded":"2024-04-25T05:34:57.450212+0000","last_fullsized":"2024-04-25T05:34:57.450212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-25T05:34:20.908611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:34:20.908611+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.546403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164954+0000","last_change":"2024-04-25T05:33:58.679230+0000","last_active":"2024-04-25T05:34:56.164954+0000","last_peered":"2024-04-25T05:34:56.164954+0000","last_clean":"2024-04-25T05:34:56.164954+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:34:56.164954+0000","last_undegraded":"2024-04-25T05:34:56.164954+0000","last_fullsized":"2024-04-25T05:34:56.164954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.518886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412600+0000","last_change":"2024-04-25T05:34:01.002201+0000","last_active":"2024-04-25T05:34:56.412600+0000","last_peered":"2024-04-25T05:34:56.412600+0000","last_clean":"2024-04-25T05:34:56.412600+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:34:56.412600+0000","last_undegraded":"2024-04-25T05:34:56.412600+0000","last_fullsized":"2024-04-25T05:34:56.412600+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.821400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449936+0000","last_change":"2024-04-25T05:34:02.701047+0000","last_active":"2024-04-25T05:34:57.449936+0000","last_peered":"2024-04-25T05:34:57.449936+0000","last_clean":"2024-04-25T05:34:57.449936+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:34:57.449936+0000","last_undegraded":"2024-04-25T05:34:57.449936+0000","last_fullsized":"2024-04-25T05:34:57.449936+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.440947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.149254+0000","last_change":"2024-04-25T05:34:04.726534+0000","last_active":"2024-04-25T05:34:57.149254+0000","last_peered":"2024-04-25T05:34:57.149254+0000","last_clean":"2024-04-25T05:34:57.149254+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:34:57.149254+0000","last_undegraded":"2024-04-25T05:34:57.149254+0000","last_fullsized":"2024-04-25T05:34:57.149254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.888044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202234+0000","last_change":"2024-04-25T05:33:58.682492+0000","last_active":"2024-04-25T05:34:57.202234+0000","last_peered":"2024-04-25T05:34:57.202234+0000","last_clean":"2024-04-25T05:34:57.202234+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:34:57.202234+0000","last_undegraded":"2024-04-25T05:34:57.202234+0000","last_fullsized":"2024-04-25T05:34:57.202234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.900372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":111,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450104+0000","last_change":"2024-04-25T05:34:01.000301+0000","last_active":"2024-04-25T05:34:57.450104+0000","last_peered":"2024-04-25T05:34:57.450104+0000","last_clean":"2024-04-25T05:34:57.450104+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:34:57.450104+0000","last_undegraded":"2024-04-25T05:34:57.450104+0000","last_fullsized":"2024-04-25T05:34:57.450104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.243918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201985+0000","last_change":"2024-04-25T05:34:00.694619+0000","last_active":"2024-04-25T05:34:57.201985+0000","last_peered":"2024-04-25T05:34:57.201985+0000","last_clean":"2024-04-25T05:34:57.201985+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:34:57.201985+0000","last_undegraded":"2024-04-25T05:34:57.201985+0000","last_fullsized":"2024-04-25T05:34:57.201985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:01.840274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.451127+0000","last_change":"2024-04-25T05:34:02.709328+0000","last_active":"2024-04-25T05:34:57.451127+0000","last_peered":"2024-04-25T05:34:57.451127+0000","last_clean":"2024-04-25T05:34:57.451127+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:34:57.451127+0000","last_undegraded":"2024-04-25T05:34:57.451127+0000","last_fullsized":"2024-04-25T05:34:57.451127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.085576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148612+0000","last_change":"2024-04-25T05:34:04.759135+0000","last_active":"2024-04-25T05:34:57.148612+0000","last_peered":"2024-04-25T05:34:57.148612+0000","last_clean":"2024-04-25T05:34:57.148612+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:34:57.148612+0000","last_undegraded":"2024-04-25T05:34:57.148612+0000","last_fullsized":"2024-04-25T05:34:57.148612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150090+0000","last_change":"2024-04-25T05:33:58.677172+0000","last_active":"2024-04-25T05:34:57.150090+0000","last_peered":"2024-04-25T05:34:57.150090+0000","last_clean":"2024-04-25T05:34:57.150090+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:34:57.150090+0000","last_undegraded":"2024-04-25T05:34:57.150090+0000","last_fullsized":"2024-04-25T05:34:57.150090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:03.326701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202454+0000","last_change":"2024-04-25T05:34:00.999939+0000","last_active":"2024-04-25T05:34:57.202454+0000","last_peered":"2024-04-25T05:34:57.202454+0000","last_clean":"2024-04-25T05:34:57.202454+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:34:57.202454+0000","last_undegraded":"2024-04-25T05:34:57.202454+0000","last_fullsized":"2024-04-25T05:34:57.202454+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:12.684394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412458+0000","last_change":"2024-04-25T05:34:02.708443+0000","last_active":"2024-04-25T05:34:56.412458+0000","last_peered":"2024-04-25T05:34:56.412458+0000","last_clean":"2024-04-25T05:34:56.412458+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:34:56.412458+0000","last_undegraded":"2024-04-25T05:34:56.412458+0000","last_fullsized":"2024-04-25T05:34:56.412458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.195589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.201940+0000","last_change":"2024-04-25T05:34:04.728674+0000","last_active":"2024-04-25T05:34:57.201940+0000","last_peered":"2024-04-25T05:34:57.201940+0000","last_clean":"2024-04-25T05:34:57.201940+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:34:57.201940+0000","last_undegraded":"2024-04-25T05:34:57.201940+0000","last_fullsized":"2024-04-25T05:34:57.201940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.707833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450455+0000","last_change":"2024-04-25T05:33:58.678744+0000","last_active":"2024-04-25T05:34:57.450455+0000","last_peered":"2024-04-25T05:34:57.450455+0000","last_clean":"2024-04-25T05:34:57.450455+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:34:57.450455+0000","last_undegraded":"2024-04-25T05:34:57.450455+0000","last_fullsized":"2024-04-25T05:34:57.450455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.579714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148413+0000","last_change":"2024-04-25T05:34:00.688978+0000","last_active":"2024-04-25T05:34:57.148413+0000","last_peered":"2024-04-25T05:34:57.148413+0000","last_clean":"2024-04-25T05:34:57.148413+0000","last_became_active":"2024-04-25T05:34:00.688453+0000","last_became_peered":"2024-04-25T05:34:00.688453+0000","last_unstale":"2024-04-25T05:34:57.148413+0000","last_undegraded":"2024-04-25T05:34:57.148413+0000","last_fullsized":"2024-04-25T05:34:57.148413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.622999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202680+0000","last_change":"2024-04-25T05:34:02.708185+0000","last_active":"2024-04-25T05:34:57.202680+0000","last_peered":"2024-04-25T05:34:57.202680+0000","last_clean":"2024-04-25T05:34:57.202680+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:34:57.202680+0000","last_undegraded":"2024-04-25T05:34:57.202680+0000","last_fullsized":"2024-04-25T05:34:57.202680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.047597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.449373+0000","last_change":"2024-04-25T05:34:04.761218+0000","last_active":"2024-04-25T05:34:57.449373+0000","last_peered":"2024-04-25T05:34:57.449373+0000","last_clean":"2024-04-25T05:34:57.449373+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:34:57.449373+0000","last_undegraded":"2024-04-25T05:34:57.449373+0000","last_fullsized":"2024-04-25T05:34:57.449373+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.789438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.202392+0000","last_change":"2024-04-25T05:33:58.682409+0000","last_active":"2024-04-25T05:34:57.202392+0000","last_peered":"2024-04-25T05:34:57.202392+0000","last_clean":"2024-04-25T05:34:57.202392+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:34:57.202392+0000","last_undegraded":"2024-04-25T05:34:57.202392+0000","last_fullsized":"2024-04-25T05:34:57.202392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164311+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:34:56.164311+0000","last_peered":"2024-04-25T05:34:56.164311+0000","last_clean":"2024-04-25T05:34:56.164311+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:34:56.164311+0000","last_undegraded":"2024-04-25T05:34:56.164311+0000","last_fullsized":"2024-04-25T05:34:56.164311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.826192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164597+0000","last_change":"2024-04-25T05:34:02.701774+0000","last_active":"2024-04-25T05:34:56.164597+0000","last_peered":"2024-04-25T05:34:56.164597+0000","last_clean":"2024-04-25T05:34:56.164597+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:34:56.164597+0000","last_undegraded":"2024-04-25T05:34:56.164597+0000","last_fullsized":"2024-04-25T05:34:56.164597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:08.422624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164139+0000","last_change":"2024-04-25T05:34:04.718464+0000","last_active":"2024-04-25T05:34:56.164139+0000","last_peered":"2024-04-25T05:34:56.164139+0000","last_clean":"2024-04-25T05:34:56.164139+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:34:56.164139+0000","last_undegraded":"2024-04-25T05:34:56.164139+0000","last_fullsized":"2024-04-25T05:34:56.164139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.166256+0000","last_change":"2024-04-25T05:33:58.676147+0000","last_active":"2024-04-25T05:34:56.166256+0000","last_peered":"2024-04-25T05:34:56.166256+0000","last_clean":"2024-04-25T05:34:56.166256+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:34:56.166256+0000","last_undegraded":"2024-04-25T05:34:56.166256+0000","last_fullsized":"2024-04-25T05:34:56.166256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:04.509783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":99,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164458+0000","last_change":"2024-04-25T05:34:00.999824+0000","last_active":"2024-04-25T05:34:56.164458+0000","last_peered":"2024-04-25T05:34:56.164458+0000","last_clean":"2024-04-25T05:34:56.164458+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:34:56.164458+0000","last_undegraded":"2024-04-25T05:34:56.164458+0000","last_fullsized":"2024-04-25T05:34:56.164458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:05.441893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164046+0000","last_change":"2024-04-25T05:34:02.699472+0000","last_active":"2024-04-25T05:34:56.164046+0000","last_peered":"2024-04-25T05:34:56.164046+0000","last_clean":"2024-04-25T05:34:56.164046+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:34:56.164046+0000","last_undegraded":"2024-04-25T05:34:56.164046+0000","last_fullsized":"2024-04-25T05:34:56.164046+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.651777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.450121+0000","last_change":"2024-04-25T05:34:04.724152+0000","last_active":"2024-04-25T05:34:57.450121+0000","last_peered":"2024-04-25T05:34:57.450121+0000","last_clean":"2024-04-25T05:34:57.450121+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:34:57.450121+0000","last_undegraded":"2024-04-25T05:34:57.450121+0000","last_fullsized":"2024-04-25T05:34:57.450121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.599745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165221+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:34:56.165221+0000","last_peered":"2024-04-25T05:34:56.165221+0000","last_clean":"2024-04-25T05:34:56.165221+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:34:56.165221+0000","last_undegraded":"2024-04-25T05:34:56.165221+0000","last_fullsized":"2024-04-25T05:34:56.165221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.390939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164334+0000","last_change":"2024-04-25T05:33:58.681579+0000","last_active":"2024-04-25T05:34:56.164334+0000","last_peered":"2024-04-25T05:34:56.164334+0000","last_clean":"2024-04-25T05:34:56.164334+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:34:56.164334+0000","last_undegraded":"2024-04-25T05:34:56.164334+0000","last_fullsized":"2024-04-25T05:34:56.164334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.706716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.148511+0000","last_change":"2024-04-25T05:34:00.691497+0000","last_active":"2024-04-25T05:34:57.148511+0000","last_peered":"2024-04-25T05:34:57.148511+0000","last_clean":"2024-04-25T05:34:57.148511+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:34:57.148511+0000","last_undegraded":"2024-04-25T05:34:57.148511+0000","last_fullsized":"2024-04-25T05:34:57.148511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.049287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165051+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:34:56.165051+0000","last_peered":"2024-04-25T05:34:56.165051+0000","last_clean":"2024-04-25T05:34:56.165051+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:34:56.165051+0000","last_undegraded":"2024-04-25T05:34:56.165051+0000","last_fullsized":"2024-04-25T05:34:56.165051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:31.872714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.451031+0000","last_change":"2024-04-25T05:34:02.699262+0000","last_active":"2024-04-25T05:34:57.451031+0000","last_peered":"2024-04-25T05:34:57.451031+0000","last_clean":"2024-04-25T05:34:57.451031+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:34:57.451031+0000","last_undegraded":"2024-04-25T05:34:57.451031+0000","last_fullsized":"2024-04-25T05:34:57.451031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:07.861356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164103+0000","last_change":"2024-04-25T05:33:58.675050+0000","last_active":"2024-04-25T05:34:56.164103+0000","last_peered":"2024-04-25T05:34:56.164103+0000","last_clean":"2024-04-25T05:34:56.164103+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:34:56.164103+0000","last_undegraded":"2024-04-25T05:34:56.164103+0000","last_fullsized":"2024-04-25T05:34:56.164103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:09.277179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165094+0000","last_change":"2024-04-25T05:34:00.692677+0000","last_active":"2024-04-25T05:34:56.165094+0000","last_peered":"2024-04-25T05:34:56.165094+0000","last_clean":"2024-04-25T05:34:56.165094+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:34:56.165094+0000","last_undegraded":"2024-04-25T05:34:56.165094+0000","last_fullsized":"2024-04-25T05:34:56.165094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:13.912050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165793+0000","last_change":"2024-04-25T05:34:04.723087+0000","last_active":"2024-04-25T05:34:56.165793+0000","last_peered":"2024-04-25T05:34:56.165793+0000","last_clean":"2024-04-25T05:34:56.165793+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:34:56.165793+0000","last_undegraded":"2024-04-25T05:34:56.165793+0000","last_fullsized":"2024-04-25T05:34:56.165793+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.795993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:57.150189+0000","last_change":"2024-04-25T05:34:02.708656+0000","last_active":"2024-04-25T05:34:57.150189+0000","last_peered":"2024-04-25T05:34:57.150189+0000","last_clean":"2024-04-25T05:34:57.150189+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:34:57.150189+0000","last_undegraded":"2024-04-25T05:34:57.150189+0000","last_fullsized":"2024-04-25T05:34:57.150189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:06.620659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.412253+0000","last_change":"2024-04-25T05:33:58.684035+0000","last_active":"2024-04-25T05:34:56.412253+0000","last_peered":"2024-04-25T05:34:56.412253+0000","last_clean":"2024-04-25T05:34:56.412253+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:34:56.412253+0000","last_undegraded":"2024-04-25T05:34:56.412253+0000","last_fullsized":"2024-04-25T05:34:56.412253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:34:58.247558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164959+0000","last_change":"2024-04-25T05:34:00.692475+0000","last_active":"2024-04-25T05:34:56.164959+0000","last_peered":"2024-04-25T05:34:56.164959+0000","last_clean":"2024-04-25T05:34:56.164959+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:34:56.164959+0000","last_undegraded":"2024-04-25T05:34:56.164959+0000","last_fullsized":"2024-04-25T05:34:56.164959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:10.725418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165261+0000","last_change":"2024-04-25T05:34:04.719644+0000","last_active":"2024-04-25T05:34:56.165261+0000","last_peered":"2024-04-25T05:34:56.165261+0000","last_clean":"2024-04-25T05:34:56.165261+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:34:56.165261+0000","last_undegraded":"2024-04-25T05:34:56.165261+0000","last_fullsized":"2024-04-25T05:34:56.165261+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165498+0000","last_change":"2024-04-25T05:34:02.705065+0000","last_active":"2024-04-25T05:34:56.165498+0000","last_peered":"2024-04-25T05:34:56.165498+0000","last_clean":"2024-04-25T05:34:56.165498+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:34:56.165498+0000","last_undegraded":"2024-04-25T05:34:56.165498+0000","last_fullsized":"2024-04-25T05:34:56.165498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.937446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.165317+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:34:56.165317+0000","last_peered":"2024-04-25T05:34:56.165317+0000","last_clean":"2024-04-25T05:34:56.165317+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:34:56.165317+0000","last_undegraded":"2024-04-25T05:34:56.165317+0000","last_fullsized":"2024-04-25T05:34:56.165317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.422993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T05:34:56.164684+0000","last_change":"2024-04-25T05:34:00.686532+0000","last_active":"2024-04-25T05:34:56.164684+0000","last_peered":"2024-04-25T05:34:56.164684+0000","last_clean":"2024-04-25T05:34:56.164684+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:34:56.164684+0000","last_undegraded":"2024-04-25T05:34:56.164684+0000","last_fullsized":"2024-04-25T05:34:56.164684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.818346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070481,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28328,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716824,"statfs":{"total":95995035648,"available":95966027776,"internally_reserved":0,"allocated":1339392,"data_stored":909804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364821,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1327104,"data_stored":915128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":778240,"data_stored":318694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28320,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716832,"statfs":{"total":95995035648,"available":95966035968,"internally_reserved":0,"allocated":1331200,"data_stored":908071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":724992,"data_stored":323492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":761856,"data_stored":324222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":761856,"data_stored":324942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691866,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":317704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:35:01.618 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T05:35:01.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump 2024-04-25T05:35:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:02.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[37492]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:02.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:02.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[27010]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:02.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:02 smithi060 ceph-mon[30898]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 85 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-04-25T05:28:34.192515+0000 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:modified 2024-04-25T05:34:56.157708+0000 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:crush_version 18 2024-04-25T05:35:02.590 INFO:teuthology.orchestra.run.smithi043.stdout:full_ratio 0.95 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:backfillfull_ratio 0.9 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:nearfull_ratio 0.85 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:require_min_compat_client luminous 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:min_compat_client jewel 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:require_osd_release squid 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:stretch_mode_enabled false 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:max_osd 8 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6801/3557191248 v2:172.21.15.43:6802/3557191248 exists,up ef370ce4-85d8-41ed-9655-2e2eaab3b48e 2024-04-25T05:35:02.591 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6805/235229295 v2:172.21.15.43:6806/235229295 exists,up ed504d08-dc59-4584-bd5f-8059e02a4161 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6809/3416612512 v2:172.21.15.43:6810/3416612512 exists,up 11ff9a46-8686-4b36-a530-5cc1dca1f088 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6813/3391766005 v2:172.21.15.43:6814/3391766005 exists,up f3314c45-11c4-46ee-be0b-6f639faa7da2 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6800/1310286745 v2:172.21.15.60:6801/1310286745 exists,up 8c5337b2-94d1-4784-9321-db771de75f4c 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6804/3586186657 v2:172.21.15.60:6805/3586186657 exists,up fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6808/1675137598 v2:172.21.15.60:6809/1675137598 exists,up 5b946188-0dd1-4e68-bcfc-a3daa342edaa 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6812/3982012700 v2:172.21.15.60:6813/3982012700 exists,up dec84bb8-9d2b-469f-9641-62ff40b9b9b2 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3423182021 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2082036420 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3958950586 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3953560108 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/4077364595 expires 2024-04-26T05:29:30.781325+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/830746728 expires 2024-04-26T05:29:30.781325+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/786479021 expires 2024-04-26T05:29:02.328794+0000 2024-04-25T05:35:02.592 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/3735093998 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1835545567 expires 2024-04-26T05:29:02.328794+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:6800/149142905 expires 2024-04-26T05:34:31.636703+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1408327650 expires 2024-04-26T05:29:02.328794+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/2239747746 expires 2024-04-26T05:29:02.328794+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/4084484879 expires 2024-04-26T05:29:30.781325+0000 2024-04-25T05:35:02.593 INFO:teuthology.orchestra.run.smithi043.stdout:blocklist 172.21.15.43:0/1247294293 expires 2024-04-26T05:29:30.781325+0000 2024-04-25T05:35:02.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:03.274 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6801/3557191248 v2:172.21.15.43:6802/3557191248 exists,up ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6805/235229295 v2:172.21.15.43:6806/235229295 exists,up ed504d08-dc59-4584-bd5f-8059e02a4161', 'osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6809/3416612512 v2:172.21.15.43:6810/3416612512 exists,up 11ff9a46-8686-4b36-a530-5cc1dca1f088', 'osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.43:6813/3391766005 v2:172.21.15.43:6814/3391766005 exists,up f3314c45-11c4-46ee-be0b-6f639faa7da2', 'osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6800/1310286745 v2:172.21.15.60:6801/1310286745 exists,up 8c5337b2-94d1-4784-9321-db771de75f4c', 'osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6804/3586186657 v2:172.21.15.60:6805/3586186657 exists,up fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6808/1675137598 v2:172.21.15.60:6809/1675137598 exists,up 5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.60:6812/3982012700 v2:172.21.15.60:6813/3982012700 exists,up dec84bb8-9d2b-469f-9641-62ff40b9b9b2'] 2024-04-25T05:35:03.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon mon.a config show 2024-04-25T05:35:03.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.a deep-scrub starts 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.a deep-scrub ok 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2399820481' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:35:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:03 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:35:03.523 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:35:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:35:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:35:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.a deep-scrub starts 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.a deep-scrub ok 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2399820481' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.a deep-scrub starts 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.a deep-scrub ok 2024-04-25T05:35:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2399820481' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T05:35:03.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:35:03.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:35:03.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:35:03.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:03 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:35:04.052 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:35:04.052 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "mon.a", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster": "ceph", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "admin_socket_mode": "", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_client_required": "cephx, none", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_cluster_required": "cephx", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_debug": "true", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_service_required": "cephx", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "auth_supported": "", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio": "true", 2024-04-25T05:35:04.053 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_poll_ms": "250", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_aio_reap_max": "16", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_async_discard_threads": "0", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_block_size": "4096", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio": "false", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_debug_inflight_ios": "false", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_enable_discard": "false", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_flock_retry": "3", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_inject_crash": "0", 2024-04-25T05:35:04.054 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring_hipri": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bdev_type": "", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_alloc_size": "1048576", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_allocator": "hybrid", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_buffered_io": "true", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_for_zeros": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_compact_log_sync": "false", 2024-04-25T05:35:04.055 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_max_log_runway": "4194304", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_max_prefetch": "1048576", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_min_flush_size": "524288", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_min_log_runway": "1048576", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_replay_recovery": "false", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluefs_sync_write": "false", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_allocation_from_file": "true", 2024-04-25T05:35:04.056 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_allocator": "hybrid", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_create": "true", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_create": "false", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_path": "", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_db_size": "0", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_path": "", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_preallocate_file": "false", 2024-04-25T05:35:04.057 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_size": "107374182400", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_create": "false", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_path": "", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_block_wal_size": "100663296", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs": "true", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T05:35:04.058 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_autotune": "true", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size": "0", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cache_type": "2q", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_clone_cow": "true", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-25T05:35:04.059 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_mode": "none", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_csum_type": "crc32c", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_freelist": "false", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-25T05:35:04.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_small_allocations": "0", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_default_buffered_read": "true", 2024-04-25T05:35:04.061 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_default_buffered_write": "false", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-25T05:35:04.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fail_eio": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mount": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_umount": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-25T05:35:04.063 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_ignore_data_csum": "false", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_log_op_age": "5.000000", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_alloc_size": "0", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size": "0", 2024-04-25T05:35:04.064 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_max_deferred_txc": "32", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size": "0", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_nid_prealloc": "1024", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_retry_disk_reads": "3", 2024-04-25T05:35:04.065 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_cf": "true", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_spdk_mem": "512", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-25T05:35:04.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_tracing": "false", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-25T05:35:04.067 INFO:teuthology.orchestra.run.smithi043.stdout: "bluestore_zero_block_detection": "false", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-25T05:35:04.068 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_cluster_require_signatures": "false", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_cluster_require_version": "2", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_require_signatures": "false", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_require_version": "2", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_service_require_signatures": "false", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_service_require_version": "2", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "cephx_sign_messages": "true", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "chdir": "", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_acl_type": "", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_asio_thread_count": "2", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_cache_mid": "0.750000", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_cache_size": "16384", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_caps_release_delay": "5", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_check_pool_perm": "true", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_force_sync_read": "false", 2024-04-25T05:35:04.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_getattr_caps": "false", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_debug_inject_tick_delay": "0", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_die_on_failed_remount": "false", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_dirsize_rbytes": "true", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_force_lazyio": "false", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_fs": "", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_inject_release_failure": "false", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_max_inline_size": "4096", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mds_namespace": "", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_metadata": "", 2024-04-25T05:35:04.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_gid": "-1", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_timeout": "300", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mount_uid": "-1", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_mountpoint": "/", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_notify_timeout": "10", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc": "true", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_dirty": "104857600", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_max_objects": "1000", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_size": "209715200", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_oc_target_dirty": "8388608", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_permissions": "true", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_quota": "true", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_quota_df": "true", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_max_bytes": "0", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_max_periods": "4", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_readahead_min": "131072", 2024-04-25T05:35:04.071 INFO:teuthology.orchestra.run.smithi043.stdout: "client_reconnect_stale": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_shutdown_timeout": "30", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_snapdir": ".snap", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_tick_interval": "1", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_trace": "", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_try_dentry_invalidate": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_use_faked_inos": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "client_use_random_mds": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_graylog_port": "12201", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_monitors": "default=true", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog": "false", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "clog_to_syslog_level": "info", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "-", 2024-04-25T05:35:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_network": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_network_interface": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_isal": "false", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_level": "5", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zlib_winsize": "-15", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "compressor_zstd_level": "1", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_alien_op_num_threads": "6", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_osd_stat_interval": "0", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crimson_seastar_cpu_cores": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location_hook": "", 2024-04-25T05:35:04.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location_hook_timeout": "10", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "daemonize": "false", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "daos_pool": "tank", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_alienstore": "0/5", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_allow_any_pool_priority": "false", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asok": "1/5", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asok_assert_abort": "false", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_asserts_on_shutdown": "true", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_auth": "1/5", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bdev": "1/3", 2024-04-25T05:35:04.074 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bluefs": "1/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_bluestore": "1/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_buffer": "0/1", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_ceph_dedup": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_ceph_exporter": "1/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cephfs_mirror": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cephsqlite": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_client": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_compressor": "1/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_context": "0/1", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crimson_interrupt": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crush": "1/1", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_crypto": "1/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_cyanstore": "0/5", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_deliberately_leak_memory": "false", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_disable_randomized_ping": "false", 2024-04-25T05:35:04.075 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_dpdk": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_eventtrace": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_filer": "0/1", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_filestore": "1/3", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_finisher": "1/1", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_fuse": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_heartbeat_testing_span": "0", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_heartbeatmap": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_javaclient": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_journal": "1/3", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_journaler": "0/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_kstore": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_lockdep": "0/1", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mclock": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds": "1/5", 2024-04-25T05:35:04.076 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_balancer": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_locker": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_log": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_log_expire": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_migrator": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mds_quiesce": "3/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_memstore": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mgr": "2/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mgrc": "1/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_mon": "20/20", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_monc": "0/10", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_ms": "1/1", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_none": "0/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objclass": "0/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objectcacher": "0/5", 2024-04-25T05:35:04.077 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_objecter": "0/1", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_optracker": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_osd": "1/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_paxos": "20/20", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_perfcounter": "1/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_prioritycache": "1/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rados": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_mirror": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_pwl": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rbd_replay": "0/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_refs": "0/0", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_reserver": "1/1", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw": "1/5", 2024-04-25T05:35:04.078 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_access": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_datacache": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_dbstore": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_flight": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_notification": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rgw_sync": "1/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_rocksdb": "4/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_backref": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_cache": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_cleaner": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_device": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_epm": "0/5", 2024-04-25T05:35:04.079 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-25T05:35:04.080 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_journal": "0/5", 2024-04-25T05:35:04.080 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_lba": "0/5", 2024-04-25T05:35:04.080 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_odata": "0/5", 2024-04-25T05:35:04.080 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_omap": "0/5", 2024-04-25T05:35:04.080 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_onode": "0/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_t": "0/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_seastore_tm": "0/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_striper": "0/1", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_test": "0/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_throttle": "1/1", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_timer": "0/1", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_tp": "0/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "debug_trace": "1/5", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "device_failure_prediction_mode": "none", 2024-04-25T05:35:04.081 INFO:teuthology.orchestra.run.smithi043.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_graylog": "false", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_journald": "false", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_stderr": "true", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "err_to_syslog": "false", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event_tracing": "false", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_addr": "0.0.0.0", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_http_port": "9926", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_prio_limit": "5", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_sort_metrics": "true", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "exporter_stats_period": "5", 2024-04-25T05:35:04.082 INFO:teuthology.orchestra.run.smithi043.stdout: "fake_statfs_for_testing": "0", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "fatal_signal_handlers": "true", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filer_max_purge_ops": "10", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filer_max_truncate_ops": "128", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_apply_finisher_threads": "1", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_blackhole": "false", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_btrfs_clone_range": "true", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_btrfs_snap": "true", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_caller_concurrency": "10", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_collect_device_partition_information": "true", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_commit_timeout": "600.000000", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_inject_read_err": "false", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_omap_check": "false", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_debug_verify_split": "false", 2024-04-25T05:35:04.083 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_dump_file": "", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fadvise": "true", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fail_eio": "true", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fd_cache_shards": "16", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fd_cache_size": "128", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fiemap": "false", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fiemap_threshold": "4096", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_inject_stall": "0", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_parallel": "false", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_trailing": "false", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_journal_writeahead": "false", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_kill_at": "0", 2024-04-25T05:35:04.084 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs": "0", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size": "0", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_merge_threshold": "-10", 2024-04-25T05:35:04.085 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_odsync_write": "false", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_backend": "rocksdb", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_backend_path": "", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_thread_timeout": "60", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_op_threads": "2", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_punch_hole": "false", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-25T05:35:04.086 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_queue_max_ops": "50", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_seek_data_hole": "false", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_sloppy_crc": "false", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_splice": "false", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_split_multiple": "2", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_split_rand_factor": "20", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_update_to": "1000", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-25T05:35:04.087 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_enable": "true", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_xfs_extsize": "false", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "filestore_zfs_snap": "false", 2024-04-25T05:35:04.088 INFO:teuthology.orchestra.run.smithi043.stdout: "fio_dir": "/tmp/fio", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_allow_other": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_atomic_o_trunc": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_big_writes": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_debug": "false", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_default_permissions": "false", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_disable_pagecache": "false", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_max_write": "0", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_multithreaded": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_require_active_mds": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_set_user_groups": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_move": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_read": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_splice_write": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-25T05:35:04.089 INFO:teuthology.orchestra.run.smithi043.stdout: "fuse_use_invalidate_cb": "true", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "gss_target_name": "ceph", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_file": "", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_inject_failure": "0", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_interval": "5", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "host": "smithi043", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-25T05:35:04.090 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "inject_early_sigterm": "false", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "jaeger_agent_port": "6799", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "jaeger_tracing_enable": "false", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_aio": "true", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_align_min_size": "65536", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_block_align": "true", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_block_size": "4096", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_dio": "true", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_discard": "false", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_force_aio": "false", 2024-04-25T05:35:04.091 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_ignore_corruption": "false", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_max_write_bytes": "10485760", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_max_write_entries": "100", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_replay_from": "0", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_write_header_frequency": "0", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journal_zero_on_create": "false", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_prefetch_periods": "10", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_prezero_periods": "5", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "journaler_write_head_interval": "15", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "keyfile": "", 2024-04-25T05:35:04.092 INFO:teuthology.orchestra.run.smithi043.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_backend": "rocksdb", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_default_stripe_size": "65536", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_fsck_on_mount": "false", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_max_bytes": "67108864", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_max_ops": "512", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_nid_prealloc": "1024", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_onode_map_size": "1024", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_sync_submit_transaction": "false", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "kstore_sync_transaction": "false", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "librados_thread_count": "2", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "lockdep": "false", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "lockdep_force_backtrace": "false", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "log_coarse_timestamps": "true", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-25T05:35:04.093 INFO:teuthology.orchestra.run.smithi043.stdout: "log_flush_on_exit": "false", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_graylog_host": "127.0.0.1", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_graylog_port": "12201", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_max_new": "1000", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_max_recent": "10000", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_stderr_prefix": "", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_stop_at_utilization": "0.970000", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_file": "true", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_graylog": "false", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_journald": "false", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_stderr": "false", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "log_to_syslog": "false", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "max_rotating_auth_attempts": "10", 2024-04-25T05:35:04.094 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_action_on_write_error": "1", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_alternate_name_max": "8192", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_asio_thread_count": "2", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_export_pin": "true", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_dirs": "true", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_interval": "5", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_interval": "10", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_max": "-1", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_max_until": "-1", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_merge_size": "50", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_midchunk": "0.300000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_min_start": "0.200000", 2024-04-25T05:35:04.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_minchunk": "0.001000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_mode": "0", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_need_max": "1.200000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_need_min": "0.800000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_overload_epochs": "2", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_bits": "3", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_size": "10000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_target_decay": "10.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_grace": "15.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_interval": "4.000000", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-25T05:35:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_mid": "0.700000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_delay": "0", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_release_free_interval": "10", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_reservation": "0.050000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_interval": "1", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cache_trim_threshold": "262144", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_prealloc_inos": "1000", 2024-04-25T05:35:04.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_connect_bootstrapping": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_damage_table_max_entries": "10000", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_auth_pins": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_frag": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_scatterstat": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_debug_subtrees": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_decay_halflife": "5.000000", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_default_dir_hash": "2", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_defer_session_stale": "true", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_deny_all_reconnect": "false", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_keys_per_op": "16384", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_max_commit_size": "10", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_max_entries": "0", 2024-04-25T05:35:04.098 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dir_prefetch": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_on_map": "false", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_early_reply": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_enable_op_tracker": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_enforce_unique_name": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_random": "true", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-25T05:35:04.099 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_health_summarize_threshold": "10", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_health_dummy": "false", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_migrator_session_race": "false", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_join_fs": "", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_journal_format": "1", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_create_at": "0", 2024-04-25T05:35:04.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_export_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_import_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_expire_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_journal_replay_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_link_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_mdstable_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_openc_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_rename_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_shutdown_at": "0", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_event_large_threshold": "524288", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_events_per_segment": "1024", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_max_events": "-1", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_max_segments": "128", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_pause": "false", 2024-04-25T05:35:04.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_segment_size": "0", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_threshold": "128", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_log_warn_factor": "2.000000", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_caps_per_client": "1048576", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_completed_flushes": "100000", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_completed_requests": "100000", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_export_size": "20971520", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_file_recover": "32", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_files": "64", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_ops": "8192", 2024-04-25T05:35:04.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_max_snaps_per_dir": "100", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_metrics_update_interval": "2", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_min_caps_per_client": "100", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_min_caps_working_set": "10000", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_numa_node": "-1", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_complaint_time": "30.000000", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_duration": "600", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_size": "20", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_slow_op_size": "20", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_op_log_threshold": "5", 2024-04-25T05:35:04.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_ping_grace": "15", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_ping_interval": "5", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_caps": "30000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_recall_warning_threshold": "262144", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_replay_interval": "1.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_root_ino_gid": "0", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_root_ino_uid": "0", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-25T05:35:04.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_shutdown_check": "0", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_skip_ino": "0", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_max_uid": "4294967294", 2024-04-25T05:35:04.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_min_uid": "0", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_snap_rstat": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_standby_replay_damaged": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_symlink_recovery": "true", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_thrash_exports": "0", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_thrash_fragments": "0", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_tick_interval": "5.000000", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_valgrind_exit": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_verify_backtrace": "1", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_verify_scatter": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mds_wipe_sessions": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mempool_debug": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-25T05:35:04.106 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_device_bytes": "1073741824", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_page_set": "false", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "memstore_page_size": "65536", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_bytes": "134217728", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_messages": "512", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_disabled_modules": "", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_enable_op_tracker": "true", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_max_pg_creating": "1024", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_max_pg_num_change": "128", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mds_bytes": "134217728", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mds_messages": "128", 2024-04-25T05:35:04.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mon_bytes": "134217728", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_mon_messages": "128", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_num_op_tracker_shard": "32", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_complaint_time": "30.000000", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_history_duration": "600", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_history_size": "20", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_history_slow_op_size": "20", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_op_log_threshold": "5", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_osd_bytes": "536870912", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_osd_messages": "8192", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_pool": "true", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_standby_modules": "true", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_stats_period": "5", 2024-04-25T05:35:04.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_stats_threshold": "5", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_tick_period": "2", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_allow_pool_delete": "true", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_allow_pool_size_one": "false", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_auth_validate_all_caps": "true", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_bytes": "104857600", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_directed_command_retry": "2", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_hunt_parallel": "3", 2024-04-25T05:35:04.109 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_log_interval": "1.000000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_ping_interval": "10.000000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_client_target_rank": "-1", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_level": "debug", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_file": "true", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-25T05:35:04.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_journald": "true", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_bootstrap": "false", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_start": "false", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_compact_on_trim": "true", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_cpu_threads": "4", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_crush_min_required_version": "hammer", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_daemon_bytes": "419430400", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_avail_crit": "5", 2024-04-25T05:35:04.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_avail_warn": "5", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_data_size_warn": "16106127360", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_json": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_dump_transactions": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_extra_checks": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_reef": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_no_require_squid": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_down_mkfs_grace": "60", 2024-04-25T05:35:04.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_down_uptime_grace": "60", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_election_default_strategy": "1", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_election_timeout": "5.000000", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ping_divisor": "2", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_enable_op_tracker": "true", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_fake_pool_delete": "false", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_force_quorum_join": "false", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_globalid_prealloc": "10000", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_detail_to_clog": "true", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_log_update_period": "5", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_max_detail": "50", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog": "true", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog_interval": "600", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-25T05:35:04.113 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_host": "[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0]", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_host_override": "", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_initial_members": "", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease": "5.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_full_interval": "50", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_max": "10000", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_log_max_summary": "50", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-25T05:35:04.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_log_epochs": "500", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_osd": "10000", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_pg_per_osd": "10000", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_pool_pg_num": "65536", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_force_trim_to": "0", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mds_skip_sanity": "false", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_memory_autotune": "true", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_memory_target": "2147483648", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_beacon_grace": "30", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_digest_period": "5", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_inactive_grace": "60", 2024-04-25T05:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_min_osdmap_epochs": "50", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_complaint_time": "30", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_duration": "600", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_size": "20", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_slow_op_size": "20", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_op_log_threshold": "5", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_in": "false", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-25T05:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_cache_size": "500", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_down_out_interval": "600", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_force_trim_to": "0", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_down_reporters": "2", 2024-04-25T05:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_report_timeout": "900", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-25T05:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_stuck_threshold": "60", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_probe_timeout": "2.000000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_max_change": "0.050000", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_max_osds": "4", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-25T05:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_interval": "86400", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_max_keys": "100", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_scrub_timeout": "300", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_session_timeout": "300", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_smart_report_timeout": "5", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stat_smooth_intervals": "6", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_pool_min_size": "2", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_pool_size": "4", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-25T05:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_debug": "false", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_provider_kill_at": "0", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_requester_kill_at": "0", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_sync_timeout": "60.000000", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_target_pg_per_osd": "100", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_tick_interval": "5", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_timecheck_interval": "300.000000", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_use_min_delay_socket": "false", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_older_version_delay": "604800", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-25T05:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_misplaced": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_older_version": "true", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-25T05:35:04.122 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": "", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "motr_tracing_enabled": "false", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_op_threads": "3", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_cm": "false", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_device_name": "", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_dscp": "96", 2024-04-25T05:35:04.123 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_local_gid": "", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_port_num": "1", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_sl": "3", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_support_srq": "true", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_rdma_type": "ib", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_async_reap_threshold": "5", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_before_connect": "false", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_ipv4": "true", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_ipv6": "false", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_msgr1": "false", 2024-04-25T05:35:04.124 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_msgr2": "true", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_port_max": "7568", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_port_min": "6800", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_retry_count": "3", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_bind_retry_delay": "5", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_client": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mds": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mgr": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_mon": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_blackhole_osd": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_client_mode": "crc secure", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_cluster_mode": "crc secure", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_cluster_type": "", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_compress_secure": "false", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_connection_idle_timeout": "900", 2024-04-25T05:35:04.125 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_connection_ready_timeout": "10", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_crc_data": "true", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_crc_header": "true", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_bad_msg": "false", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_bug": "true", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_old_message": "true", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_skipped_message": "false", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_coremask": "0xF", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_enable_tso": "true", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-25T05:35:04.126 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hugepages": "", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_lro": "true", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_memory_channel": "4", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_pmd": "", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_port_id": "0", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_dump_on_send": "false", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_initial_backoff": "0.200000", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_max": "1.000000", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_delay_type": "", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-25T05:35:04.127 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_network_congestion": "0", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_inject_socket_failures": "0", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_learn_addr_from_peer": "true", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_max_accept_failures": "4", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_max_backoff": "15.000000", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_client_mode": "secure crc", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_mon_service_mode": "secure crc", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compress_min_size": "1024", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compress_mode": "none", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_pq_min_cost": "65536", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_public_type": "", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_service_mode": "crc secure", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_listen_backlog": "512", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_nodelay": "true", 2024-04-25T05:35:04.128 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_tcp_rcvbuf": "0", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "ms_type": "async+posix", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "no_config_file": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_completion_locks_per_session": "32", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inflight_ops": "1024", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_tick_interval": "5.000000", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objecter_timeout": "10.000000", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objectstore_blackhole": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "openssl_engine_opts": "", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_delay_time": "5.000000", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_hist_halflife": "1000", 2024-04-25T05:35:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_max_low_ops": "2", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_max_ops": "4", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_agent_slop": "0.020000", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_asio_thread_count": "2", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_async_recovery_min_cost": "100", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_scan_max": "512", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backfill_scan_min": "64", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_degraded": "false", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_peering": "false", 2024-04-25T05:35:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_backoff_on_unfound": "true", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_beacon_report_interval": "300", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_duration": "30", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_max_block_size": "67108864", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_blkin_trace_all": "false", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_check_for_log_corruption": "false", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-25T05:35:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_class_update_on_start": "true", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_message_cap": "256", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_message_size_cap": "524288000", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_op_priority": "63", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_client_watch_timeout": "30", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_command_max_records": "256", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_compact_on_start": "false", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_update_on_start": "true", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_crush_update_weight_set": "true", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-25T05:35:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_feed_pullee": "-1", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_misdirected_ops": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_no_acting_change": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_no_purge_strays": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_op_order": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-25T05:35:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_shutdown": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_trim_objects": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_snaps": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_keys": "1024", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-25T05:35:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_stride": "524288", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_default_notify_timeout": "30", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep": "0.000000", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_discard_disconnected_ops": "true", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_enable_op_tracker": "true", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-25T05:35:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_info": "true", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown": "true", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_function_tracing": "false", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_grace": "20", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_interval": "6", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_peers": "10", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_min_size": "2000", 2024-04-25T05:35:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_stale": "600", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_max_size": "100000", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_min_size": "1000", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_journal_size": "5120", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_kill_backfill_at": "0", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_cache_size": "50", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_dedup": "true", 2024-04-25T05:35:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_message_max": "40", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_map_share_max_epochs": "40", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_attr_name_len": "100", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_attr_size": "0", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_backfills": "1", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_markdown_count": "5", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_markdown_period": "600", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_name_len": "2048", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_namespace_len": "256", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_object_size": "134217728", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_blocked_by": "16", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_log_entries": "10000", 2024-04-25T05:35:04.138 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_pgls": "1024", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_push_cost": "8388608", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_push_objects": "10", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_scrubs": "3", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_trimming_pgs": "2", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_write_op_reply_len": "64", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_max_write_size": "90", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-25T05:35:04.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_profile": "balanced", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_base": "805306368", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_cache_min": "134217728", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-25T05:35:04.140 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target": "4294967296", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target_autotune": "false", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_min_pg_log_entries": "250", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_min_recovery_priority": "0", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_report_interval": "5", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_num_cache_shards": "32", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_num_op_tracker_shard": "32", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_auto_affinity": "true", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_node": "-1", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_numa_prefer_iface": "true", 2024-04-25T05:35:04.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objecter_finishers": "1", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore": "bluestore", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore_fuse": "false", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_objectstore_tracing": "false", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_complaint_time": "30.000000", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_duration": "600", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_size": "20", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_slow_op_size": "20", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_log_threshold": "5", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards": "0", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards_hdd": "5", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_shards_ssd": "8", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-25T05:35:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_pq_min_cost": "65536", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_queue_cut_off": "high", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_op_thread_timeout": "15", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_open_classes_on_start": "true", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_os_flags": "0", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_peering_op_priority": "255", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_delete_cost": "1048576", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_delete_priority": "5", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-25T05:35:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_trim_max": "10000", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_log_trim_min": "100", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_crimson": "false", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-25T05:35:04.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-25T05:35:04.145 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-25T05:35:04.145 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-25T05:35:04.145 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-25T05:35:04.145 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_flags": "0", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_min_size": "0", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pg_num": "32", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_pgp_num": "0", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_read_ratio": "70", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_size": "3", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_default_type": "replicated", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_push_per_object_cost": "1000", 2024-04-25T05:35:04.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recover_clone_overlap": "true", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_cost": "20971520", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active": "0", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_max_single_start": "1", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_op_priority": "3", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_priority": "5", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-25T05:35:04.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep": "0.000000", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_repair_during_recovery": "false", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_requested_scrub_priority": "120", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_auto_repair": "false", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_begin_hour": "0", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_begin_week_day": "0", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_chunk_max": "25", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_chunk_min": "5", 2024-04-25T05:35:04.148 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_cost": "52428800", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_during_recovery": "false", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_end_hour": "0", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_end_week_day": "0", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_event_cost": "4096", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_invalid_stats": "true", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_max_preemptions": "5", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_priority": "5", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_scrub_sleep": "0.000000", 2024-04-25T05:35:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_skip_data_digest": "false", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_smart_report_timeout": "5", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_cost": "1048576", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_priority": "5", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-25T05:35:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_target_transaction_size": "30", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_tracing": "false", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_use_stale_snap": "false", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "osdc_blkin_trace_all": "false", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_kill_at": "0", 2024-04-25T05:35:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_max_join_drift": "10", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_min": "500", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_min_wait": "0.050000", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_propose_interval": "1.000000", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_max": "500", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_service_trim_min": "10", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_stash_full_interval": "25", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_trim_max": "500", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "paxos_trim_min": "250", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "perf": "true", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "pid_file": "", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "-", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrv": "", 2024-04-25T05:35:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_bind_addr": "-", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "public_network": "", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "public_network_interface": "", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "qat_compressor_enabled": "false", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "qat_compressor_session_max_number": "256", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_mon_op_timeout": "0", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_osd_op_timeout": "0", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rados_tracing": "false", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_atime_update_interval": "60", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_balance_parent_reads": "false", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_balance_snap_reads": "false", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blkin_trace_all": "false", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-25T05:35:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache": "true", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_policy": "writearound", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_size": "33554432", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_clone_copy_on_read": "false", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_compression_hint": "none", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_concurrent_management_ops": "10", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_clone_format": "auto", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_data_pool": "", 2024-04-25T05:35:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_features": "61", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_format": "2", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_map_options": "", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_order": "22", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_pool": "rbd", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_stripe_count": "0", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_default_stripe_unit": "0", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_enable_alloc_hint": "true", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_io_scheduler": "simple", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-25T05:35:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_order": "24", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_pool": "", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_journal_splay_width": "4", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_localize_parent_reads": "false", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_localize_snap_reads": "false", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-25T05:35:04.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-25T05:35:04.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-25T05:35:04.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_mtime_update_interval": "60", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_non_blocking_aio": "true", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_op_thread_timeout": "60", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_op_threads": "1", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_parent_cache_enabled": "false", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_plugins": "", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_burst": "0", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_bps_limit": "0", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_exclude_ops": "0", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_burst": "0", 2024-04-25T05:35:04.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_iops_limit": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-25T05:35:04.160 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_read_from_replica_policy": "default", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_skip_partial_discard": "true", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_tracing": "false", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_validate_names": "true", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd_validate_pool": "true", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "restapi_base_url": "", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "restapi_log_level": "", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-04-25T05:35:04.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_account_default_quota_max_size": "-1", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_acl_grants_max_num": "100", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_admin_entry": "admin", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_backend_store": "rados", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_barbican_url": "", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_beast_enable_async": "true", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_counters_cache": "false", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-25T05:35:04.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_enabled": "true", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_expiry_interval": "900", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cache_lru_size": "10000", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_config_store": "rados", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_content_length_compat": "false", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_copy_obj_progress": "true", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cors_rules_max_num": "100", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_cross_domain_policy": "", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_addr": "", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-25T05:35:04.163 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_password": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_kmip_username": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_require_ssl": "true", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-25T05:35:04.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_addr": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_auth": "token", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_namespace": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_prefix": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_token_file": "", 2024-04-25T05:35:04.165 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_buffersize": "524288", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_low_speed_time": "300", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-25T05:35:04.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_d4n_port": "6379", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_changes_size": "1000", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_num_shards": "128", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-25T05:35:04.167 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_log_window": "30", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-25T05:35:04.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dns_name": "", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dns_s3website_name": "", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_dynamic_resharding": "true", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-25T05:35:04.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_gc_threads": "true", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_lc_threads": "true", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_ops_log": "false", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_quota_threads": "true", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_static_website": "false", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enable_usage_log": "false", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_enforce_swift_acls": "true", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_exit_timeout_secs": "120", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_expose_bucket": "false", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_extended_http_attrs": "", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_filter": "none", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_frontends": "beast port=7480", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_deferred": "50", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-25T05:35:04.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_objs": "32", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_gc_processor_period": "3600", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_init_timeout": "300", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_connection_idle": "300", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-25T05:35:04.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_domain": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_password": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_password_path": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_project": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_tenant": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_token": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_token_path": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_admin_user": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_api_version": "2", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_domain": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_password": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_project": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_barbican_user": "", 2024-04-25T05:35:04.172 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_url": "", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_debug_interval": "-1", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_lock_max_time": "90", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_objs": "32", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_rules": "1000", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_worker": "3", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lc_thread_delay": "0", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_dnattr": "uid", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-25T05:35:04.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_searchfilter": "", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ldap_uri": "", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_http_headers": "", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_log_object_name_utc": "false", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attr_name_len": "0", 2024-04-25T05:35:04.174 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attr_size": "0", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_chunk_size": "4194304", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_listing_results": "1000", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_notify_retries": "10", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_put_param_size": "1048576", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_put_size": "5368709120", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_max_slo_entries": "1000", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_md_log_max_shards": "64", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-25T05:35:04.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_mp_lock_max_time": "600", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_max_gc": "300", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-25T05:35:04.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_num_async_rados_threads": "32", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_num_control_oids": "8", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_numa_node": "-1", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_chunk_size": "100", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_gc_interval": "600", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-25T05:35:04.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_thread_timeout": "600", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_op_tracing": "false", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_token": "", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_url": "", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_opa_verify_ssl": "true", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_rados": "false", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_ops_log_socket_path": "", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_push_interval": "2.000000", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-25T05:35:04.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_lanes": "3", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_cache_partitions": "3", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_print_continue": "true", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rados_tracing": "false", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm": "", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm_id": "", 2024-04-25T05:35:04.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_region": "", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_request_uri": "", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_batch_size": "64", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_max_aio": "128", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_num_logs": "16", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_reshard_thread_interval": "600", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_resolve_cname": "false", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_run_sync_thread": "true", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-25T05:35:04.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_scheduler_type": "throttler", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_script_uri": "", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_service_provider_name": "", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_client_id": "", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_client_secret": "", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_entry": "sts", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_key": "sts", 2024-04-25T05:35:04.181 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_min_session_duration": "900", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sts_token_introspection_url": "", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_account_in_url": "false", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_auth_entry": "auth", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_auth_url": "", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_custom_header": "", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_need_stats": "true", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_tenant_name": "", 2024-04-25T05:35:04.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_token_expiration": "86400", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_url": "", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_url_prefix": "swift", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_lease_period": "120", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-25T05:35:04.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_thread_pool_size": "512", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_comment": "", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_createby": "", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_encoding": "", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_flag": "false", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_origin": "", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_torrent_tracker": "", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_trust_forwarded_https": "false", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_max_shards": "32", 2024-04-25T05:35:04.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_usage_max_user_shards": "1", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_use_opa_authz": "false", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_counters_cache": "false", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_max_buckets": "1000", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_policies_max_num": "100", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_user_unique_email": "true", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_verify_ssl": "true", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone": "", 2024-04-25T05:35:04.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone_id": "", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup": "", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup_id": "", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgwlc_auto_session_clear": "true", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_block_size": "4096", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_size": "536870912", 2024-04-25T05:35:04.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_index_type": "binary_search", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_partition_filters": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_perf": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "run_dir": "/var/run/ceph", 2024-04-25T05:35:04.187 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_block_create": "true", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_cache_lru_size": "67108864", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_default_max_object_size": "16777216", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_device_size": "53687091200", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_capacity": "16", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_main_device_type": "SSD", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-25T05:35:04.188 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "seastore_segment_size": "67108864", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "set_keepcaps": "false", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "setgroup": "ceph", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "setuser": "ceph", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "setuser_match_path": "", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "thp": "false", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "threadpool_default_timeout": "60", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout: "throttler_perf_counter": "true" 2024-04-25T05:35:04.189 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:35:04.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:04.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-25T05:35:05.146 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:05.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:35:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:05 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:35:05.515 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:35:05.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:05.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:05.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:35:05.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:05 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:35:05.631 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a: {} 2024-04-25T05:35:05.642 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b: {} 2024-04-25T05:35:05.653 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c: {} 2024-04-25T05:35:05.654 INFO:teuthology.orchestra.run.smithi043.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:35:05.654 INFO:teuthology.orchestra.run.smithi043.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:35:05.654 INFO:teuthology.orchestra.run.smithi043.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:35:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:35:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:35:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: osdmap e86: 8 total, 8 up, 8 in 2024-04-25T05:35:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3491922450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:35:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: osdmap e86: 8 total, 8 up, 8 in 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3491922450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:35:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:06 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:35:06.479 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-25T05:35:06.489 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-25T05:35:06.490 INFO:tasks.radosbench:Beginning radosbench... 2024-04-25T05:35:06.491 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-25T05:35:06.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-04-25T05:35:06.491 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-25T05:35:06.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:35:06.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:35:06.492 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-25T05:35:06.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:06.492 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-25T05:35:06.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:35:06.493 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-25T05:35:06.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:35:06.493 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-25T05:35:06.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:35:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:35:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:35:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: osdmap e86: 8 total, 8 up, 8 in 2024-04-25T05:35:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3491922450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:35:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:35:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:06 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:35:06.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:06.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:06.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:06.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:06.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:06.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:06.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:06.798 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:06.800 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:06.801 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:06.803 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:06.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:06.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:06.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:06.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:07.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:07.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:07.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:07.260 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:07.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:07.383 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:35:07.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.1c deep-scrub starts 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[27010]: 3.1c deep-scrub ok 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.1c deep-scrub starts 2024-04-25T05:35:07.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:07 smithi043 ceph-mon[37492]: 3.1c deep-scrub ok 2024-04-25T05:35:07.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:35:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:35:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:35:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:35:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.1c deep-scrub starts 2024-04-25T05:35:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:07 smithi060 ceph-mon[30898]: 3.1c deep-scrub ok 2024-04-25T05:35:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.506 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:07.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:07.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:07.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:07.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:07.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:07.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:07.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:07.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:07.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:08.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:08.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:08.219 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:35:08.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:08.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:08.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:35:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:35:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1221872625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1221872625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1221872625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:08.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:08.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:08.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:08.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:08.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:35:08.703 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:35:08.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:08.724 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:35:08.742 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:08.746 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:35:08.767 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:35:08.787 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:35:08.809 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:35:08.829 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:35:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:35:08.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:08.934 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:35:08.935 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:35:08.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:08.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:08.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:09.089 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:09.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:09.359 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_0' created 2024-04-25T05:35:09.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:35:09.121Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001405138s 2024-04-25T05:35:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:35:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:35:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: osdmap e87: 8 total, 8 up, 8 in 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: 4.17 scrub starts 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: 4.17 scrub ok 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3429882869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: osdmap e87: 8 total, 8 up, 8 in 2024-04-25T05:35:09.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: 4.17 scrub starts 2024-04-25T05:35:09.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: 4.17 scrub ok 2024-04-25T05:35:09.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-25T05:35:09.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3429882869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:09.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:09.486 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:35:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:35:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: osdmap e87: 8 total, 8 up, 8 in 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: 4.17 scrub starts 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: 4.17 scrub ok 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-25T05:35:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3429882869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:09.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:09.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:09.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:09.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:09.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.857 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:09.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:09.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:09.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:09.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:10.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:10.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:10.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:10.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:10.221 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 systemd[1]: Starting Ceph grafana.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:35:10.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:10.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:10.349 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:10.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:10.472 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:10.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-25T05:35:10.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-mon[30898]: osdmap e88: 8 total, 8 up, 8 in 2024-04-25T05:35:10.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-mon[30898]: osdmap e89: 8 total, 8 up, 8 in 2024-04-25T05:35:10.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 podman[61056]: 2024-04-25 05:35:10.212003275 +0000 UTC m=+0.019349135 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-04-25T05:35:10.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 podman[61056]: 2024-04-25 05:35:10.311737473 +0000 UTC m=+0.119083376 container create 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:35:10.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:10.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:10.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-25T05:35:10.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[37492]: osdmap e88: 8 total, 8 up, 8 in 2024-04-25T05:35:10.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[37492]: osdmap e89: 8 total, 8 up, 8 in 2024-04-25T05:35:10.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1578359146' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-25T05:35:10.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[27010]: osdmap e88: 8 total, 8 up, 8 in 2024-04-25T05:35:10.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:10 smithi043 ceph-mon[27010]: osdmap e89: 8 total, 8 up, 8 in 2024-04-25T05:35:10.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:10.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:10.786 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 podman[61056]: 2024-04-25 05:35:10.62285185 +0000 UTC m=+0.430197698 container init 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:35:10.787 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 podman[61056]: 2024-04-25 05:35:10.625905524 +0000 UTC m=+0.433251372 container start 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:35:10.787 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 bash[61056]: 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 2024-04-25T05:35:10.787 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 systemd[1]: Started Ceph grafana.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:35:10.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:10.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:10.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:10.968 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:10.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:11.037 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787434455Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787619821Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787682901Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787693235Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.78769838Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787705048Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787709589Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787713605Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787717719Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787722299Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787727378Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787731704Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-25T05:35:11.038 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787736398Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787741141Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787745889Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787750509Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787755294Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787759598Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=settings t=2024-04-25T05:35:10.787764681Z level=info msg="App mode production" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=sqlstore t=2024-04-25T05:35:10.787823898Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=sqlstore t=2024-04-25T05:35:10.787843112Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.788739759Z level=info msg="Starting DB migrations" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.78880758Z level=info msg="Executing migration" id="create migration_log table" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.78933874Z level=info msg="Executing migration" id="create user table" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.789745857Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.790213707Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.790617007Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-25T05:35:11.039 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.791194543Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.791602282Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.792359373Z level=info msg="Executing migration" id="create user table v2" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.792777369Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.793177478Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.793608041Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.793912876Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.794291414Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.79471145Z level=info msg="Executing migration" id="Update user table charset" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.794951972Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-25T05:35:11.040 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.795366255Z level=info msg="Executing migration" id="Add missing user data" 2024-04-25T05:35:11.041 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.795735016Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-25T05:35:11.041 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.796166496Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.796528194Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.796935695Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.79832682Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.798752286Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.79923761Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.79964649Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.800056698Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.800471693Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.8007267Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.801136698Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.801506316Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.801851774Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-25T05:35:11.042 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.802245168Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.803042985Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.803418918Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.803780503Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.804177046Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.80455707Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.804966415Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.805294165Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.805599117Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.805902428Z level=info msg="Executing migration" id="create star table" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.806256396Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.806681069Z level=info msg="Executing migration" id="create org table v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.807078985Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.807479667Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.807842088Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.808242266Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-25T05:35:11.043 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.808638331Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.80902817Z level=info msg="Executing migration" id="Update org table charset" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.809304Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.809550888Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.809817235Z level=info msg="Executing migration" id="create dashboard table" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.810185393Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.810576942Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.810967082Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.811364962Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.811814195Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.812232064Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.813318388Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.81369671Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.814113842Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-25T05:35:11.044 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.814519175Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.81481191Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.815286015Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.815563516Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.816077931Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.816567455Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.817028154Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.817437754Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.817921618Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.818343714Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.818749316Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.819003031Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.819280527Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.819785358Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.820327494Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-25T05:35:11.045 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.820826084Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.821331757Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.821637044Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.822041549Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.822524086Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.822759157Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.823177684Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.823531036Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.824576842Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.824948297Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.825382183Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.825786809Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.826084487Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.826449226Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-25T05:35:11.046 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.826909671Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.827299838Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.827642717Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.827900954Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.828292009Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.828816501Z level=info msg="Executing migration" id="create data_source table" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.829230163Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.829624646Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.830007882Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.830375492Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.830714322Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.831904691Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.832300386Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.832652149Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.833064267Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-25T05:35:11.047 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.833441007Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.833928472Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.834487529Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.834720077Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.834970086Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.83551797Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.835788036Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.836088672Z level=info msg="Executing migration" id="Add uid column" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.836620062Z level=info msg="Executing migration" id="Update uid value" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.83688999Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.837306689Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.837723429Z level=info msg="Executing migration" id="create api_key table" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.838210913Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.838612361Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.839047426Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.839493605Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-25T05:35:11.048 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.839825101Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.840209866Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.840582324Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.841903565Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.842285318Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.842744917Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.843168796Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.843563468Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.843829748Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.844192474Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.844453836Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.844965086Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.845520151Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.845789239Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.846377013Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-25T05:35:11.049 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.846959238Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.847350766Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.847668174Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.848064602Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.848594291Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.849010905Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.849425412Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.849681011Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.849917845Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.850522938Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.851121827Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.851425544Z level=info msg="Executing migration" id="create quota table v1" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.851801838Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-25T05:35:11.050 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.852238687Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.852532506Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.85290563Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.853319076Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.853911389Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.854195138Z level=info msg="Executing migration" id="create session table" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.85460679Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.854864888Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.85520168Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.8555717Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.855931162Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.856214501Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.856471604Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.856725079Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.856988694Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.857445753Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-25T05:35:11.051 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.857696717Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-25T05:35:11.052 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.858363382Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-25T05:35:11.052 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.858671662Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-25T05:35:11.052 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.859339056Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-25T05:35:11.052 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.859969524Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-25T05:35:11.052 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.860265241Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-25T05:35:11.054 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.860685868Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-25T05:35:11.054 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.8611511Z level=info msg="Executing migration" id="create alert table v1" 2024-04-25T05:35:11.054 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.861595445Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-25T05:35:11.054 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.86205967Z level=info msg="Executing migration" id="add index alert state" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.862494389Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.86292306Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.863287895Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.863729362Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.864160914Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.865874858Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.866259651Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.866686851Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.866965049Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.867337458Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.867683694Z level=info msg="Executing migration" id="Add column is_default" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.86839286Z level=info msg="Executing migration" id="Add column frequency" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.869110995Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.869796067Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-25T05:35:11.055 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.870495156Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.870901479Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.871182728Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.871477883Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.871847939Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.872281528Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.872723924Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.873157439Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.873600449Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.874378542Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.87510875Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-25T05:35:11.056 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.875416584Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.875861867Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.87631017Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.877038626Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.877329122Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.877727162Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.878153441Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.878465898Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.878909221Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.879367194Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.879807498Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.880266809Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.880738018Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.881196763Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-25T05:35:11.057 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.881486637Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.882250023Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.882721299Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.883471451Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.883896399Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.884368636Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.88483239Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.886837091Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.887221181Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.887680564Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.887964773Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.888378583Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.888646557Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.889471485Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.890203919Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-25T05:35:11.058 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.890641166Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.891073137Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.891384206Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.892158329Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.892654117Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.89292772Z level=info msg="Executing migration" id="Move region to single row" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.893296917Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.893714036Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.894172496Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.894625122Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.895040656Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-25T05:35:11.059 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.895508955Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.895897062Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.896200573Z level=info msg="Executing migration" id="create test_data table" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.896617169Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.89703279Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.897484785Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.897920191Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.89822221Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.898545409Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.898807925Z level=info msg="Executing migration" id="create team table" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.899213333Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.899678476Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.900136235Z level=info msg="Executing migration" id="create team member table" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.900537396Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-25T05:35:11.060 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.900988762Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.901443005Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.901877871Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.902719632Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.90356071Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.904408483Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.904800351Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.905279257Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.905717275Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.906170298Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.906669375Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.90711806Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.90756838Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.908035717Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-25T05:35:11.061 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.908471811Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.908756181Z level=info msg="Executing migration" id="create tag table" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.909151983Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.909615633Z level=info msg="Executing migration" id="create login attempt table" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.91003489Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.910488436Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.910898618Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.913289912Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.91366652Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.914106584Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.914418641Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.914752497Z level=info msg="Executing migration" id="create user auth table" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.915167647Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.915626314Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.915880332Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-25T05:35:11.062 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.916722025Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.917584465Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.918454811Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.919377044Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.919812315Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92069377Z level=info msg="Executing migration" id="create server_lock table" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.921097847Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92156304Z level=info msg="Executing migration" id="create user auth token table" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.922011068Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.922473731Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.922924582Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.923381826Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92430053Z level=info msg="Executing migration" id="create cache_data table" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.924711866Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-25T05:35:11.063 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92517366Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.925607729Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.926056254Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92638666Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.926670321Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.927143194Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.92761042Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.928064513Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.928385287Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.928813991Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.929247775Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.929695637Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.93015359Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.931038928Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.931482338Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-25T05:35:11.064 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.931753174Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.932193183Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.932661398Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.933110411Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.933421563Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.933849127Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.934265518Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.934739581Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.935187958Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.936092172Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.936534403Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.936926436Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.941445621Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.945894568Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.946348389Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-25T05:35:11.065 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.946991599Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.947963662Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.948400849Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.948859012Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.949304606Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.949801964Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.950107528Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.951073484Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.951966267Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.952860984Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.953294223Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.953744544Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.954663851Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.955586632Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-25T05:35:11.066 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.956043098Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.956953566Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.957892614Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.958201834Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.958644352Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.959075674Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.95951733Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.959767551Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.960739554Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.961674902Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.962599988Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-25T05:35:11.067 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.963544394Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.964470498Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.964757075Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.965193224Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.966112314Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.966401132Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.967385913Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.967827021Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.968762374Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.969155259Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.969585695Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.97055474Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.970942453Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.971409265Z level=info msg="Executing migration" id="create alert_image table" 2024-04-25T05:35:11.068 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.971790617Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.972259463Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.97264919Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.973060129Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.973500035Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.973586095Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.973814841Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.974175379Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.974607044Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.97474611Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.9748589Z level=info msg="alerts found to migrate" alerts=0 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.97524282Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.975677128Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.976201394Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.976594334Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-25T05:35:11.069 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.977038925Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.977515748Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.977765021Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.978143753Z level=info msg="Executing migration" id="create data_keys table" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.978551299Z level=info msg="Executing migration" id="create secrets table" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.978985465Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.984610908Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.985684227Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.985992542Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.99168872Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.997370085Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.997762411Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.998223101Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.9985586Z level=info msg="Executing migration" id="create permission table" 2024-04-25T05:35:11.070 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.998929922Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.999399433Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:10 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:10.999854681Z level=info msg="Executing migration" id="create role table" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.000252134Z level=info msg="Executing migration" id="add column display_name" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.001341798Z level=info msg="Executing migration" id="add column group_name" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.002404897Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.002850526Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.003304341Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.003758776Z level=info msg="Executing migration" id="create team role table" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.004154963Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.004604148Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.00512046Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.005582505Z level=info msg="Executing migration" id="create user role table" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.005958882Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.006422409Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-25T05:35:11.071 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.006876083Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.007333507Z level=info msg="Executing migration" id="create builtin role table" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.007754605Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.008206389Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.008656803Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.009782051Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.010277681Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.010745097Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.011197805Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.01163817Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.012022487Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.012484068Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.013613413Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-25T05:35:11.072 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.014013032Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.014486144Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.014734104Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.015166043Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.015626251Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.015972443Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.016303492Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.016619441Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.017006106Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.017476612Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.018615192Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.01890669Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.019347927Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-25T05:35:11.073 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.019798222Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.020254658Z level=info msg="Executing migration" id="add correlation config column" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.021393483Z level=info msg="Executing migration" id="create entity_events table" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.021824309Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.022255543Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.022347513Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.022619019Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.022710297Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.022936768Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.02340736Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.023776372Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.024255855Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-25T05:35:11.074 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.024711698Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.025146559Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.025591146Z level=info msg="Executing migration" id="Drop public config table" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.025954429Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.026399504Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.02680545Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.0273151Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.027754342Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.031584625Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.032747272Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.033881697Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.034198012Z level=info msg="Executing migration" id="add share column" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.035362265Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.035790115Z level=info msg="Executing migration" id="create file table" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.036220704Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.036658717Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-25T05:35:11.075 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.037176055Z level=info msg="Executing migration" id="create file_meta table" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.037563485Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.038007077Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.038309963Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.038641523Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.038919179Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.039369717Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.040565657Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.040867378Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.041316535Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.041698421Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.041989099Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-25T05:35:11.076 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.042359521Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-25T05:35:11.077 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.043572291Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-25T05:35:11.077 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.044783131Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-25T05:35:11.077 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.045221948Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-25T05:35:11.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:11.098 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:11.144 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-25T05:35:11.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:11.220 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:11.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:11.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: osdmap e90: 8 total, 8 up, 8 in 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:35:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: osdmap e90: 8 total, 8 up, 8 in 2024-04-25T05:35:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:11.465 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:11.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:35:11.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:35:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: osdmap e90: 8 total, 8 up, 8 in 2024-04-25T05:35:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.058408706Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.058861556Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.059316537Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.063881284Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.064212865Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.064551832Z level=info msg="Executing migration" id="create folder table" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.064915143Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-25T05:35:11.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.065355839Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.065820914Z level=info msg="Executing migration" id="Update folder title length" 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.066105484Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=migrator t=2024-04-25T05:35:11.066516097Z level=info msg="migrations completed" performed=485 skipped=0 duration=277.729221ms 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=sqlstore t=2024-04-25T05:35:11.066911504Z level=info msg="Created default organization" 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=plugin.finder t=2024-04-25T05:35:11.101953356Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=secrets t=2024-04-25T05:35:11.102071557Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=query_data t=2024-04-25T05:35:11.106033089Z level=info msg="Query Service initialization" 2024-04-25T05:35:11.500 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=live.push_http t=2024-04-25T05:35:11.108520422Z level=info msg="Live Push Gateway initialization" 2024-04-25T05:35:11.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:11.591 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:11.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:11.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:35:11.714 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:11.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:35:11.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:11.850 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:11.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:11.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:11.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:11.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=infra.usagestats.collector t=2024-04-25T05:35:11.74406403Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=provisioning.datasources t=2024-04-25T05:35:11.745017785Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=provisioning.datasources t=2024-04-25T05:35:11.756397441Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=provisioning.alerting t=2024-04-25T05:35:11.767397063Z level=info msg="starting to provision alerting" 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=provisioning.alerting t=2024-04-25T05:35:11.767407203Z level=info msg="finished to provision alerting" 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=grafanaStorageLogger t=2024-04-25T05:35:11.76781577Z level=info msg="storage starting" 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=ngalert.state.manager t=2024-04-25T05:35:11.767860274Z level=info msg="Warming state cache for startup" 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=http.server t=2024-04-25T05:35:11.774680533Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=ngalert.state.manager t=2024-04-25T05:35:11.78073486Z level=info msg="State cache has been initialized" states=0 duration=12.873786ms 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=ticker t=2024-04-25T05:35:11.780793539Z level=info msg=starting first_tick=2024-04-25T05:35:20Z 2024-04-25T05:35:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:35:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=ngalert.multiorg.alertmanager t=2024-04-25T05:35:11.780805283Z level=info msg="starting MultiOrg Alertmanager" 2024-04-25T05:35:12.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:12.097 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:12.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:12.223 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:12.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:12 smithi043 ceph-mon[27010]: Deploying daemon node-exporter.a on smithi043 2024-04-25T05:35:12.255 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:12.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:12 smithi043 ceph-mon[37492]: Deploying daemon node-exporter.a on smithi043 2024-04-25T05:35:12.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:12.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:12.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:12.358 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:12 smithi060 ceph-mon[30898]: Deploying daemon node-exporter.a on smithi043 2024-04-25T05:35:12.358 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:12.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:12.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:12.505 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:12 smithi043 systemd[1]: Starting Ceph node-exporter.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:35:12.560 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:12.560 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:12.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:12.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:12.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:35:12.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:12.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:12.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:12.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:12.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:12.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:12.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:12.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:12 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:13.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:13.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:13.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:13.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-25T05:35:13.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:13.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:13.365 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:35:13.365 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:35:13.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[27010]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:35:13.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3982421239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[37492]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:35:13.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3982421239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:13.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:13 smithi060 ceph-mon[30898]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T05:35:13.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3982421239' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:35:13.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:13.595 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:13.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:13.718 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:13.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:13.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:13.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:13.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:14.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:14.090 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:14.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:14.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:14.242 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:35:14.314 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:14.336 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:14.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:35:14.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:14.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: osdmap e91: 8 total, 8 up, 8 in 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:14.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: osdmap e91: 8 total, 8 up, 8 in 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: osdmap e91: 8 total, 8 up, 8 in 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:35:14.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:14.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:14.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:14.659 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:14.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:14.713 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:14.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:14.839 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:14.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:14.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:14.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:15.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.089 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:15.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:15.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:15.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:15.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:15.339 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[37492]: osdmap e92: 8 total, 8 up, 8 in 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[37492]: osdmap e93: 8 total, 8 up, 8 in 2024-04-25T05:35:15.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:35:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[27010]: osdmap e92: 8 total, 8 up, 8 in 2024-04-25T05:35:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:15 smithi043 ceph-mon[27010]: osdmap e93: 8 total, 8 up, 8 in 2024-04-25T05:35:15.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:15.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:15 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:35:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:15 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1638597475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:15 smithi060 ceph-mon[30898]: osdmap e92: 8 total, 8 up, 8 in 2024-04-25T05:35:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:15 smithi060 ceph-mon[30898]: osdmap e93: 8 total, 8 up, 8 in 2024-04-25T05:35:15.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:15.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:15.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:15.712 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:15.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:15.840 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:15.886 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:15.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:15.967 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:15.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:16.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:16.132 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:35:16.132 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:15 smithi043 bash[74159]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-25T05:35:16.154 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:35:16.173 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:35:16.192 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:35:16.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:16.213 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:35:16.225 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:16.233 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:35:16.254 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:35:16.275 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:35:16.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:35:16.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:16.352 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:16.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:16 smithi043 ceph-mon[27010]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:35:16.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1304056085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:16.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:16 smithi043 ceph-mon[37492]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:35:16.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1304056085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:16.391 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:16 smithi043 bash[74159]: Getting image source signatures 2024-04-25T05:35:16.391 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:16 smithi043 bash[74159]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-25T05:35:16.392 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:16 smithi043 bash[74159]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-25T05:35:16.392 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:16 smithi043 bash[74159]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-25T05:35:16.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:16.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:16.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:16 smithi060 ceph-mon[30898]: pgmap v40: 148 pgs: 148 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:35:16.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1304056085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:16.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:16.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:16.618 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-25T05:35:16.619 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-25T05:35:16.619 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-25T05:35:16.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:35:16.621 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-25T05:35:16.621 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-25T05:35:16.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:16.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-04-25T05:35:16.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-25T05:35:16.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_74356 2024-04-25T05:35:16.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:35:16.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-04-25T05:35:16.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:16.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:16.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:16.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:16.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.021 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:17.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:17.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:17.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:35:17.148 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:17.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:17.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:17.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:17.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:17.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:17.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:17.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:17.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:17.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 2934 2918 182.365 182.375 0.00196797 0.00546499 2024-04-25T05:35:17.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:17.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:17.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:17.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:17.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:17.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:17.902 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:17.902 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:269 7.9 7:96349df5:::benchmark_data_smithi043_74356_object268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:16.814523+0000", 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.0824597570000001, 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0097822870000000006, 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 269 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814523+0000", 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814523+0000", 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814524+0000", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.077e-06 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814547+0000", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3421000000000002e-05 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814564+0000", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6124999999999999e-05 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.814572+0000", 2024-04-25T05:35:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5960000000000004e-06 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.815116+0000", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000544062 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.815267+0000", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015049099999999999 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.815515+0000", 2024-04-25T05:35:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024809399999999997 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.816741+0000", 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001225706 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.818056+0000", 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001315203 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.823903+0000", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058470579999999996 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.824259+0000", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00035646099999999997 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:16.824305+0000", 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5992999999999998e-05 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:1268 7.4 7:26da3b84:::benchmark_data_smithi043_74356_object1267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.124815+0000", 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.77216744400000004, 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010510195999999999, 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 1268 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124815+0000", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124815+0000", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124816+0000", 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.85e-07 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124862+0000", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5368000000000003e-05 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124882+0000", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0919000000000001e-05 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124891+0000", 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9889999999999998e-06 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126022+0000", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001130797 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126130+0000", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010773900000000001 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126272+0000", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141691 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.128098+0000", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018265180000000001 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.128111+0000", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3132999999999999e-05 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135220+0000", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071091269999999998 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135283+0000", 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3042999999999999e-05 2024-04-25T05:35:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135325+0000", 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1987000000000003e-05 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:1270 7.4 7:2179abd6:::benchmark_data_smithi043_74356_object1269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.124904+0000", 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.77207911500000004, 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010543547, 2024-04-25T05:35:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 1270 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124904+0000", 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124904+0000", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124904+0000", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5899999999999996e-07 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124930+0000", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5854999999999999e-05 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124973+0000", 2024-04-25T05:35:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2865999999999999e-05 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.124982+0000", 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1090000000000004e-06 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126629+0000", 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016468710000000001 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126737+0000", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108565 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.126878+0000", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000140846 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.128117+0000", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012382529999999999 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.128205+0000", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8275000000000004e-05 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135365+0000", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071602590000000004 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135413+0000", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8241000000000001e-05 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.135447+0000", 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3847999999999998e-05 2024-04-25T05:35:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:1280 7.9 7:986e7344:::benchmark_data_smithi043_74356_object1279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.130459+0000", 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.76652357000000004, 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011723460999999999, 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 1280 2024-04-25T05:35:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130459+0000", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130459+0000", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130460+0000", 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3399999999999998e-07 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130488+0000", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.796e-05 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130498+0000", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7690000000000008e-06 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130506+0000", 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0660000000000004e-06 2024-04-25T05:35:17.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130563+0000", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7194000000000002e-05 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130688+0000", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124972 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.130820+0000", 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013213 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.132565+0000", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017453880000000001 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.134051+0000", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001486064 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.142090+0000", 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080387040000000007 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.142140+0000", 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0064e-05 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.142183+0000", 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2416000000000002e-05 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:1968 7.6 7:6a886cd2:::benchmark_data_smithi043_74356_object1967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.925 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.363358+0000", 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.53362423599999997, 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010102168999999999, 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 1968 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363358+0000", 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363358+0000", 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363359+0000", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7899999999999998e-07 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363382+0000", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2660999999999999e-05 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363389+0000", 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0310000000000001e-06 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363395+0000", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3910000000000003e-06 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363453+0000", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7793e-05 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363694+0000", 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024050100000000001 2024-04-25T05:35:17.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.363807+0000", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011337599999999999 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.364649+0000", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00084180400000000003 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.365438+0000", 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00078884099999999998 2024-04-25T05:35:17.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.372522+0000", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070846240000000003 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.373402+0000", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00088020899999999996 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.373461+0000", 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8258999999999999e-05 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.931 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2162 7.9 7:9041be98:::benchmark_data_smithi043_74356_object2161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.428120+0000", 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.46886251699999998, 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011059391, 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2162 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428120+0000", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428120+0000", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428121+0000", 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.263e-06 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428146+0000", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4689e-05 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428157+0000", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0378e-05 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428165+0000", 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1589999999999993e-06 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428221+0000", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6416000000000003e-05 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428342+0000", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121185 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428481+0000", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138987 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429155+0000", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00067324500000000005 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.430258+0000", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011036259999999999 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439080+0000", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0088223620000000003 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439136+0000", 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5377999999999997e-05 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439180+0000", 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3702999999999998e-05 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2164 7.4 7:2da80c39:::benchmark_data_smithi043_74356_object2163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.428980+0000", 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.46800260700000001, 2024-04-25T05:35:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010430663, 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2164 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428980+0000", 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428980+0000", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.428981+0000", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1999999999999998e-07 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429011+0000", 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9631000000000001e-05 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429021+0000", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0292000000000001e-05 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429029+0000", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0339999999999997e-06 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429314+0000", 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00028502200000000001 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429449+0000", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000134979 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.429594+0000", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000145143 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.430137+0000", 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000543429 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.432406+0000", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022687100000000002 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439311+0000", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0069046560000000003 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439366+0000", 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4818000000000003e-05 2024-04-25T05:35:17.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.439411+0000", 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5129000000000001e-05 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2312 7.9 7:9a0da4f8:::benchmark_data_smithi043_74356_object2311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.490048+0000", 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.40693458700000001, 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0097287339999999993, 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.943 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2312 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.490048+0000", 2024-04-25T05:35:17.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.490048+0000", 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.490050+0000", 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5740000000000001e-06 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.491260+0000", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012106059999999999 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.491271+0000", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0747000000000001e-05 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.491280+0000", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8680000000000005e-06 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.494684+0000", 2024-04-25T05:35:17.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034043839999999999 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.494807+0000", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012267100000000001 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.494974+0000", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016689 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.497209+0000", 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022346549999999999 2024-04-25T05:35:17.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.497250+0000", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1288000000000002e-05 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.499675+0000", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024252399999999999 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.499731+0000", 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5482000000000002e-05 2024-04-25T05:35:17.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.499777+0000", 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6329000000000003e-05 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2731 7.4 7:251e36bf:::benchmark_data_smithi043_74356_object2730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.639724+0000", 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.25725874500000001, 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0097235529999999994, 2024-04-25T05:35:17.950 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2731 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639724+0000", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639724+0000", 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639725+0000", 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3099999999999996e-07 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639750+0000", 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4788000000000001e-05 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639760+0000", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0375e-05 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.639768+0000", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5669999999999997e-06 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.642611+0000", 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028437319999999999 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.642730+0000", 2024-04-25T05:35:17.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118749 2024-04-25T05:35:17.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.642955+0000", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000224568 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.644020+0000", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010656680000000001 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.646272+0000", 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022517850000000001 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.648916+0000", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026440119999999998 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.649398+0000", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00048152399999999999 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.649448+0000", 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9854000000000001e-05 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2755 7.6 7:6f56187f:::benchmark_data_smithi043_74356_object2754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.651591+0000", 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.24539170699999999, 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013447422000000001, 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2755 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.957 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651591+0000", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651591+0000", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651593+0000", 2024-04-25T05:35:17.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5030000000000001e-06 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651638+0000", 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5599000000000002e-05 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651655+0000", 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6617999999999999e-05 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651664+0000", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0480000000000004e-06 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.652570+0000", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00090604900000000003 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.652688+0000", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118134 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.652828+0000", 2024-04-25T05:35:17.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013984699999999999 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.656276+0000", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034482100000000002 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.659146+0000", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028703130000000002 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.664929+0000", 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005782662 2024-04-25T05:35:17.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.664990+0000", 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1501000000000006e-05 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.665038+0000", 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7938000000000001e-05 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2760 7.f 7:fac73332:::benchmark_data_smithi043_74356_object2759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.651742+0000", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.24524119599999999, 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014339648, 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2760 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651742+0000", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651742+0000", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651742+0000", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3099999999999998e-07 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651768+0000", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6148000000000001e-05 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651776+0000", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3860000000000001e-06 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651782+0000", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3550000000000001e-06 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.651915+0000", 2024-04-25T05:35:17.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013329499999999999 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.652048+0000", 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000132676 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.652176+0000", 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012835900000000001 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.656237+0000", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040604409999999997 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.658619+0000", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023822119999999999 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.665983+0000", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0073640069999999997 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.666038+0000", 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5352000000000001e-05 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.666081+0000", 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2886000000000002e-05 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:2768 7.f 7:fe620ff9:::benchmark_data_smithi043_74356_object2767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.654173+0000", 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.242809889, 2024-04-25T05:35:17.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012039985, 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 2768 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654173+0000", 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654173+0000", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654173+0000", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6300000000000005e-07 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654199+0000", 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5228e-05 2024-04-25T05:35:17.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654206+0000", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0720000000000002e-06 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.654211+0000", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7830000000000004e-06 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.655478+0000", 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012668720000000001 2024-04-25T05:35:17.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.655593+0000", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115158 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.655707+0000", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011380299999999999 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.658640+0000", 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029322319999999999 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.658678+0000", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8036e-05 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.666123+0000", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074454459999999997 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.666179+0000", 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6030999999999999e-05 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.666213+0000", 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3760999999999997e-05 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3061 7.6 7:65e44f36:::benchmark_data_smithi043_74356_object3060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.768247+0000", 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.12873564300000001, 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099620279999999995, 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.974 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3061 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768247+0000", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768247+0000", 2024-04-25T05:35:17.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768248+0000", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8400000000000003e-07 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768275+0000", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6968999999999999e-05 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768286+0000", 2024-04-25T05:35:17.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0747000000000001e-05 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768294+0000", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6379999999999993e-06 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768460+0000", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016579 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768575+0000", 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115362 2024-04-25T05:35:17.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.768755+0000", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017970099999999999 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.770308+0000", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001552632 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.770787+0000", 2024-04-25T05:35:17.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000479025 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.776780+0000", 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059935040000000002 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.778164+0000", 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013833459999999999 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:17.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.778209+0000", 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5529999999999999e-05 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3079 7.6 7:6c651f3e:::benchmark_data_smithi043_74356_object3078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.774004+0000", 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.122979187, 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010711281, 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.981 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3079 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774004+0000", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774004+0000", 2024-04-25T05:35:17.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774004+0000", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5999999999999991e-07 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774027+0000", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2248000000000001e-05 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774038+0000", 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0878e-05 2024-04-25T05:35:17.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.774046+0000", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6000000000000007e-06 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.778245+0000", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041985240000000004 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.778403+0000", 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000158663 2024-04-25T05:35:17.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.779019+0000", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00061591599999999999 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.783664+0000", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046445059999999996 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.783710+0000", 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6109999999999997e-05 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.783871+0000", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000161106 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.783921+0000", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9866e-05 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.784715+0000", 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00079390400000000001 2024-04-25T05:35:17.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3137 7.f 7:fb825a1e:::benchmark_data_smithi043_74356_object3136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.795604+0000", 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.101378248, 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011264507, 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3137 2024-04-25T05:35:17.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795604+0000", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795604+0000", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795606+0000", 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.689e-06 2024-04-25T05:35:17.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795635+0000", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8770999999999999e-05 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795651+0000", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6220999999999999e-05 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.795661+0000", 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8900000000000002e-06 2024-04-25T05:35:17.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.800229+0000", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045679739999999998 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.800360+0000", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000131497 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.800506+0000", 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014540899999999999 2024-04-25T05:35:17.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.803930+0000", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003423616 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.805821+0000", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018910660000000001 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.806763+0000", 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00094276500000000003 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.806817+0000", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3489000000000002e-05 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.806869+0000", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2120000000000002e-05 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:3236 7.7 e93/88 7:ea926614:::benchmark_data_smithi043_74356_object3235:head v 93'211, mlcod=93'207)", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.838874+0000", 2024-04-25T05:35:17.992 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.058109152999999997, 2024-04-25T05:35:17.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011583354000000001, 2024-04-25T05:35:17.993 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.993 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.993 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838874+0000", 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838874+0000", 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838874+0000", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1199999999999999e-07 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838905+0000", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1432e-05 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838913+0000", 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3209999999999999e-06 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.838919+0000", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9800000000000003e-06 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839589+0000", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00067077100000000002 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839608+0000", 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8488e-05 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850421+0000", 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010813409 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850457+0000", 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5641e-05 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3242 7.f 7:ff49d9ab:::benchmark_data_smithi043_74356_object3241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:17.997 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.839720+0000", 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.057263137999999998, 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011043267000000001, 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3242 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:17.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839720+0000", 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839720+0000", 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839720+0000", 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2200000000000003e-07 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:17.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839743+0000", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2425999999999998e-05 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839750+0000", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7030000000000006e-06 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.839757+0000", 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5749999999999997e-06 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841427+0000", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016697210000000001 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841573+0000", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014631400000000001 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841721+0000", 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014819200000000001 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850472+0000", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0087506800000000003 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850493+0000", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1046000000000001e-05 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850638+0000", 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014466699999999999 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850717+0000", 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9820999999999994e-05 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850763+0000", 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5399999999999999e-05 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3243 7.f 7:fdd84249:::benchmark_data_smithi043_74356_object3242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:18.003 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.841324+0000", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.055658258000000002, 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0096571780000000006, 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3243 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841324+0000", 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841324+0000", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841326+0000", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.238e-06 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841353+0000", 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7514000000000001e-05 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841364+0000", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0879999999999999e-05 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.841372+0000", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0399999999999993e-06 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.842188+0000", 2024-04-25T05:35:18.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00081558300000000004 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.842296+0000", 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108527 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.842433+0000", 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013701699999999999 2024-04-25T05:35:18.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850502+0000", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080689980000000008 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850819+0000", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00031638700000000001 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850890+0000", 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0920000000000005e-05 2024-04-25T05:35:18.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850943+0000", 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3881000000000001e-05 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.850982+0000", 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8192999999999998e-05 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3274 7.9 7:96901d16:::benchmark_data_smithi043_74356_object3273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.854295+0000", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.042687587999999999, 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0096746950000000005, 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3274 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:18.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854295+0000", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854295+0000", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854296+0000", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.75e-07 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854322+0000", 2024-04-25T05:35:18.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6117000000000001e-05 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854330+0000", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.943e-06 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854337+0000", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9600000000000003e-06 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854811+0000", 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000474329 2024-04-25T05:35:18.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.854928+0000", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011720800000000001 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.855080+0000", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000151714 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.857339+0000", 2024-04-25T05:35:18.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022591619999999999 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.857763+0000", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00042403100000000001 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.862825+0000", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050621279999999999 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.863897+0000", 2024-04-25T05:35:18.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001072026 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.863970+0000", 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2502000000000005e-05 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:3276 7.f 7:fbba8180:::benchmark_data_smithi043_74356_object3275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:17.856194+0000", 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.040789179000000002, 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010270253999999999, 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:18.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 3276 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856194+0000", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856194+0000", 2024-04-25T05:35:18.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856194+0000", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2900000000000003e-07 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856221+0000", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6772e-05 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856229+0000", 2024-04-25T05:35:18.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7379999999999994e-06 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.856236+0000", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1010000000000003e-06 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.857103+0000", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000867025 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.857214+0000", 2024-04-25T05:35:18.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111314 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.857460+0000", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024578300000000001 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.859683+0000", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022233690000000002 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.859696+0000", 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3006e-05 2024-04-25T05:35:18.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.866363+0000", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066661699999999999 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.866421+0000", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8208999999999998e-05 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:17.866464+0000", 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3037999999999997e-05 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:18.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:18.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:18.022 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:35:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:18.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:18.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:18.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:18.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:18.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:18.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:18.417 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:18.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:18.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:18.675 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:18.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 15 5630 5615 175.437 168.562 0.00442773 0.00568911 2024-04-25T05:35:18.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:18.817 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:18.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:18.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:19.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:19.080 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:18 smithi043 ceph-mon[27010]: pgmap v41: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 29 MiB/s wr, 461 op/s 2024-04-25T05:35:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:18 smithi043 ceph-mon[37492]: pgmap v41: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 29 MiB/s wr, 461 op/s 2024-04-25T05:35:19.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:18 smithi043 bash[74159]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-25T05:35:19.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:19.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:18 smithi060 ceph-mon[30898]: pgmap v41: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 29 MiB/s wr, 461 op/s 2024-04-25T05:35:19.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:19.351 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:19.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:19.481 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:19.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:35:19.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:19.611 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:19.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:19.721 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 8537 8521 177.494 181.625 0.0100519 0.00562294 2024-04-25T05:35:19.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:19.770 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:19.771 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:19 smithi043 bash[74159]: Writing manifest to image destination 2024-04-25T05:35:19.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:19.869 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:19.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:19.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:20.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:20.032 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:19 smithi043 podman[74159]: 2024-04-25 05:35:19.741833082 +0000 UTC m=+5.038108865 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-25T05:35:20.032 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:19 smithi043 podman[74159]: 2024-04-25 05:35:19.863752681 +0000 UTC m=+5.160028438 container create 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:35:20.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:35:20.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:20.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:20.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:20.259 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:20.262 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:20.279 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:20.352 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:20.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 podman[74159]: 2024-04-25 05:35:20.150748047 +0000 UTC m=+5.447023822 container init 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 podman[74159]: 2024-04-25 05:35:20.156299183 +0000 UTC m=+5.452574948 container start 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.167Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.167Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.168Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-25T05:35:20.415 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.169Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.169Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.170Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.171Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.171Z caller=node_exporter.go:117 level=info collector=arp 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.171Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-25T05:35:20.416 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=edac 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-25T05:35:20.417 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=os 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-25T05:35:20.418 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.172Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=stat 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=time 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=uname 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a[74727]: ts=2024-04-25T05:35:20.173Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 bash[74159]: 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa 2024-04-25T05:35:20.419 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:35:20 smithi043 systemd[1]: Started Ceph node-exporter.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:35:20.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:20.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:20.564 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:20.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.691 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:20.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:20.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 10908 10892 170.165 148.188 0.0015442 0.00582025 2024-04-25T05:35:20.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:20.825 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:20.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:20.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:20.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:21.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[27010]: pgmap v42: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 27 MiB/s wr, 430 op/s 2024-04-25T05:35:21.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.008 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.008 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[37492]: pgmap v42: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 27 MiB/s wr, 430 op/s 2024-04-25T05:35:21.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.009 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:20 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:21.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:21.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:21.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:20 smithi060 ceph-mon[30898]: pgmap v42: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 796 B/s rd, 27 MiB/s wr, 430 op/s 2024-04-25T05:35:21.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:20 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:20 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:20 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:21.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:21.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:21.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:21.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:21.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:21.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:21.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:21.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 13310 13294 166.156 150.125 0.00416102 0.00600937 2024-04-25T05:35:21.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:21.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:21.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:21.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:21.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:21 smithi060 ceph-mon[30898]: Deploying daemon node-exporter.b on smithi060 2024-04-25T05:35:21.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:22.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:35:22.069 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:35:22.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:21 smithi043 ceph-mon[27010]: Deploying daemon node-exporter.b on smithi060 2024-04-25T05:35:22.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:22.069 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:21 smithi043 ceph-mon[37492]: Deploying daemon node-exporter.b on smithi060 2024-04-25T05:35:22.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:22.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:22.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:22.322 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 systemd[1]: Starting Ceph node-exporter.b for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:35:22.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:22.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:22.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:22.495 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:22.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:22.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:22.709 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 bash[62270]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:15610 7.b 7:de71de1f:::benchmark_data_smithi043_74356_object15609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:22.710814+0000", 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0045467859999999997, 2024-04-25T05:35:22.716 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004583078, 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 15610 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710814+0000", 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710814+0000", 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710815+0000", 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2699999999999999e-06 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:22.718 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710836+0000", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0896000000000001e-05 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710848+0000", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1355000000000001e-05 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710863+0000", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5610000000000001e-05 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:22.719 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.710967+0000", 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010336300000000001 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.711163+0000", 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000195949 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.711346+0000", 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018310900000000001 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.720 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.712643+0000", 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012971460000000001 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:15616 7.b 7:d5f31b8b:::benchmark_data_smithi043_74356_object15615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:22.713620+0000", 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0017407309999999999, 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001890624, 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:22.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 15616 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713620+0000", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713620+0000", 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713622+0000", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7910000000000001e-06 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713644+0000", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2045e-05 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713661+0000", 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7166000000000001e-05 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713670+0000", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3829999999999992e-06 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713777+0000", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107259 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.713994+0000", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000217078 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.714139+0000", 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000144995 2024-04-25T05:35:22.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24734.0:15616 7.b e93/88 ondisk, result = 0)", 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:22.715348+0000", 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 1.3078000000000001e-05, 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025289799999999999, 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715348+0000", 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715348+0000", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715349+0000", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6400000000000001e-07 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715353+0000", 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5299999999999998e-06 2024-04-25T05:35:22.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715410+0000", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7018000000000002e-05 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715418+0000", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2120000000000003e-06 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715527+0000", 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109071 2024-04-25T05:35:22.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:22.715536+0000", 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2099999999999993e-06 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 3 2024-04-25T05:35:22.728 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:35:22.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 15635 15619 162.68 145.312 0.014121 0.00613769 2024-04-25T05:35:22.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:22.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:22.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:22.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:22.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:22.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:22 smithi060 ceph-mon[30898]: pgmap v43: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 345 op/s 2024-04-25T05:35:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:22 smithi060 ceph-mon[30898]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:22.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:22.999 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 bash[62270]: Getting image source signatures 2024-04-25T05:35:22.999 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 bash[62270]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-25T05:35:22.999 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 bash[62270]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-25T05:35:22.999 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:22 smithi060 bash[62270]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-25T05:35:23.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:23.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:23.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:22 smithi043 ceph-mon[27010]: pgmap v43: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 345 op/s 2024-04-25T05:35:23.120 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:22 smithi043 ceph-mon[27010]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:23.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:22 smithi043 ceph-mon[37492]: pgmap v43: 148 pgs: 148 active+clean; 173 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 22 MiB/s wr, 345 op/s 2024-04-25T05:35:23.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:22 smithi043 ceph-mon[37492]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:35:23.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:23.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:23.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:23.309 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:23.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:23.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:23.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:23.576 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:23.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:23.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:23.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 17969 17953 160.277 145.875 0.00608089 0.00623299 2024-04-25T05:35:23.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:23.849 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:23.853 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:35:23.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:23.990 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:23.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:24.031 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-25T05:35:21.665346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181541403,"num_objects":2991,"num_object_clones":0,"num_object_copies":8973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":1295,"num_read_kb":1109,"num_write":3568,"num_write_kb":179165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3554,"ondisk_log_size":3554,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":508256,"kb_used_data":258812,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":749452960,"statfs":{"total":767960285184,"available":767439831040,"internally_reserved":0,"allocated":265023488,"data_stored":261245153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2761,"num_write_kb":176704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002677"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671123+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:35:14.671123+0000","last_peered":"2024-04-25T05:35:14.671123+0000","last_clean":"2024-04-25T05:35:14.671123+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:35:14.671123+0000","last_undegraded":"2024-04-25T05:35:14.671123+0000","last_fullsized":"2024-04-25T05:35:14.671123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.921834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.674309+0000","last_change":"2024-04-25T05:34:02.700825+0000","last_active":"2024-04-25T05:35:14.674309+0000","last_peered":"2024-04-25T05:35:14.674309+0000","last_clean":"2024-04-25T05:35:14.674309+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:35:14.674309+0000","last_undegraded":"2024-04-25T05:35:14.674309+0000","last_fullsized":"2024-04-25T05:35:14.674309+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.901521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202488+0000","last_change":"2024-04-25T05:33:58.677608+0000","last_active":"2024-04-25T05:35:15.202488+0000","last_peered":"2024-04-25T05:35:15.202488+0000","last_clean":"2024-04-25T05:35:15.202488+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:35:15.202488+0000","last_undegraded":"2024-04-25T05:35:15.202488+0000","last_fullsized":"2024-04-25T05:35:15.202488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.073153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671111+0000","last_change":"2024-04-25T05:34:00.692628+0000","last_active":"2024-04-25T05:35:14.671111+0000","last_peered":"2024-04-25T05:35:14.671111+0000","last_clean":"2024-04-25T05:35:14.671111+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:35:14.671111+0000","last_undegraded":"2024-04-25T05:35:14.671111+0000","last_fullsized":"2024-04-25T05:35:14.671111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.762564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673706+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:35:14.673706+0000","last_peered":"2024-04-25T05:35:14.673706+0000","last_clean":"2024-04-25T05:35:14.673706+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:35:14.673706+0000","last_undegraded":"2024-04-25T05:35:14.673706+0000","last_fullsized":"2024-04-25T05:35:14.673706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.498112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671189+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:35:14.671189+0000","last_peered":"2024-04-25T05:35:14.671189+0000","last_clean":"2024-04-25T05:35:14.671189+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:35:14.671189+0000","last_undegraded":"2024-04-25T05:35:14.671189+0000","last_fullsized":"2024-04-25T05:35:14.671189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.555566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672901+0000","last_change":"2024-04-25T05:33:58.679726+0000","last_active":"2024-04-25T05:35:14.672901+0000","last_peered":"2024-04-25T05:35:14.672901+0000","last_clean":"2024-04-25T05:35:14.672901+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:35:14.672901+0000","last_undegraded":"2024-04-25T05:35:14.672901+0000","last_fullsized":"2024-04-25T05:35:14.672901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.377018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672278+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:35:14.672278+0000","last_peered":"2024-04-25T05:35:14.672278+0000","last_clean":"2024-04-25T05:35:14.672278+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:35:14.672278+0000","last_undegraded":"2024-04-25T05:35:14.672278+0000","last_fullsized":"2024-04-25T05:35:14.672278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.111490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201690+0000","last_change":"2024-04-25T05:34:04.724350+0000","last_active":"2024-04-25T05:35:15.201690+0000","last_peered":"2024-04-25T05:35:15.201690+0000","last_clean":"2024-04-25T05:35:15.201690+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:35:15.201690+0000","last_undegraded":"2024-04-25T05:35:15.201690+0000","last_fullsized":"2024-04-25T05:35:15.201690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.204881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756467+0000","last_change":"2024-04-25T05:34:02.709504+0000","last_active":"2024-04-25T05:35:14.756467+0000","last_peered":"2024-04-25T05:35:14.756467+0000","last_clean":"2024-04-25T05:35:14.756467+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:35:14.756467+0000","last_undegraded":"2024-04-25T05:35:14.756467+0000","last_fullsized":"2024-04-25T05:35:14.756467+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.151709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756637+0000","last_change":"2024-04-25T05:35:00.716285+0000","last_active":"2024-04-25T05:35:14.756637+0000","last_peered":"2024-04-25T05:35:14.756637+0000","last_clean":"2024-04-25T05:35:14.756637+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:35:14.756637+0000","last_undegraded":"2024-04-25T05:35:14.756637+0000","last_fullsized":"2024-04-25T05:35:14.756637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:35:00.716191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:22.501130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671453+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:35:14.671453+0000","last_peered":"2024-04-25T05:35:14.671453+0000","last_clean":"2024-04-25T05:35:14.671453+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:35:14.671453+0000","last_undegraded":"2024-04-25T05:35:14.671453+0000","last_fullsized":"2024-04-25T05:35:14.671453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.181248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756151+0000","last_change":"2024-04-25T05:34:04.722769+0000","last_active":"2024-04-25T05:35:14.756151+0000","last_peered":"2024-04-25T05:35:14.756151+0000","last_clean":"2024-04-25T05:35:14.756151+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:35:14.756151+0000","last_undegraded":"2024-04-25T05:35:14.756151+0000","last_fullsized":"2024-04-25T05:35:14.756151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.785848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729006+0000","last_change":"2024-04-25T05:34:02.701665+0000","last_active":"2024-04-25T05:35:14.729006+0000","last_peered":"2024-04-25T05:35:14.729006+0000","last_clean":"2024-04-25T05:35:14.729006+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:35:14.729006+0000","last_undegraded":"2024-04-25T05:35:14.729006+0000","last_fullsized":"2024-04-25T05:35:14.729006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.753203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756544+0000","last_change":"2024-04-25T05:35:06.788446+0000","last_active":"2024-04-25T05:35:14.756544+0000","last_peered":"2024-04-25T05:35:14.756544+0000","last_clean":"2024-04-25T05:35:14.756544+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:35:14.756544+0000","last_undegraded":"2024-04-25T05:35:14.756544+0000","last_fullsized":"2024-04-25T05:35:14.756544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.788358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:25.879481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671646+0000","last_change":"2024-04-25T05:34:00.693330+0000","last_active":"2024-04-25T05:35:14.671646+0000","last_peered":"2024-04-25T05:35:14.671646+0000","last_clean":"2024-04-25T05:35:14.671646+0000","last_became_active":"2024-04-25T05:34:00.693003+0000","last_became_peered":"2024-04-25T05:34:00.693003+0000","last_unstale":"2024-04-25T05:35:14.671646+0000","last_undegraded":"2024-04-25T05:35:14.671646+0000","last_fullsized":"2024-04-25T05:35:14.671646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672274+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:35:14.672274+0000","last_peered":"2024-04-25T05:35:14.672274+0000","last_clean":"2024-04-25T05:35:14.672274+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:35:14.672274+0000","last_undegraded":"2024-04-25T05:35:14.672274+0000","last_fullsized":"2024-04-25T05:35:14.672274+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.388118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671892+0000","last_change":"2024-04-25T05:34:02.708603+0000","last_active":"2024-04-25T05:35:14.671892+0000","last_peered":"2024-04-25T05:35:14.671892+0000","last_clean":"2024-04-25T05:35:14.671892+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:35:14.671892+0000","last_undegraded":"2024-04-25T05:35:14.671892+0000","last_fullsized":"2024-04-25T05:35:14.671892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202194+0000","last_change":"2024-04-25T05:35:07.169298+0000","last_active":"2024-04-25T05:35:15.202194+0000","last_peered":"2024-04-25T05:35:15.202194+0000","last_clean":"2024-04-25T05:35:15.202194+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:35:15.202194+0000","last_undegraded":"2024-04-25T05:35:15.202194+0000","last_fullsized":"2024-04-25T05:35:15.202194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:07.169214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.169214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:28.788831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726841+0000","last_change":"2024-04-25T05:34:00.690831+0000","last_active":"2024-04-25T05:35:14.726841+0000","last_peered":"2024-04-25T05:35:14.726841+0000","last_clean":"2024-04-25T05:35:14.726841+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:35:14.726841+0000","last_undegraded":"2024-04-25T05:35:14.726841+0000","last_fullsized":"2024-04-25T05:35:14.726841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.739129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672156+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:35:14.672156+0000","last_peered":"2024-04-25T05:35:14.672156+0000","last_clean":"2024-04-25T05:35:14.672156+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:35:14.672156+0000","last_undegraded":"2024-04-25T05:35:14.672156+0000","last_fullsized":"2024-04-25T05:35:14.672156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.955402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671871+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:35:14.671871+0000","last_peered":"2024-04-25T05:35:14.671871+0000","last_clean":"2024-04-25T05:35:14.671871+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:35:14.671871+0000","last_undegraded":"2024-04-25T05:35:14.671871+0000","last_fullsized":"2024-04-25T05:35:14.671871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.792766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671484+0000","last_change":"2024-04-25T05:33:58.677738+0000","last_active":"2024-04-25T05:35:14.671484+0000","last_peered":"2024-04-25T05:35:14.671484+0000","last_clean":"2024-04-25T05:35:14.671484+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:35:14.671484+0000","last_undegraded":"2024-04-25T05:35:14.671484+0000","last_fullsized":"2024-04-25T05:35:14.671484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.705248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":94,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672565+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:35:14.672565+0000","last_peered":"2024-04-25T05:35:14.672565+0000","last_clean":"2024-04-25T05:35:14.672565+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:35:14.672565+0000","last_undegraded":"2024-04-25T05:35:14.672565+0000","last_fullsized":"2024-04-25T05:35:14.672565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.537702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729846+0000","last_change":"2024-04-25T05:34:04.725886+0000","last_active":"2024-04-25T05:35:14.729846+0000","last_peered":"2024-04-25T05:35:14.729846+0000","last_clean":"2024-04-25T05:35:14.729846+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:35:14.729846+0000","last_undegraded":"2024-04-25T05:35:14.729846+0000","last_fullsized":"2024-04-25T05:35:14.729846+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":82,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915211+0000","last_change":"2024-04-25T05:34:02.701003+0000","last_active":"2024-04-25T05:35:14.915211+0000","last_peered":"2024-04-25T05:35:14.915211+0000","last_clean":"2024-04-25T05:35:14.915211+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:35:14.915211+0000","last_undegraded":"2024-04-25T05:35:14.915211+0000","last_fullsized":"2024-04-25T05:35:14.915211+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.795737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.674400+0000","last_change":"2024-04-25T05:33:58.678204+0000","last_active":"2024-04-25T05:35:14.674400+0000","last_peered":"2024-04-25T05:35:14.674400+0000","last_clean":"2024-04-25T05:35:14.674400+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:35:14.674400+0000","last_undegraded":"2024-04-25T05:35:14.674400+0000","last_fullsized":"2024-04-25T05:35:14.674400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.231648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202570+0000","last_change":"2024-04-25T05:34:01.002306+0000","last_active":"2024-04-25T05:35:15.202570+0000","last_peered":"2024-04-25T05:35:15.202570+0000","last_clean":"2024-04-25T05:35:15.202570+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:35:15.202570+0000","last_undegraded":"2024-04-25T05:35:15.202570+0000","last_fullsized":"2024-04-25T05:35:15.202570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.875946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671575+0000","last_change":"2024-04-25T05:34:04.722159+0000","last_active":"2024-04-25T05:35:14.671575+0000","last_peered":"2024-04-25T05:35:14.671575+0000","last_clean":"2024-04-25T05:35:14.671575+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:35:14.671575+0000","last_undegraded":"2024-04-25T05:35:14.671575+0000","last_fullsized":"2024-04-25T05:35:14.671575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202730+0000","last_change":"2024-04-25T05:34:02.710777+0000","last_active":"2024-04-25T05:35:15.202730+0000","last_peered":"2024-04-25T05:35:15.202730+0000","last_clean":"2024-04-25T05:35:15.202730+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:35:15.202730+0000","last_undegraded":"2024-04-25T05:35:15.202730+0000","last_fullsized":"2024-04-25T05:35:15.202730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.200275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.670984+0000","last_change":"2024-04-25T05:33:58.676498+0000","last_active":"2024-04-25T05:35:14.670984+0000","last_peered":"2024-04-25T05:35:14.670984+0000","last_clean":"2024-04-25T05:35:14.670984+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:35:14.670984+0000","last_undegraded":"2024-04-25T05:35:14.670984+0000","last_fullsized":"2024-04-25T05:35:14.670984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.377967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.916289+0000","last_change":"2024-04-25T05:34:00.691266+0000","last_active":"2024-04-25T05:35:14.916289+0000","last_peered":"2024-04-25T05:35:14.916289+0000","last_clean":"2024-04-25T05:35:14.916289+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:35:14.916289+0000","last_undegraded":"2024-04-25T05:35:14.916289+0000","last_fullsized":"2024-04-25T05:35:14.916289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.506586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729360+0000","last_change":"2024-04-25T05:34:04.725684+0000","last_active":"2024-04-25T05:35:14.729360+0000","last_peered":"2024-04-25T05:35:14.729360+0000","last_clean":"2024-04-25T05:35:14.729360+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:35:14.729360+0000","last_undegraded":"2024-04-25T05:35:14.729360+0000","last_fullsized":"2024-04-25T05:35:14.729360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.515944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915180+0000","last_change":"2024-04-25T05:34:02.701292+0000","last_active":"2024-04-25T05:35:14.915180+0000","last_peered":"2024-04-25T05:35:14.915180+0000","last_clean":"2024-04-25T05:35:14.915180+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:35:14.915180+0000","last_undegraded":"2024-04-25T05:35:14.915180+0000","last_fullsized":"2024-04-25T05:35:14.915180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.506659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202637+0000","last_change":"2024-04-25T05:33:58.678452+0000","last_active":"2024-04-25T05:35:15.202637+0000","last_peered":"2024-04-25T05:35:15.202637+0000","last_clean":"2024-04-25T05:35:15.202637+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:35:15.202637+0000","last_undegraded":"2024-04-25T05:35:15.202637+0000","last_fullsized":"2024-04-25T05:35:15.202637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.043078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671040+0000","last_change":"2024-04-25T05:34:00.692546+0000","last_active":"2024-04-25T05:35:14.671040+0000","last_peered":"2024-04-25T05:35:14.671040+0000","last_clean":"2024-04-25T05:35:14.671040+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:35:14.671040+0000","last_undegraded":"2024-04-25T05:35:14.671040+0000","last_fullsized":"2024-04-25T05:35:14.671040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.378025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.674118+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:35:14.674118+0000","last_peered":"2024-04-25T05:35:14.674118+0000","last_clean":"2024-04-25T05:35:14.674118+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:35:14.674118+0000","last_undegraded":"2024-04-25T05:35:14.674118+0000","last_fullsized":"2024-04-25T05:35:14.674118+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.872916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.728952+0000","last_change":"2024-04-25T05:34:02.701784+0000","last_active":"2024-04-25T05:35:14.728952+0000","last_peered":"2024-04-25T05:35:14.728952+0000","last_clean":"2024-04-25T05:35:14.728952+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:35:14.728952+0000","last_undegraded":"2024-04-25T05:35:14.728952+0000","last_fullsized":"2024-04-25T05:35:14.728952+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756789+0000","last_change":"2024-04-25T05:33:58.679682+0000","last_active":"2024-04-25T05:35:14.756789+0000","last_peered":"2024-04-25T05:35:14.756789+0000","last_clean":"2024-04-25T05:35:14.756789+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:35:14.756789+0000","last_undegraded":"2024-04-25T05:35:14.756789+0000","last_fullsized":"2024-04-25T05:35:14.756789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672176+0000","last_change":"2024-04-25T05:34:00.999704+0000","last_active":"2024-04-25T05:35:14.672176+0000","last_peered":"2024-04-25T05:35:14.672176+0000","last_clean":"2024-04-25T05:35:14.672176+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:35:14.672176+0000","last_undegraded":"2024-04-25T05:35:14.672176+0000","last_fullsized":"2024-04-25T05:35:14.672176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.377986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201386+0000","last_change":"2024-04-25T05:34:04.727071+0000","last_active":"2024-04-25T05:35:15.201386+0000","last_peered":"2024-04-25T05:35:15.201386+0000","last_clean":"2024-04-25T05:35:15.201386+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:35:15.201386+0000","last_undegraded":"2024-04-25T05:35:15.201386+0000","last_fullsized":"2024-04-25T05:35:15.201386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.204913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673618+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:35:14.673618+0000","last_peered":"2024-04-25T05:35:14.673618+0000","last_clean":"2024-04-25T05:35:14.673618+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:35:14.673618+0000","last_undegraded":"2024-04-25T05:35:14.673618+0000","last_fullsized":"2024-04-25T05:35:14.673618+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.038248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729825+0000","last_change":"2024-04-25T05:33:58.687406+0000","last_active":"2024-04-25T05:35:14.729825+0000","last_peered":"2024-04-25T05:35:14.729825+0000","last_clean":"2024-04-25T05:35:14.729825+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:35:14.729825+0000","last_undegraded":"2024-04-25T05:35:14.729825+0000","last_fullsized":"2024-04-25T05:35:14.729825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673354+0000","last_change":"2024-04-25T05:34:00.689589+0000","last_active":"2024-04-25T05:35:14.673354+0000","last_peered":"2024-04-25T05:35:14.673354+0000","last_clean":"2024-04-25T05:35:14.673354+0000","last_became_active":"2024-04-25T05:34:00.688897+0000","last_became_peered":"2024-04-25T05:34:00.688897+0000","last_unstale":"2024-04-25T05:35:14.673354+0000","last_undegraded":"2024-04-25T05:35:14.673354+0000","last_fullsized":"2024-04-25T05:35:14.673354+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671843+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:35:14.671843+0000","last_peered":"2024-04-25T05:35:14.671843+0000","last_clean":"2024-04-25T05:35:14.671843+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:35:14.671843+0000","last_undegraded":"2024-04-25T05:35:14.671843+0000","last_fullsized":"2024-04-25T05:35:14.671843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.871648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673103+0000","last_change":"2024-04-25T05:34:02.699322+0000","last_active":"2024-04-25T05:35:14.673103+0000","last_peered":"2024-04-25T05:35:14.673103+0000","last_clean":"2024-04-25T05:35:14.673103+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:35:14.673103+0000","last_undegraded":"2024-04-25T05:35:14.673103+0000","last_fullsized":"2024-04-25T05:35:14.673103+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.105773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671489+0000","last_change":"2024-04-25T05:35:02.707425+0000","last_active":"2024-04-25T05:35:14.671489+0000","last_peered":"2024-04-25T05:35:14.671489+0000","last_clean":"2024-04-25T05:35:14.671489+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:35:14.671489+0000","last_undegraded":"2024-04-25T05:35:14.671489+0000","last_fullsized":"2024-04-25T05:35:14.671489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:19.328355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671824+0000","last_change":"2024-04-25T05:35:07.728633+0000","last_active":"2024-04-25T05:35:14.671824+0000","last_peered":"2024-04-25T05:35:14.671824+0000","last_clean":"2024-04-25T05:35:14.671824+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:35:14.671824+0000","last_undegraded":"2024-04-25T05:35:14.671824+0000","last_fullsized":"2024-04-25T05:35:14.671824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:19.060382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202303+0000","last_change":"2024-04-25T05:34:04.724697+0000","last_active":"2024-04-25T05:35:15.202303+0000","last_peered":"2024-04-25T05:35:15.202303+0000","last_clean":"2024-04-25T05:35:15.202303+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:35:15.202303+0000","last_undegraded":"2024-04-25T05:35:15.202303+0000","last_fullsized":"2024-04-25T05:35:15.202303+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.282641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.755791+0000","last_change":"2024-04-25T05:34:02.708587+0000","last_active":"2024-04-25T05:35:14.755791+0000","last_peered":"2024-04-25T05:35:14.755791+0000","last_clean":"2024-04-25T05:35:14.755791+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:35:14.755791+0000","last_undegraded":"2024-04-25T05:35:14.755791+0000","last_fullsized":"2024-04-25T05:35:14.755791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.755018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729384+0000","last_change":"2024-04-25T05:33:58.685869+0000","last_active":"2024-04-25T05:35:14.729384+0000","last_peered":"2024-04-25T05:35:14.729384+0000","last_clean":"2024-04-25T05:35:14.729384+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:35:14.729384+0000","last_undegraded":"2024-04-25T05:35:14.729384+0000","last_fullsized":"2024-04-25T05:35:14.729384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:16.260635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672946+0000","last_change":"2024-04-25T05:34:00.999737+0000","last_active":"2024-04-25T05:35:14.672946+0000","last_peered":"2024-04-25T05:35:14.672946+0000","last_clean":"2024-04-25T05:35:14.672946+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:35:14.672946+0000","last_undegraded":"2024-04-25T05:35:14.672946+0000","last_fullsized":"2024-04-25T05:35:14.672946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672441+0000","last_change":"2024-04-25T05:34:04.715688+0000","last_active":"2024-04-25T05:35:14.672441+0000","last_peered":"2024-04-25T05:35:14.672441+0000","last_clean":"2024-04-25T05:35:14.672441+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:35:14.672441+0000","last_undegraded":"2024-04-25T05:35:14.672441+0000","last_fullsized":"2024-04-25T05:35:14.672441+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.959881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673328+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:35:14.673328+0000","last_peered":"2024-04-25T05:35:14.673328+0000","last_clean":"2024-04-25T05:35:14.673328+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:35:14.673328+0000","last_undegraded":"2024-04-25T05:35:14.673328+0000","last_fullsized":"2024-04-25T05:35:14.673328+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.077903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201946+0000","last_change":"2024-04-25T05:35:05.182398+0000","last_active":"2024-04-25T05:35:15.201946+0000","last_peered":"2024-04-25T05:35:15.201946+0000","last_clean":"2024-04-25T05:35:15.201946+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:35:15.201946+0000","last_undegraded":"2024-04-25T05:35:15.201946+0000","last_fullsized":"2024-04-25T05:35:15.201946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:05.182288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:05.182288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:16.802484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756852+0000","last_change":"2024-04-25T05:34:01.002652+0000","last_active":"2024-04-25T05:35:14.756852+0000","last_peered":"2024-04-25T05:35:14.756852+0000","last_clean":"2024-04-25T05:35:14.756852+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:35:14.756852+0000","last_undegraded":"2024-04-25T05:35:14.756852+0000","last_fullsized":"2024-04-25T05:35:14.756852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.164431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.727237+0000","last_change":"2024-04-25T05:34:04.719603+0000","last_active":"2024-04-25T05:35:14.727237+0000","last_peered":"2024-04-25T05:35:14.727237+0000","last_clean":"2024-04-25T05:35:14.727237+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:35:14.727237+0000","last_undegraded":"2024-04-25T05:35:14.727237+0000","last_fullsized":"2024-04-25T05:35:14.727237+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.818750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672011+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:35:14.672011+0000","last_peered":"2024-04-25T05:35:14.672011+0000","last_clean":"2024-04-25T05:35:14.672011+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:35:14.672011+0000","last_undegraded":"2024-04-25T05:35:14.672011+0000","last_fullsized":"2024-04-25T05:35:14.672011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.004188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729421+0000","last_change":"2024-04-25T05:33:58.685792+0000","last_active":"2024-04-25T05:35:14.729421+0000","last_peered":"2024-04-25T05:35:14.729421+0000","last_clean":"2024-04-25T05:35:14.729421+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:35:14.729421+0000","last_undegraded":"2024-04-25T05:35:14.729421+0000","last_fullsized":"2024-04-25T05:35:14.729421+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202675+0000","last_change":"2024-04-25T05:34:01.002399+0000","last_active":"2024-04-25T05:35:15.202675+0000","last_peered":"2024-04-25T05:35:15.202675+0000","last_clean":"2024-04-25T05:35:15.202675+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:35:15.202675+0000","last_undegraded":"2024-04-25T05:35:15.202675+0000","last_fullsized":"2024-04-25T05:35:15.202675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.611153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729823+0000","last_change":"2024-04-25T05:34:04.760885+0000","last_active":"2024-04-25T05:35:14.729823+0000","last_peered":"2024-04-25T05:35:14.729823+0000","last_clean":"2024-04-25T05:35:14.729823+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:35:14.729823+0000","last_undegraded":"2024-04-25T05:35:14.729823+0000","last_fullsized":"2024-04-25T05:35:14.729823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.397373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.755801+0000","last_change":"2024-04-25T05:34:02.700150+0000","last_active":"2024-04-25T05:35:14.755801+0000","last_peered":"2024-04-25T05:35:14.755801+0000","last_clean":"2024-04-25T05:35:14.755801+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:35:14.755801+0000","last_undegraded":"2024-04-25T05:35:14.755801+0000","last_fullsized":"2024-04-25T05:35:14.755801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.726607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915319+0000","last_change":"2024-04-25T05:33:58.682812+0000","last_active":"2024-04-25T05:35:14.915319+0000","last_peered":"2024-04-25T05:35:14.915319+0000","last_clean":"2024-04-25T05:35:14.915319+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:35:14.915319+0000","last_undegraded":"2024-04-25T05:35:14.915319+0000","last_fullsized":"2024-04-25T05:35:14.915319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.737056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672247+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:35:14.672247+0000","last_peered":"2024-04-25T05:35:14.672247+0000","last_clean":"2024-04-25T05:35:14.672247+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:35:14.672247+0000","last_undegraded":"2024-04-25T05:35:14.672247+0000","last_fullsized":"2024-04-25T05:35:14.672247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:26.633917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"93'188","reported_seq":208,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.630655+0000","last_change":"2024-04-25T05:35:09.679731+0000","last_active":"2024-04-25T05:35:17.630655+0000","last_peered":"2024-04-25T05:35:17.630655+0000","last_clean":"2024-04-25T05:35:17.630655+0000","last_became_active":"2024-04-25T05:35:09.679411+0000","last_became_peered":"2024-04-25T05:35:09.679411+0000","last_unstale":"2024-04-25T05:35:17.630655+0000","last_undegraded":"2024-04-25T05:35:17.630655+0000","last_fullsized":"2024-04-25T05:35:17.630655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:10.925440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756393+0000","last_change":"2024-04-25T05:34:04.759369+0000","last_active":"2024-04-25T05:35:14.756393+0000","last_peered":"2024-04-25T05:35:14.756393+0000","last_clean":"2024-04-25T05:35:14.756393+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:35:14.756393+0000","last_undegraded":"2024-04-25T05:35:14.756393+0000","last_fullsized":"2024-04-25T05:35:14.756393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.785793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.730170+0000","last_change":"2024-04-25T05:33:58.689200+0000","last_active":"2024-04-25T05:35:14.730170+0000","last_peered":"2024-04-25T05:35:14.730170+0000","last_clean":"2024-04-25T05:35:14.730170+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:35:14.730170+0000","last_undegraded":"2024-04-25T05:35:14.730170+0000","last_fullsized":"2024-04-25T05:35:14.730170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.468188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756090+0000","last_change":"2024-04-25T05:34:01.000840+0000","last_active":"2024-04-25T05:35:14.756090+0000","last_peered":"2024-04-25T05:35:14.756090+0000","last_clean":"2024-04-25T05:35:14.756090+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:35:14.756090+0000","last_undegraded":"2024-04-25T05:35:14.756090+0000","last_fullsized":"2024-04-25T05:35:14.756090+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.859671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.728990+0000","last_change":"2024-04-25T05:35:05.873060+0000","last_active":"2024-04-25T05:35:14.728990+0000","last_peered":"2024-04-25T05:35:14.728990+0000","last_clean":"2024-04-25T05:35:14.728990+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:35:14.728990+0000","last_undegraded":"2024-04-25T05:35:14.728990+0000","last_fullsized":"2024-04-25T05:35:14.728990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:05.871952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:05.871952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:12.019385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"93'178","reported_seq":198,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.645075+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:35:17.645075+0000","last_peered":"2024-04-25T05:35:17.645075+0000","last_clean":"2024-04-25T05:35:17.645075+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:35:17.645075+0000","last_undegraded":"2024-04-25T05:35:17.645075+0000","last_fullsized":"2024-04-25T05:35:17.645075+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:33.209644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673843+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:35:14.673843+0000","last_peered":"2024-04-25T05:35:14.673843+0000","last_clean":"2024-04-25T05:35:14.673843+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:35:14.673843+0000","last_undegraded":"2024-04-25T05:35:14.673843+0000","last_fullsized":"2024-04-25T05:35:14.673843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.987326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.730031+0000","last_change":"2024-04-25T05:35:02.907524+0000","last_active":"2024-04-25T05:35:14.730031+0000","last_peered":"2024-04-25T05:35:14.730031+0000","last_clean":"2024-04-25T05:35:14.730031+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:35:14.730031+0000","last_undegraded":"2024-04-25T05:35:14.730031+0000","last_fullsized":"2024-04-25T05:35:14.730031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:02.907405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.907405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:24.819879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":105,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671926+0000","last_change":"2024-04-25T05:34:00.694001+0000","last_active":"2024-04-25T05:35:14.671926+0000","last_peered":"2024-04-25T05:35:14.671926+0000","last_clean":"2024-04-25T05:35:14.671926+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:35:14.671926+0000","last_undegraded":"2024-04-25T05:35:14.671926+0000","last_fullsized":"2024-04-25T05:35:14.671926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726216+0000","last_change":"2024-04-25T05:34:02.710703+0000","last_active":"2024-04-25T05:35:14.726216+0000","last_peered":"2024-04-25T05:35:14.726216+0000","last_clean":"2024-04-25T05:35:14.726216+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:35:14.726216+0000","last_undegraded":"2024-04-25T05:35:14.726216+0000","last_fullsized":"2024-04-25T05:35:14.726216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.457617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"93'155","reported_seq":175,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.656811+0000","last_change":"2024-04-25T05:35:09.681598+0000","last_active":"2024-04-25T05:35:17.656811+0000","last_peered":"2024-04-25T05:35:17.656811+0000","last_clean":"2024-04-25T05:35:17.656811+0000","last_became_active":"2024-04-25T05:35:09.681316+0000","last_became_peered":"2024-04-25T05:35:09.681316+0000","last_unstale":"2024-04-25T05:35:17.656811+0000","last_undegraded":"2024-04-25T05:35:17.656811+0000","last_fullsized":"2024-04-25T05:35:17.656811+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:23.447693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10158080,"num_objects":155,"num_object_clones":0,"num_object_copies":465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729310+0000","last_change":"2024-04-25T05:34:04.727176+0000","last_active":"2024-04-25T05:35:14.729310+0000","last_peered":"2024-04-25T05:35:14.729310+0000","last_clean":"2024-04-25T05:35:14.729310+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:35:14.729310+0000","last_undegraded":"2024-04-25T05:35:14.729310+0000","last_fullsized":"2024-04-25T05:35:14.729310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726900+0000","last_change":"2024-04-25T05:34:02.710395+0000","last_active":"2024-04-25T05:35:14.726900+0000","last_peered":"2024-04-25T05:35:14.726900+0000","last_clean":"2024-04-25T05:35:14.726900+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:35:14.726900+0000","last_undegraded":"2024-04-25T05:35:14.726900+0000","last_fullsized":"2024-04-25T05:35:14.726900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.749912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.730058+0000","last_change":"2024-04-25T05:35:04.909850+0000","last_active":"2024-04-25T05:35:14.730058+0000","last_peered":"2024-04-25T05:35:14.730058+0000","last_clean":"2024-04-25T05:35:14.730058+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:35:14.730058+0000","last_undegraded":"2024-04-25T05:35:14.730058+0000","last_fullsized":"2024-04-25T05:35:14.730058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:04.909753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:04.909753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:17.185937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":102,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915724+0000","last_change":"2024-04-25T05:34:00.999892+0000","last_active":"2024-04-25T05:35:14.915724+0000","last_peered":"2024-04-25T05:35:14.915724+0000","last_clean":"2024-04-25T05:35:14.915724+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:35:14.915724+0000","last_undegraded":"2024-04-25T05:35:14.915724+0000","last_fullsized":"2024-04-25T05:35:14.915724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"93'180","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.625917+0000","last_change":"2024-04-25T05:35:09.681082+0000","last_active":"2024-04-25T05:35:17.625917+0000","last_peered":"2024-04-25T05:35:17.625917+0000","last_clean":"2024-04-25T05:35:17.625917+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:35:17.625917+0000","last_undegraded":"2024-04-25T05:35:17.625917+0000","last_fullsized":"2024-04-25T05:35:17.625917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:21.212732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796480,"num_objects":180,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":180,"num_write_kb":11520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201355+0000","last_change":"2024-04-25T05:34:04.725833+0000","last_active":"2024-04-25T05:35:15.201355+0000","last_peered":"2024-04-25T05:35:15.201355+0000","last_clean":"2024-04-25T05:35:15.201355+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:35:15.201355+0000","last_undegraded":"2024-04-25T05:35:15.201355+0000","last_fullsized":"2024-04-25T05:35:15.201355+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:30.963442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726216+0000","last_change":"2024-04-25T05:34:02.700901+0000","last_active":"2024-04-25T05:35:14.726216+0000","last_peered":"2024-04-25T05:35:14.726216+0000","last_clean":"2024-04-25T05:35:14.726216+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:35:14.726216+0000","last_undegraded":"2024-04-25T05:35:14.726216+0000","last_fullsized":"2024-04-25T05:35:14.726216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.464318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756614+0000","last_change":"2024-04-25T05:33:58.685937+0000","last_active":"2024-04-25T05:35:14.756614+0000","last_peered":"2024-04-25T05:35:14.756614+0000","last_clean":"2024-04-25T05:35:14.756614+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:35:14.756614+0000","last_undegraded":"2024-04-25T05:35:14.756614+0000","last_fullsized":"2024-04-25T05:35:14.756614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.785863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202311+0000","last_change":"2024-04-25T05:34:00.696605+0000","last_active":"2024-04-25T05:35:15.202311+0000","last_peered":"2024-04-25T05:35:15.202311+0000","last_clean":"2024-04-25T05:35:15.202311+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:35:15.202311+0000","last_undegraded":"2024-04-25T05:35:15.202311+0000","last_fullsized":"2024-04-25T05:35:15.202311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.205020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"93'176","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.641340+0000","last_change":"2024-04-25T05:35:09.684480+0000","last_active":"2024-04-25T05:35:17.641340+0000","last_peered":"2024-04-25T05:35:17.641340+0000","last_clean":"2024-04-25T05:35:17.641340+0000","last_became_active":"2024-04-25T05:35:09.683860+0000","last_became_peered":"2024-04-25T05:35:09.683860+0000","last_unstale":"2024-04-25T05:35:17.641340+0000","last_undegraded":"2024-04-25T05:35:17.641340+0000","last_fullsized":"2024-04-25T05:35:17.641340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:34.249446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":11264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673239+0000","last_change":"2024-04-25T05:34:04.722629+0000","last_active":"2024-04-25T05:35:14.673239+0000","last_peered":"2024-04-25T05:35:14.673239+0000","last_clean":"2024-04-25T05:35:14.673239+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:35:14.673239+0000","last_undegraded":"2024-04-25T05:35:14.673239+0000","last_fullsized":"2024-04-25T05:35:14.673239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756260+0000","last_change":"2024-04-25T05:34:00.675901+0000","last_active":"2024-04-25T05:35:14.756260+0000","last_peered":"2024-04-25T05:35:14.756260+0000","last_clean":"2024-04-25T05:35:14.756260+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:35:14.756260+0000","last_undegraded":"2024-04-25T05:35:14.756260+0000","last_fullsized":"2024-04-25T05:35:14.756260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.111301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671485+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:35:14.671485+0000","last_peered":"2024-04-25T05:35:14.671485+0000","last_clean":"2024-04-25T05:35:14.671485+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:35:14.671485+0000","last_undegraded":"2024-04-25T05:35:14.671485+0000","last_fullsized":"2024-04-25T05:35:14.671485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.968115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'34","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673836+0000","last_change":"2024-04-25T05:34:00.688687+0000","last_active":"2024-04-25T05:35:14.673836+0000","last_peered":"2024-04-25T05:35:14.673836+0000","last_clean":"2024-04-25T05:35:14.673836+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:35:14.673836+0000","last_undegraded":"2024-04-25T05:35:14.673836+0000","last_fullsized":"2024-04-25T05:35:14.673836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.582841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201137+0000","last_change":"2024-04-25T05:34:02.708391+0000","last_active":"2024-04-25T05:35:15.201137+0000","last_peered":"2024-04-25T05:35:15.201137+0000","last_clean":"2024-04-25T05:35:15.201137+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:35:15.201137+0000","last_undegraded":"2024-04-25T05:35:15.201137+0000","last_fullsized":"2024-04-25T05:35:15.201137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:24.712592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"93'178","reported_seq":197,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.657327+0000","last_change":"2024-04-25T05:35:09.679881+0000","last_active":"2024-04-25T05:35:17.657327+0000","last_peered":"2024-04-25T05:35:17.657327+0000","last_clean":"2024-04-25T05:35:17.657327+0000","last_became_active":"2024-04-25T05:35:09.679580+0000","last_became_peered":"2024-04-25T05:35:09.679580+0000","last_unstale":"2024-04-25T05:35:17.657327+0000","last_undegraded":"2024-04-25T05:35:17.657327+0000","last_fullsized":"2024-04-25T05:35:17.657327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:31.041110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.730162+0000","last_change":"2024-04-25T05:34:04.760767+0000","last_active":"2024-04-25T05:35:14.730162+0000","last_peered":"2024-04-25T05:35:14.730162+0000","last_clean":"2024-04-25T05:35:14.730162+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:35:14.730162+0000","last_undegraded":"2024-04-25T05:35:14.730162+0000","last_fullsized":"2024-04-25T05:35:14.730162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672472+0000","last_change":"2024-04-25T05:33:58.679884+0000","last_active":"2024-04-25T05:35:14.672472+0000","last_peered":"2024-04-25T05:35:14.672472+0000","last_clean":"2024-04-25T05:35:14.672472+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:35:14.672472+0000","last_undegraded":"2024-04-25T05:35:14.672472+0000","last_fullsized":"2024-04-25T05:35:14.672472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.742024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726527+0000","last_change":"2024-04-25T05:35:01.703740+0000","last_active":"2024-04-25T05:35:14.726527+0000","last_peered":"2024-04-25T05:35:14.726527+0000","last_clean":"2024-04-25T05:35:14.726527+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:35:14.726527+0000","last_undegraded":"2024-04-25T05:35:14.726527+0000","last_fullsized":"2024-04-25T05:35:14.726527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:01.703657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:35:01.703657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:05.485939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'19","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672365+0000","last_change":"2024-04-25T05:34:00.694199+0000","last_active":"2024-04-25T05:35:14.672365+0000","last_peered":"2024-04-25T05:35:14.672365+0000","last_clean":"2024-04-25T05:35:14.672365+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:35:14.672365+0000","last_undegraded":"2024-04-25T05:35:14.672365+0000","last_fullsized":"2024-04-25T05:35:14.672365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.727134+0000","last_change":"2024-04-25T05:34:02.702812+0000","last_active":"2024-04-25T05:35:14.727134+0000","last_peered":"2024-04-25T05:35:14.727134+0000","last_clean":"2024-04-25T05:35:14.727134+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:35:14.727134+0000","last_undegraded":"2024-04-25T05:35:14.727134+0000","last_fullsized":"2024-04-25T05:35:14.727134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.890640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"93'160","reported_seq":180,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.641475+0000","last_change":"2024-04-25T05:35:09.683951+0000","last_active":"2024-04-25T05:35:17.641475+0000","last_peered":"2024-04-25T05:35:17.641475+0000","last_clean":"2024-04-25T05:35:17.641475+0000","last_became_active":"2024-04-25T05:35:09.682815+0000","last_became_peered":"2024-04-25T05:35:09.682815+0000","last_unstale":"2024-04-25T05:35:17.641475+0000","last_undegraded":"2024-04-25T05:35:17.641475+0000","last_fullsized":"2024-04-25T05:35:17.641475+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:36.688405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":160,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756825+0000","last_change":"2024-04-25T05:34:04.722965+0000","last_active":"2024-04-25T05:35:14.756825+0000","last_peered":"2024-04-25T05:35:14.756825+0000","last_clean":"2024-04-25T05:35:14.756825+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:35:14.756825+0000","last_undegraded":"2024-04-25T05:35:14.756825+0000","last_fullsized":"2024-04-25T05:35:14.756825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:27.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":314,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729487+0000","last_change":"2024-04-25T05:34:20.908770+0000","last_active":"2024-04-25T05:35:14.729487+0000","last_peered":"2024-04-25T05:35:14.729487+0000","last_clean":"2024-04-25T05:35:14.729487+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:35:14.729487+0000","last_undegraded":"2024-04-25T05:35:14.729487+0000","last_fullsized":"2024-04-25T05:35:14.729487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-04-25T05:34:20.908611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:34:20.908611+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:33.546403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671474+0000","last_change":"2024-04-25T05:33:58.679230+0000","last_active":"2024-04-25T05:35:14.671474+0000","last_peered":"2024-04-25T05:35:14.671474+0000","last_clean":"2024-04-25T05:35:14.671474+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:35:14.671474+0000","last_undegraded":"2024-04-25T05:35:14.671474+0000","last_fullsized":"2024-04-25T05:35:14.671474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.378062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.916152+0000","last_change":"2024-04-25T05:34:01.002201+0000","last_active":"2024-04-25T05:35:14.916152+0000","last_peered":"2024-04-25T05:35:14.916152+0000","last_clean":"2024-04-25T05:35:14.916152+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:35:14.916152+0000","last_undegraded":"2024-04-25T05:35:14.916152+0000","last_fullsized":"2024-04-25T05:35:14.916152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.506626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729823+0000","last_change":"2024-04-25T05:34:02.701047+0000","last_active":"2024-04-25T05:35:14.729823+0000","last_peered":"2024-04-25T05:35:14.729823+0000","last_clean":"2024-04-25T05:35:14.729823+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:35:14.729823+0000","last_undegraded":"2024-04-25T05:35:14.729823+0000","last_fullsized":"2024-04-25T05:35:14.729823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"93'170","reported_seq":190,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.639077+0000","last_change":"2024-04-25T05:35:09.683621+0000","last_active":"2024-04-25T05:35:17.639077+0000","last_peered":"2024-04-25T05:35:17.639077+0000","last_clean":"2024-04-25T05:35:17.639077+0000","last_became_active":"2024-04-25T05:35:09.683250+0000","last_became_peered":"2024-04-25T05:35:09.683250+0000","last_unstale":"2024-04-25T05:35:17.639077+0000","last_undegraded":"2024-04-25T05:35:17.639077+0000","last_fullsized":"2024-04-25T05:35:17.639077+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:32.531876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11141120,"num_objects":170,"num_object_clones":0,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":170,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673526+0000","last_change":"2024-04-25T05:34:04.726534+0000","last_active":"2024-04-25T05:35:14.673526+0000","last_peered":"2024-04-25T05:35:14.673526+0000","last_clean":"2024-04-25T05:35:14.673526+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:35:14.673526+0000","last_undegraded":"2024-04-25T05:35:14.673526+0000","last_fullsized":"2024-04-25T05:35:14.673526+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201852+0000","last_change":"2024-04-25T05:33:58.682492+0000","last_active":"2024-04-25T05:35:15.201852+0000","last_peered":"2024-04-25T05:35:15.201852+0000","last_clean":"2024-04-25T05:35:15.201852+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:35:15.201852+0000","last_undegraded":"2024-04-25T05:35:15.201852+0000","last_fullsized":"2024-04-25T05:35:15.201852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.900372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.829039+0000","last_change":"2024-04-25T05:34:01.000301+0000","last_active":"2024-04-25T05:35:15.829039+0000","last_peered":"2024-04-25T05:35:15.829039+0000","last_clean":"2024-04-25T05:35:15.829039+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:35:15.829039+0000","last_undegraded":"2024-04-25T05:35:15.829039+0000","last_fullsized":"2024-04-25T05:35:15.829039+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:33:56.651479+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:21.243918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202438+0000","last_change":"2024-04-25T05:35:04.205015+0000","last_active":"2024-04-25T05:35:15.202438+0000","last_peered":"2024-04-25T05:35:15.202438+0000","last_clean":"2024-04-25T05:35:15.202438+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:35:15.202438+0000","last_undegraded":"2024-04-25T05:35:15.202438+0000","last_fullsized":"2024-04-25T05:35:15.202438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:04.204911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:04.204911+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:18.863990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.755857+0000","last_change":"2024-04-25T05:34:02.709328+0000","last_active":"2024-04-25T05:35:14.755857+0000","last_peered":"2024-04-25T05:35:14.755857+0000","last_clean":"2024-04-25T05:35:14.755857+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:35:14.755857+0000","last_undegraded":"2024-04-25T05:35:14.755857+0000","last_fullsized":"2024-04-25T05:35:14.755857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.754993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"93'179","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.645595+0000","last_change":"2024-04-25T05:35:09.676826+0000","last_active":"2024-04-25T05:35:17.645595+0000","last_peered":"2024-04-25T05:35:17.645595+0000","last_clean":"2024-04-25T05:35:17.645595+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:35:17.645595+0000","last_undegraded":"2024-04-25T05:35:17.645595+0000","last_fullsized":"2024-04-25T05:35:17.645595+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:25.766235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.674216+0000","last_change":"2024-04-25T05:34:04.759135+0000","last_active":"2024-04-25T05:35:14.674216+0000","last_peered":"2024-04-25T05:35:14.674216+0000","last_clean":"2024-04-25T05:35:14.674216+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:35:14.674216+0000","last_undegraded":"2024-04-25T05:35:14.674216+0000","last_fullsized":"2024-04-25T05:35:14.674216+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673746+0000","last_change":"2024-04-25T05:35:06.043700+0000","last_active":"2024-04-25T05:35:14.673746+0000","last_peered":"2024-04-25T05:35:14.673746+0000","last_clean":"2024-04-25T05:35:14.673746+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:35:14.673746+0000","last_undegraded":"2024-04-25T05:35:14.673746+0000","last_fullsized":"2024-04-25T05:35:14.673746+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.043577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.043577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:15.125915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202178+0000","last_change":"2024-04-25T05:34:00.999939+0000","last_active":"2024-04-25T05:35:15.202178+0000","last_peered":"2024-04-25T05:35:15.202178+0000","last_clean":"2024-04-25T05:35:15.202178+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:35:15.202178+0000","last_undegraded":"2024-04-25T05:35:15.202178+0000","last_fullsized":"2024-04-25T05:35:15.202178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.204950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915741+0000","last_change":"2024-04-25T05:34:02.708443+0000","last_active":"2024-04-25T05:35:14.915741+0000","last_peered":"2024-04-25T05:35:14.915741+0000","last_clean":"2024-04-25T05:35:14.915741+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:35:14.915741+0000","last_undegraded":"2024-04-25T05:35:14.915741+0000","last_fullsized":"2024-04-25T05:35:14.915741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.195589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"93'182","reported_seq":201,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.647148+0000","last_change":"2024-04-25T05:35:09.680036+0000","last_active":"2024-04-25T05:35:17.647148+0000","last_peered":"2024-04-25T05:35:17.647148+0000","last_clean":"2024-04-25T05:35:17.647148+0000","last_became_active":"2024-04-25T05:35:09.679390+0000","last_became_peered":"2024-04-25T05:35:09.679390+0000","last_unstale":"2024-04-25T05:35:17.647148+0000","last_undegraded":"2024-04-25T05:35:17.647148+0000","last_fullsized":"2024-04-25T05:35:17.647148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:24.926774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":182,"num_write_kb":11648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.201495+0000","last_change":"2024-04-25T05:34:04.728674+0000","last_active":"2024-04-25T05:35:15.201495+0000","last_peered":"2024-04-25T05:35:15.201495+0000","last_clean":"2024-04-25T05:35:15.201495+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:35:15.201495+0000","last_undegraded":"2024-04-25T05:35:15.201495+0000","last_fullsized":"2024-04-25T05:35:15.201495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.200256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756716+0000","last_change":"2024-04-25T05:33:58.678744+0000","last_active":"2024-04-25T05:35:14.756716+0000","last_peered":"2024-04-25T05:35:14.756716+0000","last_clean":"2024-04-25T05:35:14.756716+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:35:14.756716+0000","last_undegraded":"2024-04-25T05:35:14.756716+0000","last_fullsized":"2024-04-25T05:35:14.756716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.579714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673199+0000","last_change":"2024-04-25T05:34:00.688978+0000","last_active":"2024-04-25T05:35:14.673199+0000","last_peered":"2024-04-25T05:35:14.673199+0000","last_clean":"2024-04-25T05:35:14.673199+0000","last_became_active":"2024-04-25T05:34:00.688453+0000","last_became_peered":"2024-04-25T05:34:00.688453+0000","last_unstale":"2024-04-25T05:35:14.673199+0000","last_undegraded":"2024-04-25T05:35:14.673199+0000","last_fullsized":"2024-04-25T05:35:14.673199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:29.622999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202423+0000","last_change":"2024-04-25T05:34:02.708185+0000","last_active":"2024-04-25T05:35:15.202423+0000","last_peered":"2024-04-25T05:35:15.202423+0000","last_clean":"2024-04-25T05:35:15.202423+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:35:15.202423+0000","last_undegraded":"2024-04-25T05:35:15.202423+0000","last_fullsized":"2024-04-25T05:35:15.202423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.205057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"93'167","reported_seq":187,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.650697+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:35:17.650697+0000","last_peered":"2024-04-25T05:35:17.650697+0000","last_clean":"2024-04-25T05:35:17.650697+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:35:17.650697+0000","last_undegraded":"2024-04-25T05:35:17.650697+0000","last_fullsized":"2024-04-25T05:35:17.650697+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:22.800666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.729681+0000","last_change":"2024-04-25T05:34:04.761218+0000","last_active":"2024-04-25T05:35:14.729681+0000","last_peered":"2024-04-25T05:35:14.729681+0000","last_clean":"2024-04-25T05:35:14.729681+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:35:14.729681+0000","last_undegraded":"2024-04-25T05:35:14.729681+0000","last_fullsized":"2024-04-25T05:35:14.729681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:15.202441+0000","last_change":"2024-04-25T05:33:58.682409+0000","last_active":"2024-04-25T05:35:15.202441+0000","last_peered":"2024-04-25T05:35:15.202441+0000","last_clean":"2024-04-25T05:35:15.202441+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:35:15.202441+0000","last_undegraded":"2024-04-25T05:35:15.202441+0000","last_fullsized":"2024-04-25T05:35:15.202441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:19.204985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671729+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:35:14.671729+0000","last_peered":"2024-04-25T05:35:14.671729+0000","last_clean":"2024-04-25T05:35:14.671729+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:35:14.671729+0000","last_undegraded":"2024-04-25T05:35:14.671729+0000","last_fullsized":"2024-04-25T05:35:14.671729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.826192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":79,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671274+0000","last_change":"2024-04-25T05:34:02.701774+0000","last_active":"2024-04-25T05:35:14.671274+0000","last_peered":"2024-04-25T05:35:14.671274+0000","last_clean":"2024-04-25T05:35:14.671274+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:35:14.671274+0000","last_undegraded":"2024-04-25T05:35:14.671274+0000","last_fullsized":"2024-04-25T05:35:14.671274+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.378046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"93'167","reported_seq":187,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.639981+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:35:17.639981+0000","last_peered":"2024-04-25T05:35:17.639981+0000","last_clean":"2024-04-25T05:35:17.639981+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:35:17.639981+0000","last_undegraded":"2024-04-25T05:35:17.639981+0000","last_fullsized":"2024-04-25T05:35:17.639981+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:14.874672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671281+0000","last_change":"2024-04-25T05:34:04.718464+0000","last_active":"2024-04-25T05:35:14.671281+0000","last_peered":"2024-04-25T05:35:14.671281+0000","last_clean":"2024-04-25T05:35:14.671281+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:35:14.671281+0000","last_undegraded":"2024-04-25T05:35:14.671281+0000","last_fullsized":"2024-04-25T05:35:14.671281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:20.311184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671785+0000","last_change":"2024-04-25T05:35:06.377537+0000","last_active":"2024-04-25T05:35:14.671785+0000","last_peered":"2024-04-25T05:35:14.671785+0000","last_clean":"2024-04-25T05:35:14.671785+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:35:14.671785+0000","last_undegraded":"2024-04-25T05:35:14.671785+0000","last_fullsized":"2024-04-25T05:35:14.671785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:11.892479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":115,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671169+0000","last_change":"2024-04-25T05:34:00.999824+0000","last_active":"2024-04-25T05:35:14.671169+0000","last_peered":"2024-04-25T05:35:14.671169+0000","last_clean":"2024-04-25T05:35:14.671169+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:35:14.671169+0000","last_undegraded":"2024-04-25T05:35:14.671169+0000","last_fullsized":"2024-04-25T05:35:14.671169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.377946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671180+0000","last_change":"2024-04-25T05:34:02.699472+0000","last_active":"2024-04-25T05:35:14.671180+0000","last_peered":"2024-04-25T05:35:14.671180+0000","last_clean":"2024-04-25T05:35:14.671180+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:35:14.671180+0000","last_undegraded":"2024-04-25T05:35:14.671180+0000","last_fullsized":"2024-04-25T05:35:14.671180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.651777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"93'168","reported_seq":188,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.628168+0000","last_change":"2024-04-25T05:35:09.678603+0000","last_active":"2024-04-25T05:35:17.628168+0000","last_peered":"2024-04-25T05:35:17.628168+0000","last_clean":"2024-04-25T05:35:17.628168+0000","last_became_active":"2024-04-25T05:35:09.678239+0000","last_became_peered":"2024-04-25T05:35:09.678239+0000","last_unstale":"2024-04-25T05:35:17.628168+0000","last_undegraded":"2024-04-25T05:35:17.628168+0000","last_fullsized":"2024-04-25T05:35:17.628168+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:35.043454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11010048,"num_objects":168,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":10752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.756556+0000","last_change":"2024-04-25T05:34:04.724152+0000","last_active":"2024-04-25T05:35:14.756556+0000","last_peered":"2024-04-25T05:35:14.756556+0000","last_clean":"2024-04-25T05:35:14.756556+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:35:14.756556+0000","last_undegraded":"2024-04-25T05:35:14.756556+0000","last_fullsized":"2024-04-25T05:35:14.756556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.785831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672333+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:35:14.672333+0000","last_peered":"2024-04-25T05:35:14.672333+0000","last_clean":"2024-04-25T05:35:14.672333+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:35:14.672333+0000","last_undegraded":"2024-04-25T05:35:14.672333+0000","last_fullsized":"2024-04-25T05:35:14.672333+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.390939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671092+0000","last_change":"2024-04-25T05:33:58.681579+0000","last_active":"2024-04-25T05:35:14.671092+0000","last_peered":"2024-04-25T05:35:14.671092+0000","last_clean":"2024-04-25T05:35:14.671092+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:35:14.671092+0000","last_undegraded":"2024-04-25T05:35:14.671092+0000","last_fullsized":"2024-04-25T05:35:14.671092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:15.378005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672268+0000","last_change":"2024-04-25T05:34:00.691497+0000","last_active":"2024-04-25T05:35:14.672268+0000","last_peered":"2024-04-25T05:35:14.672268+0000","last_clean":"2024-04-25T05:35:14.672268+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:35:14.672268+0000","last_undegraded":"2024-04-25T05:35:14.672268+0000","last_fullsized":"2024-04-25T05:35:14.672268+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:26.049287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"93'185","reported_seq":204,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.657152+0000","last_change":"2024-04-25T05:35:09.684061+0000","last_active":"2024-04-25T05:35:17.657152+0000","last_peered":"2024-04-25T05:35:17.657152+0000","last_clean":"2024-04-25T05:35:17.657152+0000","last_became_active":"2024-04-25T05:35:09.683699+0000","last_became_peered":"2024-04-25T05:35:09.683699+0000","last_unstale":"2024-04-25T05:35:17.657152+0000","last_undegraded":"2024-04-25T05:35:17.657152+0000","last_fullsized":"2024-04-25T05:35:17.657152+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:17.957943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12124160,"num_objects":185,"num_object_clones":0,"num_object_copies":555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":11840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672903+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:35:14.672903+0000","last_peered":"2024-04-25T05:35:14.672903+0000","last_clean":"2024-04-25T05:35:14.672903+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:35:14.672903+0000","last_undegraded":"2024-04-25T05:35:14.672903+0000","last_fullsized":"2024-04-25T05:35:14.672903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:31.872714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.755865+0000","last_change":"2024-04-25T05:34:02.699262+0000","last_active":"2024-04-25T05:35:14.755865+0000","last_peered":"2024-04-25T05:35:14.755865+0000","last_clean":"2024-04-25T05:35:14.755865+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:35:14.755865+0000","last_undegraded":"2024-04-25T05:35:14.755865+0000","last_fullsized":"2024-04-25T05:35:14.755865+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.785814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671271+0000","last_change":"2024-04-25T05:33:58.675050+0000","last_active":"2024-04-25T05:35:14.671271+0000","last_peered":"2024-04-25T05:35:14.671271+0000","last_clean":"2024-04-25T05:35:14.671271+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:35:14.671271+0000","last_undegraded":"2024-04-25T05:35:14.671271+0000","last_fullsized":"2024-04-25T05:35:14.671271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:14.671754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672206+0000","last_change":"2024-04-25T05:34:00.692677+0000","last_active":"2024-04-25T05:35:14.672206+0000","last_peered":"2024-04-25T05:35:14.672206+0000","last_clean":"2024-04-25T05:35:14.672206+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:35:14.672206+0000","last_undegraded":"2024-04-25T05:35:14.672206+0000","last_fullsized":"2024-04-25T05:35:14.672206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"93'160","reported_seq":179,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.649794+0000","last_change":"2024-04-25T05:35:09.679305+0000","last_active":"2024-04-25T05:35:17.649794+0000","last_peered":"2024-04-25T05:35:17.649794+0000","last_clean":"2024-04-25T05:35:17.649794+0000","last_became_active":"2024-04-25T05:35:09.678971+0000","last_became_peered":"2024-04-25T05:35:09.678971+0000","last_unstale":"2024-04-25T05:35:17.649794+0000","last_undegraded":"2024-04-25T05:35:17.649794+0000","last_fullsized":"2024-04-25T05:35:17.649794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:26.867513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":160,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.727096+0000","last_change":"2024-04-25T05:34:04.723087+0000","last_active":"2024-04-25T05:35:14.727096+0000","last_peered":"2024-04-25T05:35:14.727096+0000","last_clean":"2024-04-25T05:35:14.727096+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:35:14.727096+0000","last_undegraded":"2024-04-25T05:35:14.727096+0000","last_fullsized":"2024-04-25T05:35:14.727096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:28.795993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.674003+0000","last_change":"2024-04-25T05:34:02.708656+0000","last_active":"2024-04-25T05:35:14.674003+0000","last_peered":"2024-04-25T05:35:14.674003+0000","last_clean":"2024-04-25T05:35:14.674003+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:35:14.674003+0000","last_undegraded":"2024-04-25T05:35:14.674003+0000","last_fullsized":"2024-04-25T05:35:14.674003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:18.079289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.915641+0000","last_change":"2024-04-25T05:35:02.563427+0000","last_active":"2024-04-25T05:35:14.915641+0000","last_peered":"2024-04-25T05:35:14.915641+0000","last_clean":"2024-04-25T05:35:14.915641+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:35:14.915641+0000","last_undegraded":"2024-04-25T05:35:14.915641+0000","last_fullsized":"2024-04-25T05:35:14.915641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.563344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:05.196036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672199+0000","last_change":"2024-04-25T05:34:00.692475+0000","last_active":"2024-04-25T05:35:14.672199+0000","last_peered":"2024-04-25T05:35:14.672199+0000","last_clean":"2024-04-25T05:35:14.672199+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:35:14.672199+0000","last_undegraded":"2024-04-25T05:35:14.672199+0000","last_fullsized":"2024-04-25T05:35:14.672199+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:17.762524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"93'168","reported_seq":188,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:17.641201+0000","last_change":"2024-04-25T05:35:09.682779+0000","last_active":"2024-04-25T05:35:17.641201+0000","last_peered":"2024-04-25T05:35:17.641201+0000","last_clean":"2024-04-25T05:35:17.641201+0000","last_became_active":"2024-04-25T05:35:09.681887+0000","last_became_peered":"2024-04-25T05:35:09.681887+0000","last_unstale":"2024-04-25T05:35:17.641201+0000","last_undegraded":"2024-04-25T05:35:17.641201+0000","last_fullsized":"2024-04-25T05:35:17.641201+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:36.456435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11010048,"num_objects":168,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":10752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.672418+0000","last_change":"2024-04-25T05:34:04.719644+0000","last_active":"2024-04-25T05:35:14.672418+0000","last_peered":"2024-04-25T05:35:14.672418+0000","last_clean":"2024-04-25T05:35:14.672418+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:35:14.672418+0000","last_undegraded":"2024-04-25T05:35:14.672418+0000","last_fullsized":"2024-04-25T05:35:14.672418+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:22.754141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.726998+0000","last_change":"2024-04-25T05:34:02.705065+0000","last_active":"2024-04-25T05:35:14.726998+0000","last_peered":"2024-04-25T05:35:14.726998+0000","last_clean":"2024-04-25T05:35:14.726998+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:35:14.726998+0000","last_undegraded":"2024-04-25T05:35:14.726998+0000","last_fullsized":"2024-04-25T05:35:14.726998+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:25.937446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.673498+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:35:14.673498+0000","last_peered":"2024-04-25T05:35:14.673498+0000","last_clean":"2024-04-25T05:35:14.673498+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:35:14.673498+0000","last_undegraded":"2024-04-25T05:35:14.673498+0000","last_fullsized":"2024-04-25T05:35:14.673498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T05:35:14.671679+0000","last_change":"2024-04-25T05:34:00.686532+0000","last_active":"2024-04-25T05:35:14.671679+0000","last_peered":"2024-04-25T05:35:14.671679+0000","last_clean":"2024-04-25T05:35:14.671679+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:35:14.671679+0000","last_undegraded":"2024-04-25T05:35:14.671679+0000","last_fullsized":"2024-04-25T05:35:14.671679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:35:23.818346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2761,"num_write_kb":176704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544604160,"data_stored":544604160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2761,"ondisk_log_size":2761,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34776,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93710376,"statfs":{"total":95995035648,"available":95959425024,"internally_reserved":0,"allocated":7938048,"data_stored":7458670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92088,"kb_used_data":56752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93653064,"statfs":{"total":95995035648,"available":95900737536,"internally_reserved":0,"allocated":58114048,"data_stored":57658148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63216,"kb_used_data":27876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93681936,"statfs":{"total":95995035648,"available":95930302464,"internally_reserved":0,"allocated":28545024,"data_stored":28035688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77324,"kb_used_data":46144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93667828,"statfs":{"total":95995035648,"available":95915855872,"internally_reserved":0,"allocated":47251456,"data_stored":46778537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":823296,"data_stored":377872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94160,"kb_used_data":62984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93650992,"statfs":{"total":95995035648,"available":95898615808,"internally_reserved":0,"allocated":64495616,"data_stored":64014058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86864,"kb_used_data":55688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658288,"statfs":{"total":95995035648,"available":95906086912,"internally_reserved":0,"allocated":57024512,"data_stored":56543674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":831488,"data_stored":378506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88997888,"data_stored":88997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68878336,"data_stored":68878336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67043328,"data_stored":67043328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":101908480,"data_stored":101908480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56688640,"data_stored":56688640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":46268416,"data_stored":46268416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":59047936,"data_stored":59047936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":55771136,"data_stored":55771136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:35:24.037 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-04-25T05:35:24.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:24.125 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: osdmap e94: 8 total, 8 up, 8 in 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/745924335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: osdmap e94: 8 total, 8 up, 8 in 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/745924335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:24.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:24.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:24.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:24.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: osdmap e94: 8 total, 8 up, 8 in 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/745924335' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:24.259 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:24.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:24.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:24.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:24.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:24.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:24.722 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 20379 20363 159.07 150.625 0.00293451 0.00627936 2024-04-25T05:35:24.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:24 smithi060 bash[62270]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-25T05:35:24.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:24 smithi060 bash[62270]: Writing manifest to image destination 2024-04-25T05:35:24.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:24.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:24.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:24.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:24.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:25.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:25.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:25.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[27010]: pgmap v45: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:35:25.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:25.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:35:25.105 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[27010]: osdmap e95: 8 total, 8 up, 8 in 2024-04-25T05:35:25.105 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[37492]: pgmap v45: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:35:25.105 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:25.105 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:35:25.105 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:24 smithi043 ceph-mon[37492]: osdmap e95: 8 total, 8 up, 8 in 2024-04-25T05:35:25.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:24 smithi060 ceph-mon[30898]: pgmap v45: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:35:25.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:24 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:25.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:35:25.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:24 smithi060 ceph-mon[30898]: osdmap e95: 8 total, 8 up, 8 in 2024-04-25T05:35:25.137 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 podman[62270]: 2024-04-25 05:35:24.908181796 +0000 UTC m=+2.735521598 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-25T05:35:25.138 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 podman[62270]: 2024-04-25 05:35:25.022548364 +0000 UTC m=+2.849888165 container create 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:35:25.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:25.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:25.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:25.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:25.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:25.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:35:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:25.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:25.499 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:25.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:25.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:25.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:25.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 22568 22552 156.593 136.812 0.00598658 0.00637813 2024-04-25T05:35:25.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 podman[62270]: 2024-04-25 05:35:25.250604739 +0000 UTC m=+3.077944558 container init 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:35:25.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 podman[62270]: 2024-04-25 05:35:25.255829047 +0000 UTC m=+3.083168852 container start 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:35:25.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.262Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-25T05:35:25.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.262Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.263Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.263Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=arp 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-25T05:35:25.750 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=edac 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-25T05:35:25.751 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-25T05:35:25.752 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=os 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=stat 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-25T05:35:25.753 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=time 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=uname 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.264Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.265Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b[62536]: ts=2024-04-25T05:35:25.265Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 bash[62270]: 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 2024-04-25T05:35:25.754 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:35:25 smithi060 systemd[1]: Started Ceph node-exporter.b for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:35:25.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:25.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:25.926 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:25.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:35:26.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:35:26.026 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:35:26.027 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:35:26.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:26.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:26.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:26.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:26.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:35:26.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:35:26.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:35:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:26.294 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:35:26.339 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:35:26.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:26.385 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:35:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:26.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:26.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:35:26.471 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:35:26.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:26.512 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:35:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:26.555 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:35:26.597 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:35:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:35:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:35:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:35:26.632 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:26.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:26.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 24184 24168 151.016 101 0.00312616 0.00661558 2024-04-25T05:35:26.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:26.789 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:26.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:26.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:26.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:26.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:35:27.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.045 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:27.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: pgmap v47: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2088203611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: pgmap v47: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:35:27.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2088203611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:26 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:27.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:27.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: pgmap v47: 148 pgs: 148 active+clean; 969 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2088203611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:27.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:26 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:27.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.324 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:27.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:27.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:27.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:27.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:27.622 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:35:27.622 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:35:27.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:27.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 25975 25959 147.462 111.938 0.00164633 0.00675942 2024-04-25T05:35:27.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:27.844 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:27.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[27010]: 4.f scrub starts 2024-04-25T05:35:27.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[37492]: 4.f scrub starts 2024-04-25T05:35:27.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:27.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:28.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:28.112 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:28.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:28.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:28.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:27 smithi060 ceph-mon[30898]: 4.f scrub starts 2024-04-25T05:35:28.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:27 smithi060 ceph-mon[30898]: 4.f scrub ok 2024-04-25T05:35:28.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:27 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:35:28.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:27 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:35:28.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.374 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:28.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[27010]: 4.f scrub ok 2024-04-25T05:35:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:35:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:35:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[37492]: 4.f scrub ok 2024-04-25T05:35:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:35:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:27 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:35:28.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:28.501 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:28.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:28.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 27696 27680 144.137 107.562 0.00787847 0.00693218 2024-04-25T05:35:28.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:28.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:28.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:28.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:28.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:28.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:29.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:29.037 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:35:29.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-25T05:35:29.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 3.1e scrub starts 2024-04-25T05:35:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 3.1e scrub ok 2024-04-25T05:35:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:35:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:35:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: pgmap v48: 148 pgs: 148 active+clean; 1.1 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 3.1e scrub starts 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 3.1e scrub ok 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:35:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: pgmap v48: 148 pgs: 148 active+clean; 1.1 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:35:29.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:28 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:35:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:29.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 3.1e scrub starts 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 3.1e scrub ok 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: pgmap v48: 148 pgs: 148 active+clean; 1.1 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:35:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:28 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:35:29.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.276 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:29.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:29.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:29.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:29.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:29.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:29.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:29.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:29.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 28990 28974 139.27 80.875 0.00363625 0.0071734 2024-04-25T05:35:29.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:29.782 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:29.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:29.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:29.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:30.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:30.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:35:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:35:30.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:35:30.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:35:30.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:35:30.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:35:30.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:29 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:35:30.177 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:29 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:35:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:30.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:30.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:30.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:30.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:30.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:30.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:30.716 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:30.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 30457 30441 135.871 91.6875 0.0150566 0.00734992 2024-04-25T05:35:30.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:30.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:30.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:30.985 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:30.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:31.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[27010]: pgmap v49: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.89k op/s 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[37492]: pgmap v49: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.89k op/s 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:35:31.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:30 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:35:31.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:30 smithi060 ceph-mon[30898]: pgmap v49: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.89k op/s 2024-04-25T05:35:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:30 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:35:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:30 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:35:31.258 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:31.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:31.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:31.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:31.536 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:31.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:31.661 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:31.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 31901 31885 132.83 90.25 0.00435161 0.00752181 2024-04-25T05:35:31.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:31.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:31.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:31.934 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:31.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:32.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:32.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:35:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:35:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:35:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:35:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:35:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-25T05:35:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:32.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:32.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:32.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:32.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-25T05:35:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:32.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:32.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:32.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:32.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:32.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:32.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 33614 33598 131.219 107.062 0.00850815 0.00761479 2024-04-25T05:35:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:32.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:32.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:32.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:32.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:32.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: pgmap v50: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 579 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: osdmap e96: 8 total, 8 up, 8 in 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:35:32.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:35:32.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:33.015 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:33.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:33.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:33.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:33.284 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: pgmap v50: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 579 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: osdmap e96: 8 total, 8 up, 8 in 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:35:33.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: pgmap v50: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 579 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: osdmap e96: 8 total, 8 up, 8 in 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:35:33.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:32 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:35:33.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:33.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:35:33.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:33.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:33.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:33.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:33.724 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 35245 35229 129.495 101.938 0.011644 0.00771475 2024-04-25T05:35:33.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:33.868 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:33.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:33.941 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:35:33.941 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:35:33.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:34.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:34.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:35:34.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:34.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:34.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:35:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:35:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:35:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:35:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:33 smithi060 ceph-mon[30898]: osdmap e97: 8 total, 8 up, 8 in 2024-04-25T05:35:34.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:35:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:35:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:35:34.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[27010]: osdmap e97: 8 total, 8 up, 8 in 2024-04-25T05:35:34.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:35:34.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:34.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:35:34.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:35:34.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:33 smithi043 ceph-mon[37492]: osdmap e97: 8 total, 8 up, 8 in 2024-04-25T05:35:34.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:34.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:34.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:34.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:34.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:34.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:34.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 15 36779 36764 127.619 95.9375 0.00800507 0.00782967 2024-04-25T05:35:34.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:34.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:34.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:34.999 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:35.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:35.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:35.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:35.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:35:35.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:35:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.25k op/s 2024-04-25T05:35:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:35:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:35:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:35:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:34 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:35:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:35.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:35.316 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:35:35.316 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:35:35.316 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.25k op/s 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.25k op/s 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:35:35.317 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:34 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:35:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:35.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:35.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:35.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:35.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:35.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:35.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 38269 38253 125.8 93.0625 0.01178 0.0079417 2024-04-25T05:35:35.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:35.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:35.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:35.966 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.0 to 1 (10000) 2024-04-25T05:35:35.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:35.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:35.979 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:35:36.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:36.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:36.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:35:36.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:35:36.238 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1284200133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.238 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1284200133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:35 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1284200133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:35 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:35:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:36.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:36.402 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:36.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:36.533 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:36.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.670 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:36.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:36.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:35:36.727553+0000 min lat: 0.00120772 max lat: 0.138412 avg lat: 0.00800449 2024-04-25T05:35:36.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:35:36.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 39974 39958 124.838 106.562 0.00951813 0.00800449 2024-04-25T05:35:36.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:36.800 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:36.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:36.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:36.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:37.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:37.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:37.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:37.207 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: osdmap e98: 8 total, 8 up, 8 in 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 3.18 deep-scrub starts 2024-04-25T05:35:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:36 smithi060 ceph-mon[30898]: 3.18 deep-scrub ok 2024-04-25T05:35:37.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:37.341 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: osdmap e98: 8 total, 8 up, 8 in 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 3.18 deep-scrub starts 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[27010]: 3.18 deep-scrub ok 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:35:37.342 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:35:37.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1385240442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:35:37.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:35:37.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: osdmap e98: 8 total, 8 up, 8 in 2024-04-25T05:35:37.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 3.18 deep-scrub starts 2024-04-25T05:35:37.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:36 smithi043 ceph-mon[37492]: 3.18 deep-scrub ok 2024-04-25T05:35:37.344 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:37.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:37.455 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:35:37.455 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:35:37.477 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:37.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:37.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:37.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 41700 41684 124.029 107.875 0.010337 0.00805732 2024-04-25T05:35:37.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:37.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:37.900 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:37.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:38.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:38.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:38.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:38.168 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:38.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:38.296 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:38.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 3.6 deep-scrub starts 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 3.6 deep-scrub ok 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 4.2 scrub starts 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 4.2 scrub ok 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:35:38.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 3.6 deep-scrub starts 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 3.6 deep-scrub ok 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 4.2 scrub starts 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 4.2 scrub ok 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:35:38.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:38 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:35:38.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:38.424 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:38.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 3.6 deep-scrub starts 2024-04-25T05:35:38.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 3.6 deep-scrub ok 2024-04-25T05:35:38.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 4.2 scrub starts 2024-04-25T05:35:38.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 4.2 scrub ok 2024-04-25T05:35:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:35:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:38 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:35:38.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:38.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:38.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:38.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:38.726 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 43370 43354 123.136 104.375 0.0112999 0.00811502 2024-04-25T05:35:38.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:38.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:38.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:39.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:39.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:35:39.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:35:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:39.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:39.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: pgmap v56: 148 pgs: 2 peering, 146 active+clean; 2.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 0 B/s, 0 objects/s recovering 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 5.d deep-scrub starts 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 5.d deep-scrub ok 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 4.11 scrub starts 2024-04-25T05:35:39.204 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[27010]: 4.11 scrub ok 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: pgmap v56: 148 pgs: 2 peering, 146 active+clean; 2.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 0 B/s, 0 objects/s recovering 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 5.d deep-scrub starts 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 5.d deep-scrub ok 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 4.11 scrub starts 2024-04-25T05:35:39.205 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-mon[37492]: 4.11 scrub ok 2024-04-25T05:35:39.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:39.307 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:39.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:39.432 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:39.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: pgmap v56: 148 pgs: 2 peering, 146 active+clean; 2.2 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 0 B/s, 0 objects/s recovering 2024-04-25T05:35:39.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 5.d deep-scrub starts 2024-04-25T05:35:39.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 5.d deep-scrub ok 2024-04-25T05:35:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:35:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:35:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 4.11 scrub starts 2024-04-25T05:35:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:39 smithi060 ceph-mon[30898]: 4.11 scrub ok 2024-04-25T05:35:39.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:39.563 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:39.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:39.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:39.723 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:39.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 44731 44715 121.475 85.0625 0.00406294 0.00822443 2024-04-25T05:35:39.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:39.869 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:35:39.870 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:39.884 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:35:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:39.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:39.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:39.930 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:35:39.975 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:35:40.019 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:35:40.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:40.052 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:40.061 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:35:40.101 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:35:40.145 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:35:40.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:35:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:35:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:35:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:35:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:35:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:35:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:35:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:35:40.178 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:40.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:40.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 4.7 deep-scrub starts 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 4.7 deep-scrub ok 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: osdmap e99: 8 total, 8 up, 8 in 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2417087344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:35:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 4.7 deep-scrub starts 2024-04-25T05:35:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 4.7 deep-scrub ok 2024-04-25T05:35:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: osdmap e99: 8 total, 8 up, 8 in 2024-04-25T05:35:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2417087344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:35:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:40 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:35:40.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:40.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:35:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:35:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 4.7 deep-scrub starts 2024-04-25T05:35:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 4.7 deep-scrub ok 2024-04-25T05:35:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: osdmap e99: 8 total, 8 up, 8 in 2024-04-25T05:35:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2417087344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:35:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:40 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:35:40.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:40.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:40.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:40.726 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:40.727 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 46068 46052 119.895 83.5625 0.00918947 0.00833487 2024-04-25T05:35:40.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:40.869 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:40.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:40.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:41.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:41.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:41.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:41.208 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:41.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:41.300 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:41.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:41.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: pgmap v57: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 28 B/s, 1 objects/s recovering 2024-04-25T05:35:41.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: osdmap e100: 8 total, 8 up, 8 in 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 4.15 deep-scrub starts 2024-04-25T05:35:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:41 smithi060 ceph-mon[30898]: 4.15 deep-scrub ok 2024-04-25T05:35:41.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: pgmap v57: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 28 B/s, 1 objects/s recovering 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:35:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: osdmap e100: 8 total, 8 up, 8 in 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 4.15 deep-scrub starts 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[27010]: 4.15 deep-scrub ok 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: pgmap v57: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 28 B/s, 1 objects/s recovering 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: osdmap e100: 8 total, 8 up, 8 in 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 4.15 deep-scrub starts 2024-04-25T05:35:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:41 smithi043 ceph-mon[37492]: 4.15 deep-scrub ok 2024-04-25T05:35:41.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:41.700 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:41.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 15 47542 47527 118.787 92.1875 0.00945511 0.00841288 2024-04-25T05:35:41.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:41.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:41.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:41.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:42.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:42.175 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:35:42.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:42.223 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:42.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:42.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2895451546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 4.d scrub starts 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[37492]: 4.d scrub ok 2024-04-25T05:35:42.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2895451546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 4.d scrub starts 2024-04-25T05:35:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:42 smithi043 ceph-mon[27010]: 4.d scrub ok 2024-04-25T05:35:42.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:42.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:42.490 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:35:42.490 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:35:42.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.490 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:48594 7.8 7:1dd46a4c:::benchmark_data_smithi043_74356_object48593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-04-25T05:35:42.490 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:35:42.489478+0000", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.001273731, 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013094809999999999, 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 48594 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489478+0000", 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:42.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489478+0000", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489480+0000", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3599999999999999e-06 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489502+0000", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2101000000000001e-05 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:35:42.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489516+0000", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4259e-05 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489524+0000", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5449999999999998e-06 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489604+0000", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0239000000000002e-05 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489720+0000", 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116224 2024-04-25T05:35:42.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:35:42.489844+0000", 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012409300000000001 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 1 2024-04-25T05:35:42.494 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2895451546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:35:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 4.d scrub starts 2024-04-25T05:35:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-mon[30898]: 4.d scrub ok 2024-04-25T05:35:42.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:42.614 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:42.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:42.729 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 48921 48905 117.53 86.125 0.0203038 0.00850367 2024-04-25T05:35:42.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:42.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:42.882 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:42.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:42.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:42.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:43.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:43.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:43.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:43.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:43.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:43.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:43.402 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:43.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:43 smithi060 ceph-mon[30898]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s; 36 B/s, 2 objects/s recovering 2024-04-25T05:35:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:35:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:43 smithi060 ceph-mon[30898]: osdmap e101: 8 total, 8 up, 8 in 2024-04-25T05:35:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:43 smithi060 ceph-mon[30898]: 3.15 deep-scrub starts 2024-04-25T05:35:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:43 smithi060 ceph-mon[30898]: 3.15 deep-scrub ok 2024-04-25T05:35:43.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:43.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[37492]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s; 36 B/s, 2 objects/s recovering 2024-04-25T05:35:43.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:35:43.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[37492]: osdmap e101: 8 total, 8 up, 8 in 2024-04-25T05:35:43.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[37492]: 3.15 deep-scrub starts 2024-04-25T05:35:43.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[37492]: 3.15 deep-scrub ok 2024-04-25T05:35:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[27010]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s; 36 B/s, 2 objects/s recovering 2024-04-25T05:35:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:35:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[27010]: osdmap e101: 8 total, 8 up, 8 in 2024-04-25T05:35:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[27010]: 3.15 deep-scrub starts 2024-04-25T05:35:43.527 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:43 smithi043 ceph-mon[27010]: 3.15 deep-scrub ok 2024-04-25T05:35:43.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:43.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:35:43.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:43.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:43.729 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 50294 50278 116.356 85.8125 0.0118127 0.00858833 2024-04-25T05:35:43.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:43.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:35:43.862 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:43.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:43.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:44.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:44.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:35:44.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:44.118 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:44.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:35:44.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:35:44.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:35:44.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:35:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:44.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:44.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:44.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:44.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:35:44.491 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:44 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:35:44.503 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:35:44.518 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:35:44.519 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:44.519 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:35:44.519 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:44 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:35:44.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:44.636 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:44.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 15 51461 51446 114.807 73 0.0088883 0.00870487 2024-04-25T05:35:44.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:44.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:44.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:44.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:44.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:45.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:45.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:45.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:45.218 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:45.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:45.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:45.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: pgmap v62: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 138 MiB/s wr, 2.21k op/s; 36 B/s, 1 objects/s recovering 2024-04-25T05:35:45.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 4.4 deep-scrub starts 2024-04-25T05:35:45.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 4.4 deep-scrub ok 2024-04-25T05:35:45.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:35:45.457 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: pgmap v62: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 138 MiB/s wr, 2.21k op/s; 36 B/s, 1 objects/s recovering 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 4.4 deep-scrub starts 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 4.4 deep-scrub ok 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:35:45.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:45 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:35:45.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:45.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: pgmap v62: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 138 MiB/s wr, 2.21k op/s; 36 B/s, 1 objects/s recovering 2024-04-25T05:35:45.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 4.4 deep-scrub starts 2024-04-25T05:35:45.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 4.4 deep-scrub ok 2024-04-25T05:35:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:35:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:35:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:35:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:45 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:35:45.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:45.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:45.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:45.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 52869 52853 113.881 87.9375 0.00720731 0.00877475 2024-04-25T05:35:45.746 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:45.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:45.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:45.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:45.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:46.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:46.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:46.141 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:46.240 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:35:46.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:46.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:35:46.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:35:46.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:35:46.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:35:46.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2995686242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2995686242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.270 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:46.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:46.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:46.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:35:46.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:35:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:35:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:35:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2995686242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:35:46.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:46.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:46.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:46.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:46.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 54364 54348 113.199 93.4375 0.00625287 0.00882914 2024-04-25T05:35:46.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:46.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:46.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:46.962 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:46.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:47.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:47.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:47.160 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:35:47.160 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:35:47.160 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:35:47.160 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:35:47.160 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:35:47.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:47.215 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:47.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[27010]: pgmap v63: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T05:35:47.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:47.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:47.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:47.490 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:47.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:47 smithi060 ceph-mon[30898]: pgmap v63: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T05:35:47.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:47 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:47.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:47 smithi060 ceph-mon[30898]: osdmap e102: 8 total, 8 up, 8 in 2024-04-25T05:35:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:47.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:47.621 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[27010]: osdmap e102: 8 total, 8 up, 8 in 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[37492]: pgmap v63: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[37492]: osdmap e102: 8 total, 8 up, 8 in 2024-04-25T05:35:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:35:47.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:47.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 55813 55797 112.468 90.5625 0.00787491 0.00888618 2024-04-25T05:35:47.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:47.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:47.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:47.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:47.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:48.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:48.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:48.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:48.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:48.301 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:48.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:48.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:48.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:48.681 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:48.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 57329 57313 111.915 94.75 0.009497 0.00893003 2024-04-25T05:35:48.779 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:48.779 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:48.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:35:48.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:48.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:48.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:48.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:49.002 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:49.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:49.129 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:49.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:49.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:49.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:49 smithi043 ceph-mon[27010]: pgmap v65: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:35:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:49 smithi043 ceph-mon[37492]: pgmap v65: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:35:49.404 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:49.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:49 smithi060 ceph-mon[30898]: pgmap v65: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:35:49.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:49.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:49.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:49.728 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 58567 58551 110.868 77.375 0.00691569 0.00901192 2024-04-25T05:35:49.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:49.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:49.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:49.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:49.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:50.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:50.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:35:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:50.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:50.125 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:50.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:50.230 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:50.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.374 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:50.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:50.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:50.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:50.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:50.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:50.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:50.713 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:35:50.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:50.734 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 59709 59693 109.702 71.375 0.00830137 0.00911008 2024-04-25T05:35:50.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:50.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:50.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:50.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:50.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: pgmap v66: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-04-25T05:35:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: osdmap e103: 8 total, 8 up, 8 in 2024-04-25T05:35:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3871556300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:50 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:51.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:51.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: pgmap v66: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-04-25T05:35:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: osdmap e103: 8 total, 8 up, 8 in 2024-04-25T05:35:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3871556300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: pgmap v66: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: osdmap e103: 8 total, 8 up, 8 in 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3871556300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:50 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:35:51.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:51.177 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:51.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:51.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:51.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:51.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:51.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:51.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:51.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:51.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 60891 60875 108.678 73.875 0.0209851 0.00919536 2024-04-25T05:35:51.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:51.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:51.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:51 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:35:51.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:51 smithi043 ceph-mon[27010]: osdmap e104: 8 total, 8 up, 8 in 2024-04-25T05:35:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:51 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:35:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:51 smithi043 ceph-mon[37492]: osdmap e104: 8 total, 8 up, 8 in 2024-04-25T05:35:51.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:51.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:51.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:51.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:35:51.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:51 smithi060 ceph-mon[30898]: osdmap e104: 8 total, 8 up, 8 in 2024-04-25T05:35:52.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:52.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:52.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:52.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:52.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:52.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:52.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:52.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:52.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:52.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:52.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 15 62103 62088 107.765 75.8125 0.012331 0.00927363 2024-04-25T05:35:52.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:52.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:52.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:52.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:52.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:52.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:52.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:52 smithi060 ceph-mon[30898]: pgmap v69: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T05:35:52.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:35:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:35:53.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:53.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:52 smithi043 ceph-mon[27010]: pgmap v69: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T05:35:53.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:52 smithi043 ceph-mon[37492]: pgmap v69: 148 pgs: 148 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-25T05:35:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:53.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:53.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:53.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:53.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:53.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:53.532 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:53.633 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:53.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:53.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 63297 63281 106.868 74.5625 0.00593982 0.00935226 2024-04-25T05:35:53.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:53.794 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:53.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:53.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:53.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:54.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:54.058 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:54.059 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:35:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.060 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:54.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:54.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:54.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:54.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:54.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:54.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:54.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:54.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:54.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:54.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 15 64400 64385 105.872 69 0.0172149 0.00943955 2024-04-25T05:35:54.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:54.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:54.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:54.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:55.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:55.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:55.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:55.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:55.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:55.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[27010]: pgmap v70: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-25T05:35:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[27010]: osdmap e105: 8 total, 8 up, 8 in 2024-04-25T05:35:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[37492]: pgmap v70: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-25T05:35:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:55 smithi043 ceph-mon[37492]: osdmap e105: 8 total, 8 up, 8 in 2024-04-25T05:35:55.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:55.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:55.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:55 smithi060 ceph-mon[30898]: pgmap v70: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-25T05:35:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:55 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:55 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:55 smithi060 ceph-mon[30898]: osdmap e105: 8 total, 8 up, 8 in 2024-04-25T05:35:55.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:55.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:55.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:55.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 65726 65710 105.28 82.8125 0.00883307 0.00949197 2024-04-25T05:35:55.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:55.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:55.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:55.971 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:55.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:56.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:56.101 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:56.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:56.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:56.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:56.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:35:56.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:35:56.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:56.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:35:56.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:35:56.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:56 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:56.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:56 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:35:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:56 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:35:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:56 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:35:56.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:56.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:56.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:35:56.731693+0000 min lat: 0.00120772 max lat: 0.140459 avg lat: 0.00957419 2024-04-25T05:35:56.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:35:56.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 66836 66820 104.383 69.375 0.0116579 0.00957419 2024-04-25T05:35:56.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:35:56.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:56.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:56.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:56.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:57.020 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:57.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:57.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:57.101 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:35:57.146 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:35:57.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:57.195 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:35:57.212 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:57.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:35:57.236 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[37492]: pgmap v72: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[37492]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[37492]: Reconfiguring daemon alertmanager.a on smithi043 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2655545506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[27010]: pgmap v72: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[27010]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[27010]: Reconfiguring daemon alertmanager.a on smithi043 2024-04-25T05:35:57.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:57 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2655545506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:57.250 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:35:57.279 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:35:57.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:57.326 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:35:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:57.368 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:35:57.419 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:35:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:35:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:35:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:35:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:35:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:35:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:35:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:35:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:35:57.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:57.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:57 smithi060 ceph-mon[30898]: pgmap v72: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-25T05:35:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:57 smithi060 ceph-mon[30898]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-25T05:35:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:57 smithi060 ceph-mon[30898]: Reconfiguring daemon alertmanager.a on smithi043 2024-04-25T05:35:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:57 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2655545506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:35:57.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:57.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:57.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:57.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 67875 67859 103.42 64.9375 0.00608301 0.00966313 2024-04-25T05:35:57.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:57.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:57.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:57.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:57.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:58.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:58.111 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:35:58.140 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:35:58.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:35:58.266 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:35:58.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:58.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:58.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:58.534 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:58.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:58.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:58.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 69209 69193 102.943 83.375 0.00968872 0.00970818 2024-04-25T05:35:58.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:58.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:58.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:58.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:59.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:59.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:59.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:35:59.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:35:59.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:35:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[27010]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 927 op/s 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[27010]: osdmap e106: 8 total, 8 up, 8 in 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[37492]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 927 op/s 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:35:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-mon[37492]: osdmap e106: 8 total, 8 up, 8 in 2024-04-25T05:35:59.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:35:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:59.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:35:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:59 smithi060 ceph-mon[30898]: pgmap v73: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 58 MiB/s wr, 927 op/s 2024-04-25T05:35:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:59 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:35:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:59 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:35:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:35:59 smithi060 ceph-mon[30898]: osdmap e106: 8 total, 8 up, 8 in 2024-04-25T05:35:59.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:35:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:35:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:35:59.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:35:59.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 70422 70406 102.312 75.8125 0.0221002 0.00976902 2024-04-25T05:35:59.751 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:35:59.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:35:59.883 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:35:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:35:59.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:35:59.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:35:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:35:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:35:59.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:00.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:00.149 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:00.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:00.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:00.429 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:00.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:00.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:00.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:00.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:00.730 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 71641 71625 101.718 76.1875 0.0185504 0.00982534 2024-04-25T05:36:00.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:00.822 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:00.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:00 smithi043 ceph-mon[37492]: pgmap v75: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:36:00.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:00 smithi043 ceph-mon[37492]: osdmap e107: 8 total, 8 up, 8 in 2024-04-25T05:36:00.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:00 smithi043 ceph-mon[27010]: pgmap v75: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:36:00.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:00 smithi043 ceph-mon[27010]: osdmap e107: 8 total, 8 up, 8 in 2024-04-25T05:36:00.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:00.951 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:00.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:00.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:00 smithi060 ceph-mon[30898]: pgmap v75: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:36:00.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:00 smithi060 ceph-mon[30898]: osdmap e107: 8 total, 8 up, 8 in 2024-04-25T05:36:01.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:01.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:01.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:36:01.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:01.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:01.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:01.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:01.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:01.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:01.669 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:01.715 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:36:01.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 15 72807 72792 101.078 72.9375 0.0141838 0.00988718 2024-04-25T05:36:01.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:01.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:01.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:01.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:01.962 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:01.962 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:01.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:02.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:02.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:02.156 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:02.156 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:02.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:02.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:02.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:02.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:02.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:02.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:02.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:02.613 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:02.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:02.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 15 73926 73911 100.401 69.9375 0.00744209 0.00995493 2024-04-25T05:36:02.744 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:02.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:02.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:02.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:02.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[27010]: pgmap v77: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-25T05:36:02.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:02.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:02.964 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[27010]: osdmap e108: 8 total, 8 up, 8 in 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[37492]: pgmap v77: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[37492]: osdmap e108: 8 total, 8 up, 8 in 2024-04-25T05:36:02.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:02.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:02.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:02.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-mon[30898]: pgmap v77: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 731 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-25T05:36:02.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3982718692' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-mon[30898]: osdmap e108: 8 total, 8 up, 8 in 2024-04-25T05:36:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:03.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:03.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:03.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:03.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:03.273 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:03.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:03.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:03.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:03.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:03.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:03.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:03.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 75178 75162 99.928 78.1875 0.0182673 0.0100023 2024-04-25T05:36:03.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:03.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:03.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:03.955 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:03.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:03.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:04.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:04.053 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:04.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:04.081 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:04.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:04.212 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:04.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:04.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:04.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:04 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:36:04.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:04.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:04.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:04.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:04.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:04.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 76391 76375 99.4255 75.8125 0.00867166 0.0100518 2024-04-25T05:36:04.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:04.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:04.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:36:04.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:04.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:04.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:04.992 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:04 smithi043 ceph-mon[27010]: pgmap v79: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-25T05:36:04.992 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:04 smithi043 ceph-mon[27010]: osdmap e109: 8 total, 8 up, 8 in 2024-04-25T05:36:04.992 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:04 smithi043 ceph-mon[37492]: pgmap v79: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-25T05:36:04.992 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:04 smithi043 ceph-mon[37492]: osdmap e109: 8 total, 8 up, 8 in 2024-04-25T05:36:04.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:04 smithi060 ceph-mon[30898]: pgmap v79: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-25T05:36:04.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:04 smithi060 ceph-mon[30898]: osdmap e109: 8 total, 8 up, 8 in 2024-04-25T05:36:05.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:05.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:05.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:05.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:05.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:05.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:05.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:36:05.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:05.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:05.649 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:05.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:05.727 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:05.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 77291 77275 98.5444 56.25 0.00860552 0.0101388 2024-04-25T05:36:05.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:05.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:05.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:05.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:06.011 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:06.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:06.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:06.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:06.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:06.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:06.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:06.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:06.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:06.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:06.684 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:06.731 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 78321 78305 97.861 64.375 0.00728921 0.0102126 2024-04-25T05:36:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:36:06.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:06.809 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:06 smithi043 ceph-mon[27010]: pgmap v81: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T05:36:06.809 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:36:06.810 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:06 smithi043 ceph-mon[37492]: pgmap v81: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T05:36:06.810 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:36:06.810 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:06 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[72032]: ts=2024-04-25T05:36:06.523Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-25T05:36:06.810 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:06 smithi043 podman[80065]: 2024-04-25 05:36:06.564500961 +0000 UTC m=+1.010466457 container died 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:06.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:06.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:06.958 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:06.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:06.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:06 smithi060 ceph-mon[30898]: pgmap v81: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-25T05:36:06.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:36:07.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:07.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:07.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:07.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:07.379 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:07 smithi043 podman[80065]: 2024-04-25 05:36:07.128392733 +0000 UTC m=+1.574358229 container cleanup 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:07.379 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:07 smithi043 bash[80065]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a 2024-04-25T05:36:07.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:07.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:07.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:07.608 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:07.641 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:07 smithi043 podman[80192]: 2024-04-25 05:36:07.379869877 +0000 UTC m=+0.803886727 container remove 853778a954df2a5301dfeb6c999403c5e148c0830fe3669a0bf2c32d845fe6ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:07.641 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:07 smithi043 podman[80192]: 2024-04-25 05:36:07.482285586 +0000 UTC m=+0.906302441 volume remove 08a8265d3f42bb1b1fd42f845c8317f1d0812a7d76166ed3124d525d04315e2a 2024-04-25T05:36:07.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:07.732 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 79523 79507 97.4147 75.125 0.00746161 0.0102599 2024-04-25T05:36:07.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:07.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:07.873 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:07.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:07.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:08.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:08.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:08.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:36:08.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:07 smithi043 ceph-mon[27010]: osdmap e110: 8 total, 8 up, 8 in 2024-04-25T05:36:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:36:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:07 smithi043 ceph-mon[37492]: osdmap e110: 8 total, 8 up, 8 in 2024-04-25T05:36:08.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:08.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2323888882' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:36:08.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:07 smithi060 ceph-mon[30898]: osdmap e110: 8 total, 8 up, 8 in 2024-04-25T05:36:08.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:08.283 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:08.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:36:08.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:08.413 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:08.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:08.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:08.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:08.736 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 15 80602 80587 96.8312 67.5 0.0195987 0.0103219 2024-04-25T05:36:08.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:08.848 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:08 smithi043 ceph-mon[27010]: pgmap v83: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:36:08.849 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:08 smithi043 ceph-mon[37492]: pgmap v83: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:36:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:08.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:08.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:09.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:09.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:09.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:09.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:09.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:08 smithi060 ceph-mon[30898]: pgmap v83: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-25T05:36:09.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:09.320 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:09.354 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:09.354 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:09 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-25T05:36:09.354 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:09 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:36:09.354 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:09 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.244s CPU time. 2024-04-25T05:36:09.354 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:09 smithi043 systemd[1]: Starting Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:36:09.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:09.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:09.399 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:09.439 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:09.479 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:09.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:09.522 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:09.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:09.568 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:09.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:09.624 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:09.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:09.649 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:09.660 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:09.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:09.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:09.737 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 81540 81524 96.1086 58.5625 0.011486 0.0103988 2024-04-25T05:36:09.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:09.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:09.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2810370439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:09.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:09 smithi043 ceph-mon[27010]: osdmap e111: 8 total, 8 up, 8 in 2024-04-25T05:36:09.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:36:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:36:09.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2810370439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:09.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:09 smithi043 ceph-mon[37492]: osdmap e111: 8 total, 8 up, 8 in 2024-04-25T05:36:09.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:09.962 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:09.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:10.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:10.091 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:10.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:10.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2810370439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:10.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:09 smithi060 ceph-mon[30898]: osdmap e111: 8 total, 8 up, 8 in 2024-04-25T05:36:10.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:10.348 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:10.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:10.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:10.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:10.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:10.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:10.737 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 15 82714 82699 95.6876 73.4375 0.00851016 0.0104456 2024-04-25T05:36:10.745 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:10.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:10.874 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:10.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:10 smithi043 ceph-mon[27010]: pgmap v84: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:36:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:10 smithi043 ceph-mon[37492]: pgmap v84: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:36:10.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:11.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:11.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:11.141 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:11.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:36:11.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:11.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:10 smithi060 ceph-mon[30898]: pgmap v84: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:36:11.267 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:11.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:11.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:11.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:11.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:11.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:11.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:11.738 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 83788 83772 95.1671 67.0625 0.0117949 0.0105016 2024-04-25T05:36:11.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:11.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:11.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:11.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.038 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:12.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:12.084 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:11 smithi043 podman[80684]: 2024-04-25 05:36:11.642377403 +0000 UTC m=+0.242432286 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-25T05:36:12.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:12.165 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:12.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:12.292 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:12.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:12.440 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:12.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:12.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:12.641 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 podman[80684]: 2024-04-25 05:36:12.346971021 +0000 UTC m=+0.947025892 volume create ee9d1421eb72f435a1b025a94dd29b669bbc2882c34d5f035ed729700dbc624e 2024-04-25T05:36:12.641 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 podman[80684]: 2024-04-25 05:36:12.445673791 +0000 UTC m=+1.045728669 container create a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:12.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:12.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:36:12.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:12.738 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 84680 84664 94.4623 55.75 0.0151613 0.0105793 2024-04-25T05:36:12.792 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:36:12.797 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:36:12.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:12.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:12.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:12.962 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:12.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:13.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:13.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[27010]: pgmap v86: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 732 B/s rd, 50 MiB/s wr, 801 op/s 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2232239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3967874970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[37492]: pgmap v86: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 732 B/s rd, 50 MiB/s wr, 801 op/s 2024-04-25T05:36:13.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2232239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3967874970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 podman[80684]: 2024-04-25 05:36:12.808257127 +0000 UTC m=+1.408312005 container init a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 podman[80684]: 2024-04-25 05:36:12.823035471 +0000 UTC m=+1.423090351 container start a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.847Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.848Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.849Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.43 port=9094 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.852Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.914Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.915Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-25T05:36:13.142 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.919Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-25T05:36:13.143 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:12 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:12.919Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-25T05:36:13.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:13.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:13.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-mon[30898]: pgmap v86: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 732 B/s rd, 50 MiB/s wr, 801 op/s 2024-04-25T05:36:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2232239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3967874970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:36:13.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:13.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:13.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:13.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:13.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:13.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:13.738 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 85555 85539 93.7645 54.6875 0.0137414 0.0106595 2024-04-25T05:36:13.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:13.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:13.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:13.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:13.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:14.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: osdmap e112: 8 total, 8 up, 8 in 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:36:14.069 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:36:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:14.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: osdmap e112: 8 total, 8 up, 8 in 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:36:14.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:13 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: osdmap e112: 8 total, 8 up, 8 in 2024-04-25T05:36:14.149 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:36:14.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:13 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:36:14.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:14.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:14.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:14.323 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:14.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:14.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:14 smithi043 bash[80684]: a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 2024-04-25T05:36:14.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:14 smithi043 systemd[1]: Started Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:36:14.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:14.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:14.646 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:14.739 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 86488 86472 93.1533 58.3125 0.0184839 0.0107298 2024-04-25T05:36:14.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:14.778 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:14.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:14.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:14.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:15.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:36:15.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:15.049 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:15.113 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:14 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:14.854Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.0019722s 2024-04-25T05:36:15.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:15.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:15.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.346 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: pgmap v88: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:36:15.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[27010]: osdmap e113: 8 total, 8 up, 8 in 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: pgmap v88: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:36:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:15 smithi043 ceph-mon[37492]: osdmap e113: 8 total, 8 up, 8 in 2024-04-25T05:36:15.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:15.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: pgmap v88: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-04-25T05:36:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:36:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:36:15.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:36:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:15 smithi060 ceph-mon[30898]: osdmap e113: 8 total, 8 up, 8 in 2024-04-25T05:36:15.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:15.606 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:15.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:15.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 87579 87563 92.7249 68.1875 0.00754646 0.0107797 2024-04-25T05:36:15.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:15.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:15.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:15.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:16.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:16.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:16.137 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:16.214 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[27010]: Reconfiguring daemon prometheus.a on smithi060 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[27010]: 4.1e scrub starts 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[27010]: 4.1e scrub ok 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[37492]: Reconfiguring daemon prometheus.a on smithi060 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[37492]: 4.1e scrub starts 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[37492]: 4.1e scrub ok 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:36:16.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:16 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:36:16.229 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:16.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:16.263 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:16.306 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:16.353 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:36:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:36:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:36:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:36:16.353 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:36:16.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:16.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:16 smithi060 ceph-mon[30898]: Reconfiguring daemon prometheus.a on smithi060 2024-04-25T05:36:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:16 smithi060 ceph-mon[30898]: 4.1e scrub starts 2024-04-25T05:36:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:16 smithi060 ceph-mon[30898]: 4.1e scrub ok 2024-04-25T05:36:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:16 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:36:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:16 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:36:16.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:16.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:16.680 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:16.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:36:16.743362+0000 min lat: 0.00120772 max lat: 0.140459 avg lat: 0.0108372 2024-04-25T05:36:16.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:36:16.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 88590 88574 92.2327 63.1875 0.0252976 0.0108372 2024-04-25T05:36:16.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:16.807 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:16.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:16.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:16.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:17.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:17.085 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:17.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:17.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:17.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: pgmap v90: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 862 op/s 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 7.b scrub starts 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:36:17.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: pgmap v90: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 862 op/s 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 7.b scrub starts 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:36:17.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:17 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:36:17.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: pgmap v90: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 54 MiB/s wr, 862 op/s 2024-04-25T05:36:17.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:36:17.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 7.b scrub starts 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:36:17.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:17 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:36:17.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:36:17.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:17.492 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:17.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:17.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:17.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:36:17.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:17.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 89578 89562 91.733 61.75 0.0204131 0.0108947 2024-04-25T05:36:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:17.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:17.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:17.912 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:17.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:18.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:18.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:18.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:36:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:18 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:36:18.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:18.327 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:36:18.369 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:36:18.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:18.459 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:18.508 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:18.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:18.524 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:18.569 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:18.580 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:18.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:18.625 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:18.651 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:18.666 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:18.712 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:18.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 90396 90380 91.0781 51.125 0.0118703 0.0109744 2024-04-25T05:36:18.748 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 systemd[1]: Stopping Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:36:18.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:18.755 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:18.778 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:18.795 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:18.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:18.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:18.803 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:18.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:18.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:18.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:18.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:19.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:19.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:19.182 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-25T05:36:19.182 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-25T05:36:19.182 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.852Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.853Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.853Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.853Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[58884]: ts=2024-04-25T05:36:18.853Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-25T05:36:19.183 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:18 smithi060 podman[66762]: 2024-04-25 05:36:18.861326553 +0000 UTC m=+0.136778618 container died be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:19.185 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:36:19.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:19.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:19.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:19.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: pgmap v91: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 762 op/s 2024-04-25T05:36:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:36:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4289048428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: 7.b scrub ok 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/35091496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: pgmap v91: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 762 op/s 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:36:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4289048428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: 7.b scrub ok 2024-04-25T05:36:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:36:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:36:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/35091496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:19.490 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:19.498 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 podman[66762]: 2024-04-25 05:36:19.292395074 +0000 UTC m=+0.567847151 container cleanup be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:19.498 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 bash[66762]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: pgmap v91: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 762 op/s 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4289048428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: 7.b scrub ok 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/35091496' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:36:19.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:19.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:19.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:19.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 91460 91444 90.688 66.5 0.0182148 0.0110216 2024-04-25T05:36:19.751 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:19.825 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 podman[66793]: 2024-04-25 05:36:19.523810388 +0000 UTC m=+0.661464006 container remove be53dfb52167be27ed6645ed0fe7ca6345eb344cfaf469ee6a24d436668371ab (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:19.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:19.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:19.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:19.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:20.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:20.098 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-25T05:36:20.098 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 systemd[1]: Stopped Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:36:20.098 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.338s CPU time. 2024-04-25T05:36:20.098 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:19 smithi060 systemd[1]: Starting Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:36:20.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:20.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:20.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:36:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[27010]: osdmap e114: 8 total, 8 up, 8 in 2024-04-25T05:36:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[27010]: osdmap e115: 8 total, 8 up, 8 in 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[37492]: osdmap e114: 8 total, 8 up, 8 in 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-mon[37492]: osdmap e115: 8 total, 8 up, 8 in 2024-04-25T05:36:20.392 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:36:20.393 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-mon[30898]: osdmap e114: 8 total, 8 up, 8 in 2024-04-25T05:36:20.393 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:36:20.393 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:36:20.393 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-mon[30898]: osdmap e115: 8 total, 8 up, 8 in 2024-04-25T05:36:20.393 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 podman[66946]: 2024-04-25 05:36:20.008590504 +0000 UTC m=+0.030803563 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-25T05:36:20.393 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 podman[66946]: 2024-04-25 05:36:20.146140886 +0000 UTC m=+0.168353957 container create 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:20.393 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 podman[66946]: 2024-04-25 05:36:20.392356859 +0000 UTC m=+0.414569926 container init 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:20.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:20.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:20.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:20.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:20.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:20.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 92452 92436 90.2398 62 0.0149834 0.0110749 2024-04-25T05:36:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 podman[66946]: 2024-04-25 05:36:20.39803947 +0000 UTC m=+0.420252537 container start 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:36:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.433Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-25T05:36:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.433Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-25T05:36:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.433Z caller=main.go:570 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi060 (none))" 2024-04-25T05:36:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.433Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.433Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.435Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.436Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.437Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.437Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.440Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.440Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.555µs 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.440Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.441Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.441Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.441Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=36.425µs wal_replay_duration=939.392µs wbl_replay_duration=144ns total_replay_duration=1.05722ms 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.442Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-25T05:36:20.750 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.442Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.442Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.472Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=30.265851ms db_storage=799ns remote_storage=1.113µs web_handler=531ns query_engine=709ns scrape=463.105µs scrape_sd=135.595µs notify=11.086µs notify_sd=9.347µs rules=29.139266ms tracing=8.644µs 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.472Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:36:20.473Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 bash[66946]: 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a 2024-04-25T05:36:20.751 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:36:20 smithi060 systemd[1]: Started Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:36:20.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:20.869 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:20.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPING 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPED 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STARTING 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Serving on http://:::9283 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STARTED 2024-04-25T05:36:20.921 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPING 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPED 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STARTING 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Serving on http://:::9283 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STARTED 2024-04-25T05:36:20.922 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPING 2024-04-25T05:36:20.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:21.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:21.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:21.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:21.207 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:36:21.207 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STOPPED 2024-04-25T05:36:21.207 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:20 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:20] ENGINE Bus STARTING 2024-04-25T05:36:21.207 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:21] ENGINE Serving on http://:::9283 2024-04-25T05:36:21.207 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:36:21] ENGINE Bus STARTED 2024-04-25T05:36:21.211 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:36:21.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:21.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:21.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:21.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 740 B/s rd, 44 MiB/s wr, 709 op/s 2024-04-25T05:36:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:36:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:36:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 4.16 scrub starts 2024-04-25T05:36:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 4.16 scrub ok 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3332824084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:36:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:36:21.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:21.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:36:21.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:21 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:36:21.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:21.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 740 B/s rd, 44 MiB/s wr, 709 op/s 2024-04-25T05:36:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 4.16 scrub starts 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 4.16 scrub ok 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3332824084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:36:21.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 740 B/s rd, 44 MiB/s wr, 709 op/s 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 4.16 scrub starts 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 4.16 scrub ok 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3332824084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:21.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:36:21.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:21 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:36:21.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:21.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:21.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 93367 93351 89.7313 57.1875 0.0150347 0.0111399 2024-04-25T05:36:21.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:21.856 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:21.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:21.992 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:21.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:22.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.123 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:22.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:22.141 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:22.141 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:22.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:22.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:22.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:22.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: osdmap e116: 8 total, 8 up, 8 in 2024-04-25T05:36:22.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:36:22.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:36:22.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:36:22.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:22.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: osdmap e116: 8 total, 8 up, 8 in 2024-04-25T05:36:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi043:9093"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi060.front.sepia.ceph.com:9095"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi060.front.sepia.ceph.com:3000"}]: dispatch 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: osdmap e116: 8 total, 8 up, 8 in 2024-04-25T05:36:22.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:36:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:36:22.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:36:22.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:22.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:22.742 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 94463 94447 89.4096 68.5 0.00687142 0.0111795 2024-04-25T05:36:22.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.823 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:22.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:22.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:36:22.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:22.941 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:36:22 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:36:22.857Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.005439504s 2024-04-25T05:36:22.954 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:22.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:23.033 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:23.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:23.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:23.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:23.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:23.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:23.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:23 smithi060 ceph-mon[30898]: pgmap v96: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 816 op/s 2024-04-25T05:36:23.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:23 smithi060 ceph-mon[30898]: 5.2 deep-scrub starts 2024-04-25T05:36:23.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:23 smithi060 ceph-mon[30898]: 5.2 deep-scrub ok 2024-04-25T05:36:23.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:23 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:36:23.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:23 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:36:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:23.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:23.531 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[27010]: pgmap v96: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 816 op/s 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[27010]: 5.2 deep-scrub starts 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[27010]: 5.2 deep-scrub ok 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[37492]: pgmap v96: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 816 op/s 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[37492]: 5.2 deep-scrub starts 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[37492]: 5.2 deep-scrub ok 2024-04-25T05:36:23.610 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:36:23.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:23 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:36:23.633 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:23.658 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:23.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:36:23.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 95527 95511 89.0675 66.5 0.00490015 0.011222 2024-04-25T05:36:23.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.798 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:23.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:23.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:23.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:23.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:24.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:24.059 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:24.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:24.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:24.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:24.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[27010]: 5.16 deep-scrub starts 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[27010]: 5.16 deep-scrub ok 2024-04-25T05:36:24.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:36:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:36:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[37492]: 5.16 deep-scrub starts 2024-04-25T05:36:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:24 smithi043 ceph-mon[37492]: 5.16 deep-scrub ok 2024-04-25T05:36:24.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:24.463 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:24 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:36:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:24 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:36:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:24 smithi060 ceph-mon[30898]: 5.16 deep-scrub starts 2024-04-25T05:36:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:24 smithi060 ceph-mon[30898]: 5.16 deep-scrub ok 2024-04-25T05:36:24.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:24.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:24.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:24.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:24.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 96586 96570 88.731 66.1875 0.0152357 0.0112638 2024-04-25T05:36:24.766 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:24.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:24.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:36:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:24.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:24.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:25.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:25.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:25.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:25.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:25.409 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:25.409 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:25.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: pgmap v97: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-04-25T05:36:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:36:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:36:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: 6.17 scrub starts 2024-04-25T05:36:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: 6.17 scrub ok 2024-04-25T05:36:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:25 smithi060 ceph-mon[30898]: osdmap e117: 8 total, 8 up, 8 in 2024-04-25T05:36:25.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:25.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:25.640 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:25.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: pgmap v97: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-04-25T05:36:25.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:36:25.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:36:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: 6.17 scrub starts 2024-04-25T05:36:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: 6.17 scrub ok 2024-04-25T05:36:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[27010]: osdmap e117: 8 total, 8 up, 8 in 2024-04-25T05:36:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: pgmap v97: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-04-25T05:36:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:36:25.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:36:25.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: 6.17 scrub starts 2024-04-25T05:36:25.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: 6.17 scrub ok 2024-04-25T05:36:25.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:25 smithi043 ceph-mon[37492]: osdmap e117: 8 total, 8 up, 8 in 2024-04-25T05:36:25.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:25.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 97590 97574 88.3545 62.75 0.0347082 0.0113127 2024-04-25T05:36:25.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:25.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:25.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:25.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:26.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:26.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:26.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.225 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:26.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:26.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:26.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:26 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:36:26.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:26 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:36:26.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:26 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:36:26.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:26 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:36:26.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:26.437 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:26.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:26.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:26.492 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:26.539 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:26.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:26.594 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:26.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:26.634 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:26.677 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:26.718 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:26.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:26.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 98517 98501 87.92 57.9375 0.0188572 0.0113689 2024-04-25T05:36:26.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:26 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:36:26.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:26 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:36:26.753 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:26.768 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:26.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:26.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:26.889 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:26.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:26.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:27.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:27.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:27.154 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:27.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:27.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: pgmap v99: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3013447114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:27.337 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:36:27.338 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:36:27.338 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 6.7 scrub starts 2024-04-25T05:36:27.338 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:27 smithi060 ceph-mon[30898]: 6.7 scrub ok 2024-04-25T05:36:27.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:36:27.386 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:27.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:27.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:27.589 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: pgmap v99: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3013447114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:36:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 6.7 scrub starts 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[27010]: 6.7 scrub ok 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: pgmap v99: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3013447114' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 6.7 scrub starts 2024-04-25T05:36:27.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:27 smithi043 ceph-mon[37492]: 6.7 scrub ok 2024-04-25T05:36:27.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:27.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:27.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 99491 99475 87.5391 60.875 0.0194691 0.0114182 2024-04-25T05:36:27.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:36:27.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:27.857 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:27.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:27.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:27.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:28.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:28.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:28.173 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:28.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:28.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:28.277 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:36:28.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:28.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:28.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.521 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:28.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:28.561 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:36:28.561 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:36:28.561 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:36:28.561 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:36:28.561 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:36:28.562 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: pgmap v100: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 728 op/s 2024-04-25T05:36:28.562 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.562 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:28 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:36:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:36:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: pgmap v100: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 728 op/s 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: pgmap v100: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 728 op/s 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:28 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:28.654 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:28.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 100430 100414 87.1383 58.6875 0.0030777 0.0114683 2024-04-25T05:36:28.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:28.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:28.865 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:28.865 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:36:28.865 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:36:28.866 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:36:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:28.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:28.931 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:28.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:29.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:29.053 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:29.155 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:29.184 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:29.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:29.312 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:29.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:29.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:29.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:29.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: osdmap e118: 8 total, 8 up, 8 in 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: 6.e deep-scrub starts 2024-04-25T05:36:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[27010]: 6.e deep-scrub ok 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: osdmap e118: 8 total, 8 up, 8 in 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: 6.e deep-scrub starts 2024-04-25T05:36:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-mon[37492]: 6.e deep-scrub ok 2024-04-25T05:36:29.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:29.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:29.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 101390 101374 86.7666 60 0.0258281 0.0115197 2024-04-25T05:36:29.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3096536618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:36:29.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: osdmap e118: 8 total, 8 up, 8 in 2024-04-25T05:36:29.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:36:29.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:36:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: 6.e deep-scrub starts 2024-04-25T05:36:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:29 smithi060 ceph-mon[30898]: 6.e deep-scrub ok 2024-04-25T05:36:29.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:29.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:29.994 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:29.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:30.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:30.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:36:30.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:36:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:36:30.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:30.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:30.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:30.464 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:30.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:30.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:30.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:30.743 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 102284 102268 86.3492 55.875 0.0111307 0.011575 2024-04-25T05:36:30.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 6.6 scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 6.6 scrub ok 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 5.9 deep-scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 5.9 deep-scrub ok 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 7.8 scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: pgmap v102: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: osdmap e119: 8 total, 8 up, 8 in 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:36:30.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 6.6 scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 6.6 scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 5.9 deep-scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 5.9 deep-scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 7.8 scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: pgmap v102: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: osdmap e119: 8 total, 8 up, 8 in 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:36:30.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:36:30.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:30 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:36:30.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:30.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:30.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 6.6 scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 6.6 scrub ok 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 5.9 deep-scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 5.9 deep-scrub ok 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 7.8 scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: pgmap v102: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: osdmap e119: 8 total, 8 up, 8 in 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:36:30.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:30 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:36:31.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:31.105 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:36:31.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:31.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:11484 7.1 7:8a37f08d:::benchmark_data_smithi043_74356_object11483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:20.936399+0000", 2024-04-25T05:36:31.241 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.302595987000004, 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035740128000000003, 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 11484 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.936399+0000", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.936399+0000", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.936400+0000", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.359e-06 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951214+0000", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0148135 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951227+0000", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3057000000000001e-05 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951235+0000", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5569999999999998e-06 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951323+0000", 2024-04-25T05:36:31.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8605999999999998e-05 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951474+0000", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015052800000000001 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.951639+0000", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000165725 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.952921+0000", 2024-04-25T05:36:31.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001282056 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.954480+0000", 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015589180000000001 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.972037+0000", 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.017556252000000001 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.972093+0000", 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6595000000000003e-05 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:20.972139+0000", 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5974999999999999e-05 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:12217 7.1 7:8cffd148:::benchmark_data_smithi043_74356_object12216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:21.266163+0000", 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 69.972832521000001, 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.038703834999999999, 2024-04-25T05:36:31.247 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12217 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266163+0000", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266163+0000", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266163+0000", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.75e-07 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266199+0000", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5840000000000002e-05 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266212+0000", 2024-04-25T05:36:31.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2944e-05 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.266219+0000", 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4719999999999998e-06 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.267157+0000", 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093743799999999996 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.267265+0000", 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108132 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.267415+0000", 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000149751 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.267899+0000", 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048420100000000002 2024-04-25T05:36:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268238+0000", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033868 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.304755+0000", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.036517079000000001 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.304819+0000", 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4088e-05 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.304866+0000", 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7635e-05 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:12220 7.1 7:80820143:::benchmark_data_smithi043_74356_object12219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:21.268676+0000", 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 69.970319095999997, 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.036327146999999997, 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12220 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268676+0000", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268676+0000", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268677+0000", 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5199999999999995e-07 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268734+0000", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6931000000000001e-05 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268746+0000", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2629000000000001e-05 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268755+0000", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4940000000000003e-06 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268810+0000", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5594000000000003e-05 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.268915+0000", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010453799999999999 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.269077+0000", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000161559 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.269613+0000", 2024-04-25T05:36:31.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053679400000000005 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.269995+0000", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038171400000000001 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.304910+0000", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.034914894000000002 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.304957+0000", 2024-04-25T05:36:31.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7413999999999997e-05 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:21.305003+0000", 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5734e-05 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:45786 7.1 7:893fc6ca:::benchmark_data_smithi043_74356_object45785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:40.467102+0000", 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 50.771892909999998, 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073939332999999996, 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.259 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 45786 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467102+0000", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467102+0000", 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467103+0000", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.356e-06 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467158+0000", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4839999999999997e-05 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467174+0000", 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.539e-05 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467183+0000", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5019999999999998e-06 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467224+0000", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1100000000000003e-05 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467319+0000", 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5014999999999997e-05 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467420+0000", 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100581 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468833+0000", 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014126449999999999 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468990+0000", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015734899999999999 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.540933+0000", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.071943453000000004 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541009+0000", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5680999999999996e-05 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541041+0000", 2024-04-25T05:36:31.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2421e-05 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:58642 7.1 7:82e4a570:::benchmark_data_smithi043_74356_object58641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:49.784555+0000", 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.454439678, 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.064574096999999997, 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.265 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 58642 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784555+0000", 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784555+0000", 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.267 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:31.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:31.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784557+0000", 2024-04-25T05:36:31.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.046e-06 2024-04-25T05:36:31.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784649+0000", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1182999999999998e-05 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784672+0000", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3074e-05 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784682+0000", 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0825999999999999e-05 2024-04-25T05:36:31.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784747+0000", 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4505999999999995e-05 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784892+0000", 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014481100000000001 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.785075+0000", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000183559 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787113+0000", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020374849999999999 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787487+0000", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00037450999999999999 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.849014+0000", 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061526330999999997 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.849077+0000", 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3643e-05 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.849129+0000", 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2123000000000002e-05 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:58647 7.1 7:84a15c38:::benchmark_data_smithi043_74356_object58646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:49.787567+0000", 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.451428346999997, 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.065297110000000005, 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 58647 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787567+0000", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787567+0000", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787568+0000", 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.488e-06 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787606+0000", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8245999999999999e-05 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787616+0000", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3759999999999997e-06 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787621+0000", 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2889999999999997e-06 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787659+0000", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8090999999999998e-05 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787794+0000", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000134599 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.787956+0000", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000161982 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.790762+0000", 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0028061000000000002 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.790814+0000", 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2462e-05 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.852793+0000", 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061979009000000002 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.852835+0000", 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1486000000000002e-05 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.852864+0000", 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8982000000000001e-05 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:58649 7.1 7:8bc80552:::benchmark_data_smithi043_74356_object58648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:31.278 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:49.788791+0000", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 41.450203946000002, 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.064221036999999995, 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 58649 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788791+0000", 2024-04-25T05:36:31.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788791+0000", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788792+0000", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.037e-06 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788853+0000", 2024-04-25T05:36:31.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1314000000000003e-05 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788865+0000", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1838000000000001e-05 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.788873+0000", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6179999999999997e-06 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.789531+0000", 2024-04-25T05:36:31.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00065804999999999998 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.789621+0000", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9844000000000002e-05 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.789737+0000", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116511 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.790908+0000", 2024-04-25T05:36:31.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001171009 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.791040+0000", 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132099 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.852912+0000", 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061871836999999999 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.852985+0000", 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2422999999999994e-05 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.853012+0000", 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7457000000000002e-05 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:59952 7.1 7:879116b0:::benchmark_data_smithi043_74356_object59951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:50.914111+0000", 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 40.324883638000003, 2024-04-25T05:36:31.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.062449557000000003, 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59952 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914111+0000", 2024-04-25T05:36:31.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914111+0000", 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914114+0000", 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1560000000000002e-06 2024-04-25T05:36:31.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914148+0000", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4388999999999999e-05 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914161+0000", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.31e-05 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914171+0000", 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8400000000000007e-06 2024-04-25T05:36:31.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914238+0000", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7500000000000001e-05 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914415+0000", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017657999999999999 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914586+0000", 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017145600000000001 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.916443+0000", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018561949999999999 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.916597+0000", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015405900000000001 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.976444+0000", 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.059847703000000002 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.976509+0000", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4209999999999997e-05 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.976561+0000", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2369e-05 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:60508 7.1 7:852d4bf6:::benchmark_data_smithi043_74356_object60507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:51.423241+0000", 2024-04-25T05:36:31.290 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.815754304000002, 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.047759154999999998, 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 60508 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423241+0000", 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423241+0000", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423242+0000", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1689999999999999e-06 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423285+0000", 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3387999999999999e-05 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423296+0000", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.101e-05 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423304+0000", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5020000000000003e-06 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423349+0000", 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4836999999999998e-05 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423443+0000", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4221999999999993e-05 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.423538+0000", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4784000000000006e-05 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.425218+0000", 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001679966 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.425253+0000", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5524000000000001e-05 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.470856+0000", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.045603151000000001 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.470932+0000", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5248999999999997e-05 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.471000+0000", 2024-04-25T05:36:31.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8353000000000001e-05 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61181 7.1 7:8dbc7052:::benchmark_data_smithi043_74356_object61180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:51.931327+0000", 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 39.307667647999999, 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.063821850999999999, 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.296 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61181 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931327+0000", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931327+0000", 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931329+0000", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1939999999999999e-06 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931368+0000", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9215999999999999e-05 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931379+0000", 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1154e-05 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931385+0000", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3570000000000003e-06 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931446+0000", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0883e-05 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931593+0000", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014713399999999999 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.931741+0000", 2024-04-25T05:36:31.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000148135 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.933998+0000", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022569370000000001 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.934180+0000", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018179599999999999 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.995082+0000", 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.060901760999999999 2024-04-25T05:36:31.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.995118+0000", 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5655000000000003e-05 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:51.995149+0000", 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1628999999999998e-05 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61989 7.1 7:81f9cbae:::benchmark_data_smithi043_74356_object61988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.615627+0000", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 38.623368407999997, 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.057927469000000002, 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61989 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615627+0000", 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615627+0000", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615628+0000", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.813e-06 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615667+0000", 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8985e-05 2024-04-25T05:36:31.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615677+0000", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9720000000000004e-06 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615685+0000", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9899999999999997e-06 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615748+0000", 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2405999999999998e-05 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.615936+0000", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018837699999999999 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.616086+0000", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000150261 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.618528+0000", 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024413500000000001 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.618900+0000", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00037227499999999998 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673429+0000", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.054529373999999999 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673497+0000", 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7986999999999998e-05 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673554+0000", 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6678999999999997e-05 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61994 7.1 7:84439222:::benchmark_data_smithi043_74356_object61993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.619678+0000", 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 38.619316765000001, 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.054055168000000001, 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61994 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619678+0000", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619678+0000", 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619684+0000", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1539999999999999e-06 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619733+0000", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8350000000000003e-05 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619743+0000", 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0180000000000001e-05 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619749+0000", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.874e-06 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619790+0000", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0670999999999997e-05 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619881+0000", 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1393999999999999e-05 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.619968+0000", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7598999999999999e-05 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.621909+0000", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019407319999999999 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.621936+0000", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7239000000000001e-05 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673646+0000", 2024-04-25T05:36:31.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.051709756000000003 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673695+0000", 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8809e-05 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.673733+0000", 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.841e-05 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.312 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:62945 7.1 7:8b486fef:::benchmark_data_smithi043_74356_object62944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:53.432207+0000", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 37.806788042000001, 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.037790460999999997, 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 62945 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432207+0000", 2024-04-25T05:36:31.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432207+0000", 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432209+0000", 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.702e-06 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432253+0000", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3862000000000001e-05 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432265+0000", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2058999999999999e-05 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432272+0000", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2389999999999996e-06 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432321+0000", 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9008000000000002e-05 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432431+0000", 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010971099999999999 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.432528+0000", 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7548999999999997e-05 2024-04-25T05:36:31.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.434196+0000", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001667965 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.434293+0000", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7032000000000002e-05 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.469892+0000", 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035598745000000001 2024-04-25T05:36:31.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.469945+0000", 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3616000000000003e-05 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:53.469997+0000", 2024-04-25T05:36:31.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1972999999999999e-05 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:63861 7.1 7:86f4cbac:::benchmark_data_smithi043_74356_object63860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:54.207014+0000", 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 37.031981268000003, 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.052014792999999997, 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.320 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 63861 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207014+0000", 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207014+0000", 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207015+0000", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.122e-06 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207064+0000", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8746999999999999e-05 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207075+0000", 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1602e-05 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207084+0000", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6309999999999992e-06 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207142+0000", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8184999999999999e-05 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207280+0000", 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013763999999999999 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.207422+0000", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014273199999999999 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.209484+0000", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020617399999999998 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.209572+0000", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8189000000000006e-05 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.258912+0000", 2024-04-25T05:36:31.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.049339635 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.258982+0000", 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9729999999999998e-05 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.259029+0000", 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6839999999999999e-05 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.325 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:87869 7.1 7:87ffb2c8:::benchmark_data_smithi043_74356_object87868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:16.010605+0000", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 15.228389630000001, 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039047410999999997, 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87869 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010605+0000", 2024-04-25T05:36:31.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010605+0000", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010606+0000", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6300000000000005e-07 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010642+0000", 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5964999999999999e-05 2024-04-25T05:36:31.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010649+0000", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3660000000000003e-06 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010656+0000", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8199999999999999e-06 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.013532+0000", 2024-04-25T05:36:31.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002875561 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.013667+0000", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000134831 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.013822+0000", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000155599 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.016265+0000", 2024-04-25T05:36:31.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024423650000000002 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.016962+0000", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000697294 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049263+0000", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.032301328999999997 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049609+0000", 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00034567499999999999 2024-04-25T05:36:31.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049653+0000", 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3943000000000003e-05 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:87872 7.1 7:81e780b5:::benchmark_data_smithi043_74356_object87871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:16.010666+0000", 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 15.228329200999999, 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039151553999999998, 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.331 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87872 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010666+0000", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010666+0000", 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.010666+0000", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5299999999999999e-07 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.029789+0000", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.019122818 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.029801+0000", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2349e-05 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.029810+0000", 2024-04-25T05:36:31.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8869999999999998e-06 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.029863+0000", 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2308999999999997e-05 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.030021+0000", 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015850799999999999 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.030174+0000", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015260999999999999 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.034957+0000", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047831779999999999 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.035344+0000", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038671700000000002 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049720+0000", 2024-04-25T05:36:31.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014375962000000001 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049775+0000", 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5850000000000002e-05 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:16.049817+0000", 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1912999999999997e-05 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:99681 7.1 7:842e73de:::benchmark_data_smithi043_74356_object99680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-04-25T05:36:31.336 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:27.931544+0000", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 3.3074514239999999, 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.074326821000000001, 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 99681 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931544+0000", 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931544+0000", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931548+0000", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5399999999999997e-06 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931645+0000", 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7140999999999997e-05 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931657+0000", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1277000000000001e-05 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931666+0000", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2529999999999993e-06 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931742+0000", 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6408000000000004e-05 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931915+0000", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017262000000000001 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.932097+0000", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018229499999999999 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.935124+0000", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030272680000000001 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.935666+0000", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054191299999999999 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.005796+0000", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.070129624000000002 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.005835+0000", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9418000000000001e-05 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.005870+0000", 2024-04-25T05:36:31.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5064000000000002e-05 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:100180 7.1 7:8513fbf1:::benchmark_data_smithi043_74356_object100179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:28.477301+0000", 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.761694265, 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.042600407999999999, 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.342 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 100180 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477301+0000", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477301+0000", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477302+0000", 2024-04-25T05:36:31.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.404e-06 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477356+0000", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3374000000000001e-05 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477366+0000", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.079e-05 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477372+0000", 2024-04-25T05:36:31.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5999999999999997e-06 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477413+0000", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1009e-05 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477514+0000", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010090399999999999 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.477619+0000", 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000104675 2024-04-25T05:36:31.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.479542+0000", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019235280000000001 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.479825+0000", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028299199999999998 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.519773+0000", 2024-04-25T05:36:31.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039947573 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.519840+0000", 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7818999999999996e-05 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.519901+0000", 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0739999999999998e-05 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.347 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:101555 7.1 7:8dd18088:::benchmark_data_smithi043_74356_object101554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:29.913441+0000", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.3255543439999999, 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.064125109999999999, 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 101555 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913441+0000", 2024-04-25T05:36:31.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913441+0000", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913443+0000", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.125e-06 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913489+0000", 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6201e-05 2024-04-25T05:36:31.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913511+0000", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2446000000000002e-05 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913524+0000", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2548000000000001e-05 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913665+0000", 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014109800000000001 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913819+0000", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000153437 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.913986+0000", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000167906 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.916455+0000", 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024681429999999999 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.917566+0000", 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00111136 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.977493+0000", 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.059927030999999999 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.977532+0000", 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9180000000000001e-05 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.977566+0000", 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3634999999999999e-05 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:102257 7.1 7:8e733645:::benchmark_data_smithi043_74356_object102256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:30.708615+0000", 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.53038020799999996, 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035793065999999998, 2024-04-25T05:36:31.353 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 102257 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708615+0000", 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708615+0000", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708617+0000", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8560000000000001e-06 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708687+0000", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0679999999999994e-05 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:31.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708698+0000", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0997e-05 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708707+0000", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9819999999999997e-06 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708755+0000", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7237999999999997e-05 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:31.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.708859+0000", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010447599999999999 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.709021+0000", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162292 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.712038+0000", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003016473 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.713774+0000", 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001736032 2024-04-25T05:36:31.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.744094+0000", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.030319797999999999 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.744358+0000", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026426599999999998 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:30.744408+0000", 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9975999999999997e-05 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.358 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.359 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.359 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:31.359 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:31.359 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:31.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:31.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:31.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:31.558 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:31.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:31.733 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:36:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:31.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:31.744 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 103094 103078 85.8717 50.625 0.0340782 0.0116378 2024-04-25T05:36:31.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:31.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:31.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:31.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:31.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:36:31.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 7.e scrub starts 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: osdmap e120: 8 total, 8 up, 8 in 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 3.12 deep-scrub starts 2024-04-25T05:36:31.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:31 smithi060 ceph-mon[30898]: 3.12 deep-scrub ok 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 7.e scrub starts 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: osdmap e120: 8 total, 8 up, 8 in 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 3.12 deep-scrub starts 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[27010]: 3.12 deep-scrub ok 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:36:32.002 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 7.e scrub starts 2024-04-25T05:36:32.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:36:32.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:36:32.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: osdmap e120: 8 total, 8 up, 8 in 2024-04-25T05:36:32.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 3.12 deep-scrub starts 2024-04-25T05:36:32.003 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:31 smithi043 ceph-mon[37492]: 3.12 deep-scrub ok 2024-04-25T05:36:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:32.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:32.077 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:32.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:32.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:36:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:32.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.359 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:32.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:32.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:32.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:32.516 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:32.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:32.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:32.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 103888 103872 85.3938 49.625 0.0168053 0.0117053 2024-04-25T05:36:32.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:32.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:32.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:32.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:36:32.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:32.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:33.038 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:33.127 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:33.127 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:104230 7.e e121/88 7:7543942f:::benchmark_data_smithi043_74356_object104229:head v 121'6510, mlcod=121'6508)", 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:33.124146+0000", 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0036474319999999999, 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036836669999999998, 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124146+0000", 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124146+0000", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124147+0000", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.187e-06 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124191+0000", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-05 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:33.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124210+0000", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9672000000000001e-05 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124220+0000", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0208e-05 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124299+0000", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8589000000000003e-05 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.124321+0000", 2024-04-25T05:36:33.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2187e-05 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:104218 7.a e121/88 7:57468ce5:::benchmark_data_smithi043_74356_object104217:head v 121'6512, mlcod=121'6509)", 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:33.125187+0000", 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0026063050000000002, 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027032050000000002, 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125187+0000", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125187+0000", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125188+0000", 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0499999999999999e-06 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125232+0000", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3327e-05 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125246+0000", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3884e-05 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125255+0000", 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0699999999999996e-06 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125338+0000", 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3156000000000001e-05 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:33.125361+0000", 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2878999999999999e-05 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 2 2024-04-25T05:36:33.134 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:33.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:33.188 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: pgmap v105: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 7.8 scrub ok 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4269416602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:32 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:33.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:33.354 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: pgmap v105: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 7.8 scrub ok 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4269416602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: pgmap v105: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 839 op/s 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 7.8 scrub ok 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:33.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4269416602' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.356 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:32 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:36:33.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:36:33.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:33.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:33.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:33.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:33.745 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 104789 104773 85.0161 56.3125 0.00839855 0.0117566 2024-04-25T05:36:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:33.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:33.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:33.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: osdmap e121: 8 total, 8 up, 8 in 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:36:33.984 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:33 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:36:33.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:34.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:34.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:36:34.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:34.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:34.286 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:34.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: osdmap e121: 8 total, 8 up, 8 in 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:36:34.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:34.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:34.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:36:34.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: osdmap e121: 8 total, 8 up, 8 in 2024-04-25T05:36:34.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:36:34.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:33 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:36:34.414 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:34.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:34.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:34.575 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:34.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:34.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:36:34.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 105771 105755 84.713 61.375 0.00613837 0.011799 2024-04-25T05:36:34.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:34.847 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:34.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:34.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:34.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:35.024 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:35.024 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:36:35.024 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:36:35.024 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:36:35.024 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:35.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:35.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 945 B/s rd, 51 MiB/s wr, 813 op/s 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: osdmap e122: 8 total, 8 up, 8 in 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/830608725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:36:35.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[27010]: osdmap e123: 8 total, 8 up, 8 in 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 945 B/s rd, 51 MiB/s wr, 813 op/s 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: osdmap e122: 8 total, 8 up, 8 in 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/830608725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:36:35.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:34 smithi043 ceph-mon[37492]: osdmap e123: 8 total, 8 up, 8 in 2024-04-25T05:36:35.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: pgmap v107: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 945 B/s rd, 51 MiB/s wr, 813 op/s 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: osdmap e122: 8 total, 8 up, 8 in 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/830608725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:36:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:34 smithi060 ceph-mon[30898]: osdmap e123: 8 total, 8 up, 8 in 2024-04-25T05:36:35.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:35.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:35.380 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:35.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:35.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:35.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:35.635 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:35.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:36:35.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:35.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 106766 106750 84.4279 62.1875 0.0111118 0.0118384 2024-04-25T05:36:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:35.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:35.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:35.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:36.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:36.058 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:36.074 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:36.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:36.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:36:36.315 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:36.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:36.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:36.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:36.581 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:36.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.736 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:36.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:36.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:36:36.747244+0000 min lat: 0.00120772 max lat: 0.140459 avg lat: 0.0118796 2024-04-25T05:36:36.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:36:36.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 107749 107733 84.1405 61.4375 0.0130677 0.0118796 2024-04-25T05:36:36.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:36.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:36.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:36.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:37.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:37.053 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:37.091 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:37.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:36:37.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:36:37.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[27010]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-04-25T05:36:37.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[27010]: osdmap e124: 8 total, 8 up, 8 in 2024-04-25T05:36:37.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:36:37.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:36:37.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[37492]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-04-25T05:36:37.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:36 smithi043 ceph-mon[37492]: osdmap e124: 8 total, 8 up, 8 in 2024-04-25T05:36:37.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:37.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:36 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:36:37.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:36 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:36:37.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:36 smithi060 ceph-mon[30898]: pgmap v110: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 887 op/s 2024-04-25T05:36:37.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:36 smithi060 ceph-mon[30898]: osdmap e124: 8 total, 8 up, 8 in 2024-04-25T05:36:37.142 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:37.146 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:37.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:37.223 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:37.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:37.270 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:37.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:37.309 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:37.317 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:37.317 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:37.354 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:37.364 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:37.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:37.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:37.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:37.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:37.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:37.694 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:37.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 108739 108723 83.8656 61.875 0.0165153 0.0119186 2024-04-25T05:36:37.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:37.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:37.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:37.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:37.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:38.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:36:38.028 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:38.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:38.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:38.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3874266677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:38.263 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:38.363 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:38.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3874266677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3874266677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:38.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:38.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:38.565 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:38.618 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:36:38.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:36:38.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:38.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:38.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 109701 109685 83.5761 60.125 0.017435 0.0119601 2024-04-25T05:36:38.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:38.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:38.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:38.989 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:38.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:39.008 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:39.008 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:39.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:39.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 MiB/s wr, 192 op/s 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:39.199 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 MiB/s wr, 192 op/s 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.200 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:39.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:39.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:39.411 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:39.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: pgmap v112: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 MiB/s wr, 192 op/s 2024-04-25T05:36:39.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:36:39.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:39.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:39.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.672 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:39.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:39.746 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 110567 110551 83.2212 54.125 0.0101939 0.0120109 2024-04-25T05:36:39.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:39.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:39.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:39.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:40.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:40.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:40.101 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:36:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:36:40.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:40.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:40.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/761406534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[27010]: osdmap e125: 8 total, 8 up, 8 in 2024-04-25T05:36:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/761406534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:40 smithi043 ceph-mon[37492]: osdmap e125: 8 total, 8 up, 8 in 2024-04-25T05:36:40.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:40.472 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/761406534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:40.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:40 smithi060 ceph-mon[30898]: osdmap e125: 8 total, 8 up, 8 in 2024-04-25T05:36:40.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:40.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:36:40.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:36:40.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:40.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 111547 111531 82.9581 61.25 0.0521825 0.0120485 2024-04-25T05:36:40.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:40.872 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:40.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:40.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:41.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[27010]: pgmap v113: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:41.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:41.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:41.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[27010]: osdmap e126: 8 total, 8 up, 8 in 2024-04-25T05:36:41.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:41.135 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:41.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:41 smithi060 ceph-mon[30898]: pgmap v113: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:41 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:41 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:41 smithi060 ceph-mon[30898]: osdmap e126: 8 total, 8 up, 8 in 2024-04-25T05:36:41.277 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:41.282 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:41.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[37492]: pgmap v113: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:41 smithi043 ceph-mon[37492]: osdmap e126: 8 total, 8 up, 8 in 2024-04-25T05:36:41.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:41.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:41.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:41.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:41.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 112616 112600 82.7681 66.8125 0.00760344 0.0120767 2024-04-25T05:36:41.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:41.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:41.827 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:41.878 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:41.878 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:41.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:41.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:41.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:42.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:42.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:42.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:42.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:42.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:42.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:42.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:42.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:42.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:42.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:36:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:42.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:42.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:36:42.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 113467 113451 82.4242 53.1875 0.0626921 0.0121254 2024-04-25T05:36:42.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:42.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:42.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:42.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:42.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:43.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:43.021 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:36:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:36:43.110 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:43.138 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:43.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:43.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:43.305 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:43.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:43.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:43 smithi043 ceph-mon[27010]: pgmap v116: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:43.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:43 smithi043 ceph-mon[37492]: pgmap v116: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:43.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:43.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:43 smithi060 ceph-mon[30898]: pgmap v116: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 51 MiB/s wr, 811 op/s 2024-04-25T05:36:43.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:43.528 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:43.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:43.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:43.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 15 114281 114266 82.0622 50.9375 0.017377 0.0121806 2024-04-25T05:36:43.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:43.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:36:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:43.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:43.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:43.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:44.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:44.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:44.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:44.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:44.237 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:44.238 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:44.265 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-25T05:36:41.699285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7127243291,"num_objects":108974,"num_object_clones":0,"num_object_copies":326922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108974,"num_whiteouts":0,"num_read":1375,"num_read_kb":1189,"num_write":109551,"num_write_kb":6962077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90462,"ondisk_log_size":90462,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21388596,"kb_used_data":20785180,"kb_used_omap":12,"kb_used_meta":603315,"kb_avail":728572620,"statfs":{"total":767960285184,"available":746058362880,"internally_reserved":0,"allocated":21284024320,"data_stored":21278894521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":617795120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,12,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":319094784,"num_objects":4869,"num_object_clones":0,"num_object_copies":14607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4869,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4869,"num_write_kb":311616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003943"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":113,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945309+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:36:35.945309+0000","last_peered":"2024-04-25T05:36:35.945309+0000","last_clean":"2024-04-25T05:36:35.945309+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:36:35.945309+0000","last_undegraded":"2024-04-25T05:36:35.945309+0000","last_fullsized":"2024-04-25T05:36:35.945309+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974541+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:36:35.974541+0000","last_peered":"2024-04-25T05:36:35.974541+0000","last_clean":"2024-04-25T05:36:35.974541+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:36:35.974541+0000","last_undegraded":"2024-04-25T05:36:35.974541+0000","last_fullsized":"2024-04-25T05:36:35.974541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.859796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972128+0000","last_change":"2024-04-25T05:36:18.176520+0000","last_active":"2024-04-25T05:36:35.972128+0000","last_peered":"2024-04-25T05:36:35.972128+0000","last_clean":"2024-04-25T05:36:35.972128+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:36:35.972128+0000","last_undegraded":"2024-04-25T05:36:35.972128+0000","last_fullsized":"2024-04-25T05:36:35.972128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.176394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.176394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:23.464747+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945015+0000","last_change":"2024-04-25T05:36:13.708900+0000","last_active":"2024-04-25T05:36:35.945015+0000","last_peered":"2024-04-25T05:36:35.945015+0000","last_clean":"2024-04-25T05:36:35.945015+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:36:35.945015+0000","last_undegraded":"2024-04-25T05:36:35.945015+0000","last_fullsized":"2024-04-25T05:36:35.945015+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.170106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974358+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:36:35.974358+0000","last_peered":"2024-04-25T05:36:35.974358+0000","last_clean":"2024-04-25T05:36:35.974358+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:36:35.974358+0000","last_undegraded":"2024-04-25T05:36:35.974358+0000","last_fullsized":"2024-04-25T05:36:35.974358+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":117,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945098+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:36:35.945098+0000","last_peered":"2024-04-25T05:36:35.945098+0000","last_clean":"2024-04-25T05:36:35.945098+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:36:35.945098+0000","last_undegraded":"2024-04-25T05:36:35.945098+0000","last_fullsized":"2024-04-25T05:36:35.945098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974076+0000","last_change":"2024-04-25T05:35:27.201991+0000","last_active":"2024-04-25T05:36:35.974076+0000","last_peered":"2024-04-25T05:36:35.974076+0000","last_clean":"2024-04-25T05:36:35.974076+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:36:35.974076+0000","last_undegraded":"2024-04-25T05:36:35.974076+0000","last_fullsized":"2024-04-25T05:36:35.974076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450698+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":195,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972540+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:36:35.972540+0000","last_peered":"2024-04-25T05:36:35.972540+0000","last_clean":"2024-04-25T05:36:35.972540+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:36:35.972540+0000","last_undegraded":"2024-04-25T05:36:35.972540+0000","last_fullsized":"2024-04-25T05:36:35.972540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972605+0000","last_change":"2024-04-25T05:35:32.284750+0000","last_active":"2024-04-25T05:36:35.972605+0000","last_peered":"2024-04-25T05:36:35.972605+0000","last_clean":"2024-04-25T05:36:35.972605+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:36:35.972605+0000","last_undegraded":"2024-04-25T05:36:35.972605+0000","last_fullsized":"2024-04-25T05:36:35.972605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.284637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.284637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.825789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971424+0000","last_change":"2024-04-25T05:36:29.981966+0000","last_active":"2024-04-25T05:36:35.971424+0000","last_peered":"2024-04-25T05:36:35.971424+0000","last_clean":"2024-04-25T05:36:35.971424+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:36:35.971424+0000","last_undegraded":"2024-04-25T05:36:35.971424+0000","last_fullsized":"2024-04-25T05:36:35.971424+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.093687+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971747+0000","last_change":"2024-04-25T05:36:31.998727+0000","last_active":"2024-04-25T05:36:35.971747+0000","last_peered":"2024-04-25T05:36:35.971747+0000","last_clean":"2024-04-25T05:36:35.971747+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:36:35.971747+0000","last_undegraded":"2024-04-25T05:36:35.971747+0000","last_fullsized":"2024-04-25T05:36:35.971747+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.071661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":162,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945421+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:36:35.945421+0000","last_peered":"2024-04-25T05:36:35.945421+0000","last_clean":"2024-04-25T05:36:35.945421+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:36:35.945421+0000","last_undegraded":"2024-04-25T05:36:35.945421+0000","last_fullsized":"2024-04-25T05:36:35.945421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.970734+0000","last_change":"2024-04-25T05:35:32.732758+0000","last_active":"2024-04-25T05:36:35.970734+0000","last_peered":"2024-04-25T05:36:35.970734+0000","last_clean":"2024-04-25T05:36:35.970734+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:36:35.970734+0000","last_undegraded":"2024-04-25T05:36:35.970734+0000","last_fullsized":"2024-04-25T05:36:35.970734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.732662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.732662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:49.439622+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975807+0000","last_change":"2024-04-25T05:35:44.725188+0000","last_active":"2024-04-25T05:36:35.975807+0000","last_peered":"2024-04-25T05:36:35.975807+0000","last_clean":"2024-04-25T05:36:35.975807+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:36:35.975807+0000","last_undegraded":"2024-04-25T05:36:35.975807+0000","last_fullsized":"2024-04-25T05:36:35.975807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.725019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.725019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:07.191021+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971351+0000","last_change":"2024-04-25T05:35:06.788446+0000","last_active":"2024-04-25T05:36:35.971351+0000","last_peered":"2024-04-25T05:36:35.971351+0000","last_clean":"2024-04-25T05:36:35.971351+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:36:35.971351+0000","last_undegraded":"2024-04-25T05:36:35.971351+0000","last_fullsized":"2024-04-25T05:36:35.971351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.788358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":165,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945541+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:36:35.945541+0000","last_peered":"2024-04-25T05:36:35.945541+0000","last_clean":"2024-04-25T05:36:35.945541+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:36:35.945541+0000","last_undegraded":"2024-04-25T05:36:35.945541+0000","last_fullsized":"2024-04-25T05:36:35.945541+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.256093+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":113,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946127+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:36:35.946127+0000","last_peered":"2024-04-25T05:36:35.946127+0000","last_clean":"2024-04-25T05:36:35.946127+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:36:35.946127+0000","last_undegraded":"2024-04-25T05:36:35.946127+0000","last_fullsized":"2024-04-25T05:36:35.946127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975950+0000","last_change":"2024-04-25T05:35:27.229584+0000","last_active":"2024-04-25T05:36:35.975950+0000","last_peered":"2024-04-25T05:36:35.975950+0000","last_clean":"2024-04-25T05:36:35.975950+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:36:35.975950+0000","last_undegraded":"2024-04-25T05:36:35.975950+0000","last_fullsized":"2024-04-25T05:36:35.975950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.229461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.229461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:37.564438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972984+0000","last_change":"2024-04-25T05:35:07.169298+0000","last_active":"2024-04-25T05:36:35.972984+0000","last_peered":"2024-04-25T05:36:35.972984+0000","last_clean":"2024-04-25T05:36:35.972984+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:36:35.972984+0000","last_undegraded":"2024-04-25T05:36:35.972984+0000","last_fullsized":"2024-04-25T05:36:35.972984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:07.169214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.169214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.149164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.967092+0000","last_change":"2024-04-25T05:35:28.590848+0000","last_active":"2024-04-25T05:36:35.967092+0000","last_peered":"2024-04-25T05:36:35.967092+0000","last_clean":"2024-04-25T05:36:35.967092+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:36:35.967092+0000","last_undegraded":"2024-04-25T05:36:35.967092+0000","last_fullsized":"2024-04-25T05:36:35.967092+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:28.590759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.590759+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.469480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976656+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:36:35.976656+0000","last_peered":"2024-04-25T05:36:35.976656+0000","last_clean":"2024-04-25T05:36:35.976656+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:36:35.976656+0000","last_undegraded":"2024-04-25T05:36:35.976656+0000","last_fullsized":"2024-04-25T05:36:35.976656+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":117,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945997+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:36:35.945997+0000","last_peered":"2024-04-25T05:36:35.945997+0000","last_clean":"2024-04-25T05:36:35.945997+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:36:35.945997+0000","last_undegraded":"2024-04-25T05:36:35.945997+0000","last_fullsized":"2024-04-25T05:36:35.945997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945531+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:36:35.945531+0000","last_peered":"2024-04-25T05:36:35.945531+0000","last_clean":"2024-04-25T05:36:35.945531+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:36:35.945531+0000","last_undegraded":"2024-04-25T05:36:35.945531+0000","last_fullsized":"2024-04-25T05:36:35.945531+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.515357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973861+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:36:35.973861+0000","last_peered":"2024-04-25T05:36:35.973861+0000","last_clean":"2024-04-25T05:36:35.973861+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:36:35.973861+0000","last_undegraded":"2024-04-25T05:36:35.973861+0000","last_fullsized":"2024-04-25T05:36:35.973861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":123,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976868+0000","last_change":"2024-04-25T05:35:24.795037+0000","last_active":"2024-04-25T05:36:35.976868+0000","last_peered":"2024-04-25T05:36:35.976868+0000","last_clean":"2024-04-25T05:36:35.976868+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:36:35.976868+0000","last_undegraded":"2024-04-25T05:36:35.976868+0000","last_fullsized":"2024-04-25T05:36:35.976868+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:35:24.794880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.794880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:48.363218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":169,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976054+0000","last_change":"2024-04-25T05:36:22.989438+0000","last_active":"2024-04-25T05:36:35.976054+0000","last_peered":"2024-04-25T05:36:35.976054+0000","last_clean":"2024-04-25T05:36:35.976054+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:36:35.976054+0000","last_undegraded":"2024-04-25T05:36:35.976054+0000","last_fullsized":"2024-04-25T05:36:35.976054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.372787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974621+0000","last_change":"2024-04-25T05:35:25.303821+0000","last_active":"2024-04-25T05:36:35.974621+0000","last_peered":"2024-04-25T05:36:35.974621+0000","last_clean":"2024-04-25T05:36:35.974621+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:36:35.974621+0000","last_undegraded":"2024-04-25T05:36:35.974621+0000","last_fullsized":"2024-04-25T05:36:35.974621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:25.303682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.303682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:37.162924+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.981710+0000","last_change":"2024-04-25T05:36:15.129016+0000","last_active":"2024-04-25T05:36:35.981710+0000","last_peered":"2024-04-25T05:36:35.981710+0000","last_clean":"2024-04-25T05:36:35.981710+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:36:35.981710+0000","last_undegraded":"2024-04-25T05:36:35.981710+0000","last_fullsized":"2024-04-25T05:36:35.981710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:36:15.128887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.128887+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:26.080685+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975714+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:36:35.975714+0000","last_peered":"2024-04-25T05:36:35.975714+0000","last_clean":"2024-04-25T05:36:35.975714+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:36:35.975714+0000","last_undegraded":"2024-04-25T05:36:35.975714+0000","last_fullsized":"2024-04-25T05:36:35.975714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:03.143088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.982116+0000","last_change":"2024-04-25T05:36:14.171466+0000","last_active":"2024-04-25T05:36:35.982116+0000","last_peered":"2024-04-25T05:36:35.982116+0000","last_clean":"2024-04-25T05:36:35.982116+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:36:35.982116+0000","last_undegraded":"2024-04-25T05:36:35.982116+0000","last_fullsized":"2024-04-25T05:36:35.982116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.171349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.171349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:24.814418+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972195+0000","last_change":"2024-04-25T05:35:36.324777+0000","last_active":"2024-04-25T05:36:35.972195+0000","last_peered":"2024-04-25T05:36:35.972195+0000","last_clean":"2024-04-25T05:36:35.972195+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:36:35.972195+0000","last_undegraded":"2024-04-25T05:36:35.972195+0000","last_fullsized":"2024-04-25T05:36:35.972195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.688505+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":161,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975940+0000","last_change":"2024-04-25T05:35:25.674485+0000","last_active":"2024-04-25T05:36:35.975940+0000","last_peered":"2024-04-25T05:36:35.975940+0000","last_clean":"2024-04-25T05:36:35.975940+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:36:35.975940+0000","last_undegraded":"2024-04-25T05:36:35.975940+0000","last_fullsized":"2024-04-25T05:36:35.975940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:25.674276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.674276+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":123,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975670+0000","last_change":"2024-04-25T05:36:22.428077+0000","last_active":"2024-04-25T05:36:35.975670+0000","last_peered":"2024-04-25T05:36:35.975670+0000","last_clean":"2024-04-25T05:36:35.975670+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:36:35.975670+0000","last_undegraded":"2024-04-25T05:36:35.975670+0000","last_fullsized":"2024-04-25T05:36:35.975670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:22.427930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.427930+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:25.727609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975816+0000","last_change":"2024-04-25T05:35:28.647997+0000","last_active":"2024-04-25T05:36:35.975816+0000","last_peered":"2024-04-25T05:36:35.975816+0000","last_clean":"2024-04-25T05:36:35.975816+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:36:35.975816+0000","last_undegraded":"2024-04-25T05:36:35.975816+0000","last_fullsized":"2024-04-25T05:36:35.975816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.916724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972332+0000","last_change":"2024-04-25T05:36:21.217686+0000","last_active":"2024-04-25T05:36:35.972332+0000","last_peered":"2024-04-25T05:36:35.972332+0000","last_clean":"2024-04-25T05:36:35.972332+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:36:35.972332+0000","last_undegraded":"2024-04-25T05:36:35.972332+0000","last_fullsized":"2024-04-25T05:36:35.972332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.483356+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":136,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972278+0000","last_change":"2024-04-25T05:36:19.405542+0000","last_active":"2024-04-25T05:36:35.972278+0000","last_peered":"2024-04-25T05:36:35.972278+0000","last_clean":"2024-04-25T05:36:35.972278+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:36:35.972278+0000","last_undegraded":"2024-04-25T05:36:35.972278+0000","last_fullsized":"2024-04-25T05:36:35.972278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:48.487037+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975414+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:36:35.975414+0000","last_peered":"2024-04-25T05:36:35.975414+0000","last_clean":"2024-04-25T05:36:35.975414+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:36:35.975414+0000","last_undegraded":"2024-04-25T05:36:35.975414+0000","last_fullsized":"2024-04-25T05:36:35.975414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976748+0000","last_change":"2024-04-25T05:35:45.748116+0000","last_active":"2024-04-25T05:36:35.976748+0000","last_peered":"2024-04-25T05:36:35.976748+0000","last_clean":"2024-04-25T05:36:35.976748+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:36:35.976748+0000","last_undegraded":"2024-04-25T05:36:35.976748+0000","last_fullsized":"2024-04-25T05:36:35.976748+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:45.748002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:45.748002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.126478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973367+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:36:35.973367+0000","last_peered":"2024-04-25T05:36:35.973367+0000","last_clean":"2024-04-25T05:36:35.973367+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:36:35.973367+0000","last_undegraded":"2024-04-25T05:36:35.973367+0000","last_fullsized":"2024-04-25T05:36:35.973367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:38.198166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":161,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972438+0000","last_change":"2024-04-25T05:35:38.338139+0000","last_active":"2024-04-25T05:36:35.972438+0000","last_peered":"2024-04-25T05:36:35.972438+0000","last_clean":"2024-04-25T05:36:35.972438+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:36:35.972438+0000","last_undegraded":"2024-04-25T05:36:35.972438+0000","last_fullsized":"2024-04-25T05:36:35.972438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.229898+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972814+0000","last_change":"2024-04-25T05:35:34.276800+0000","last_active":"2024-04-25T05:36:35.972814+0000","last_peered":"2024-04-25T05:36:35.972814+0000","last_clean":"2024-04-25T05:36:35.972814+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:36:35.972814+0000","last_undegraded":"2024-04-25T05:36:35.972814+0000","last_fullsized":"2024-04-25T05:36:35.972814+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.276673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.276673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.973764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":118,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974264+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:36:35.974264+0000","last_peered":"2024-04-25T05:36:35.974264+0000","last_clean":"2024-04-25T05:36:35.974264+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:36:35.974264+0000","last_undegraded":"2024-04-25T05:36:35.974264+0000","last_fullsized":"2024-04-25T05:36:35.974264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.977012+0000","last_change":"2024-04-25T05:35:42.725828+0000","last_active":"2024-04-25T05:36:35.977012+0000","last_peered":"2024-04-25T05:36:35.977012+0000","last_clean":"2024-04-25T05:36:35.977012+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:36:35.977012+0000","last_undegraded":"2024-04-25T05:36:35.977012+0000","last_fullsized":"2024-04-25T05:36:35.977012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.725737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.103408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":83,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.970741+0000","last_change":"2024-04-25T05:35:33.708929+0000","last_active":"2024-04-25T05:36:35.970741+0000","last_peered":"2024-04-25T05:36:35.970741+0000","last_clean":"2024-04-25T05:36:35.970741+0000","last_became_active":"2024-04-25T05:35:32.892093+0000","last_became_peered":"2024-04-25T05:35:32.892093+0000","last_unstale":"2024-04-25T05:36:35.970741+0000","last_undegraded":"2024-04-25T05:36:35.970741+0000","last_fullsized":"2024-04-25T05:36:35.970741+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:35:33.708629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:33.708629+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.641362+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972124+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:36:35.972124+0000","last_peered":"2024-04-25T05:36:35.972124+0000","last_clean":"2024-04-25T05:36:35.972124+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:36:35.972124+0000","last_undegraded":"2024-04-25T05:36:35.972124+0000","last_fullsized":"2024-04-25T05:36:35.972124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975136+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:36:35.975136+0000","last_peered":"2024-04-25T05:36:35.975136+0000","last_clean":"2024-04-25T05:36:35.975136+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:36:35.975136+0000","last_undegraded":"2024-04-25T05:36:35.975136+0000","last_fullsized":"2024-04-25T05:36:35.975136+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.932366+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945829+0000","last_change":"2024-04-25T05:35:02.707425+0000","last_active":"2024-04-25T05:36:35.945829+0000","last_peered":"2024-04-25T05:36:35.945829+0000","last_clean":"2024-04-25T05:36:35.945829+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:36:35.945829+0000","last_undegraded":"2024-04-25T05:36:35.945829+0000","last_fullsized":"2024-04-25T05:36:35.945829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":148,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945665+0000","last_change":"2024-04-25T05:35:07.728633+0000","last_active":"2024-04-25T05:36:35.945665+0000","last_peered":"2024-04-25T05:36:35.945665+0000","last_clean":"2024-04-25T05:36:35.945665+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:36:35.945665+0000","last_undegraded":"2024-04-25T05:36:35.945665+0000","last_fullsized":"2024-04-25T05:36:35.945665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975933+0000","last_change":"2024-04-25T05:36:30.109005+0000","last_active":"2024-04-25T05:36:35.975933+0000","last_peered":"2024-04-25T05:36:35.975933+0000","last_clean":"2024-04-25T05:36:35.975933+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:36:35.975933+0000","last_undegraded":"2024-04-25T05:36:35.975933+0000","last_fullsized":"2024-04-25T05:36:35.975933+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.212177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971835+0000","last_change":"2024-04-25T05:35:39.823582+0000","last_active":"2024-04-25T05:36:35.971835+0000","last_peered":"2024-04-25T05:36:35.971835+0000","last_clean":"2024-04-25T05:36:35.971835+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:36:35.971835+0000","last_undegraded":"2024-04-25T05:36:35.971835+0000","last_fullsized":"2024-04-25T05:36:35.971835+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:39.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.823449+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.097815+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976057+0000","last_change":"2024-04-25T05:36:15.504640+0000","last_active":"2024-04-25T05:36:35.976057+0000","last_peered":"2024-04-25T05:36:35.976057+0000","last_clean":"2024-04-25T05:36:35.976057+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:36:35.976057+0000","last_undegraded":"2024-04-25T05:36:35.976057+0000","last_fullsized":"2024-04-25T05:36:35.976057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:15.504395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.504395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:20.668339+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":169,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973757+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:36:35.973757+0000","last_peered":"2024-04-25T05:36:35.973757+0000","last_clean":"2024-04-25T05:36:35.973757+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:36:35.973757+0000","last_undegraded":"2024-04-25T05:36:35.973757+0000","last_fullsized":"2024-04-25T05:36:35.973757+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.719798+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946315+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:36:35.946315+0000","last_peered":"2024-04-25T05:36:35.946315+0000","last_clean":"2024-04-25T05:36:35.946315+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:36:35.946315+0000","last_undegraded":"2024-04-25T05:36:35.946315+0000","last_fullsized":"2024-04-25T05:36:35.946315+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.179078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974211+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:36:35.974211+0000","last_peered":"2024-04-25T05:36:35.974211+0000","last_clean":"2024-04-25T05:36:35.974211+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:36:35.974211+0000","last_undegraded":"2024-04-25T05:36:35.974211+0000","last_fullsized":"2024-04-25T05:36:35.974211+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971897+0000","last_change":"2024-04-25T05:36:31.118006+0000","last_active":"2024-04-25T05:36:35.971897+0000","last_peered":"2024-04-25T05:36:35.971897+0000","last_clean":"2024-04-25T05:36:35.971897+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:36:35.971897+0000","last_undegraded":"2024-04-25T05:36:35.971897+0000","last_fullsized":"2024-04-25T05:36:35.971897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.339192+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":163,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972080+0000","last_change":"2024-04-25T05:35:40.845196+0000","last_active":"2024-04-25T05:36:35.972080+0000","last_peered":"2024-04-25T05:36:35.972080+0000","last_clean":"2024-04-25T05:36:35.972080+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:36:35.972080+0000","last_undegraded":"2024-04-25T05:36:35.972080+0000","last_fullsized":"2024-04-25T05:36:35.972080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.845036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:08.541350+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.967389+0000","last_change":"2024-04-25T05:35:30.673162+0000","last_active":"2024-04-25T05:36:35.967389+0000","last_peered":"2024-04-25T05:36:35.967389+0000","last_clean":"2024-04-25T05:36:35.967389+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:36:35.967389+0000","last_undegraded":"2024-04-25T05:36:35.967389+0000","last_fullsized":"2024-04-25T05:36:35.967389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:30.673041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.673041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.187518+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":118,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975738+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:36:35.975738+0000","last_peered":"2024-04-25T05:36:35.975738+0000","last_clean":"2024-04-25T05:36:35.975738+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:36:35.975738+0000","last_undegraded":"2024-04-25T05:36:35.975738+0000","last_fullsized":"2024-04-25T05:36:35.975738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976291+0000","last_change":"2024-04-25T05:35:28.761829+0000","last_active":"2024-04-25T05:36:35.976291+0000","last_peered":"2024-04-25T05:36:35.976291+0000","last_clean":"2024-04-25T05:36:35.976291+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:36:35.976291+0000","last_undegraded":"2024-04-25T05:36:35.976291+0000","last_fullsized":"2024-04-25T05:36:35.976291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.761706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.761706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.404788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":170,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.981945+0000","last_change":"2024-04-25T05:36:20.155217+0000","last_active":"2024-04-25T05:36:35.981945+0000","last_peered":"2024-04-25T05:36:35.981945+0000","last_clean":"2024-04-25T05:36:35.981945+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:36:35.981945+0000","last_undegraded":"2024-04-25T05:36:35.981945+0000","last_fullsized":"2024-04-25T05:36:35.981945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:40.449029+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976912+0000","last_change":"2024-04-25T05:36:18.481176+0000","last_active":"2024-04-25T05:36:35.976912+0000","last_peered":"2024-04-25T05:36:35.976912+0000","last_clean":"2024-04-25T05:36:35.976912+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:36:35.976912+0000","last_undegraded":"2024-04-25T05:36:35.976912+0000","last_fullsized":"2024-04-25T05:36:35.976912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.481007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.481007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.599925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973797+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:36:35.973797+0000","last_peered":"2024-04-25T05:36:35.973797+0000","last_clean":"2024-04-25T05:36:35.973797+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:36:35.973797+0000","last_undegraded":"2024-04-25T05:36:35.973797+0000","last_fullsized":"2024-04-25T05:36:35.973797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.303715+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974240+0000","last_change":"2024-04-25T05:35:44.715464+0000","last_active":"2024-04-25T05:36:35.974240+0000","last_peered":"2024-04-25T05:36:35.974240+0000","last_clean":"2024-04-25T05:36:35.974240+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:36:35.974240+0000","last_undegraded":"2024-04-25T05:36:35.974240+0000","last_fullsized":"2024-04-25T05:36:35.974240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.715360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.715360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:49.881138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":151,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976748+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:36:35.976748+0000","last_peered":"2024-04-25T05:36:35.976748+0000","last_clean":"2024-04-25T05:36:35.976748+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:36:35.976748+0000","last_undegraded":"2024-04-25T05:36:35.976748+0000","last_fullsized":"2024-04-25T05:36:35.976748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"124'6702","reported_seq":6848,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.721898+0000","last_change":"2024-04-25T05:36:18.461712+0000","last_active":"2024-04-25T05:36:37.721898+0000","last_peered":"2024-04-25T05:36:37.721898+0000","last_clean":"2024-04-25T05:36:37.721898+0000","last_became_active":"2024-04-25T05:35:09.679411+0000","last_became_peered":"2024-04-25T05:35:09.679411+0000","last_unstale":"2024-04-25T05:36:37.721898+0000","last_undegraded":"2024-04-25T05:36:37.721898+0000","last_fullsized":"2024-04-25T05:36:37.721898+0000","mapping_epoch":88,"log_start":"95'1700","ondisk_log_start":"95'1700","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'5609","last_scrub_stamp":"2024-04-25T05:36:18.461345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.461345+0000","objects_scrubbed":5551,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:42.618808+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439222272,"num_objects":6702,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6702,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971121+0000","last_change":"2024-04-25T05:35:24.680523+0000","last_active":"2024-04-25T05:36:35.971121+0000","last_peered":"2024-04-25T05:36:35.971121+0000","last_clean":"2024-04-25T05:36:35.971121+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:36:35.971121+0000","last_undegraded":"2024-04-25T05:36:35.971121+0000","last_fullsized":"2024-04-25T05:36:35.971121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.680437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.414013+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976891+0000","last_change":"2024-04-25T05:36:14.472050+0000","last_active":"2024-04-25T05:36:35.976891+0000","last_peered":"2024-04-25T05:36:35.976891+0000","last_clean":"2024-04-25T05:36:35.976891+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:36:35.976891+0000","last_undegraded":"2024-04-25T05:36:35.976891+0000","last_fullsized":"2024-04-25T05:36:35.976891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:36:14.471895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.471895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:17.098974+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":179,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.970828+0000","last_change":"2024-04-25T05:36:17.062531+0000","last_active":"2024-04-25T05:36:35.970828+0000","last_peered":"2024-04-25T05:36:35.970828+0000","last_clean":"2024-04-25T05:36:35.970828+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:36:35.970828+0000","last_undegraded":"2024-04-25T05:36:35.970828+0000","last_fullsized":"2024-04-25T05:36:35.970828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:33.927079+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"118'9","reported_seq":175,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976490+0000","last_change":"2024-04-25T05:36:29.349021+0000","last_active":"2024-04-25T05:36:35.976490+0000","last_peered":"2024-04-25T05:36:35.976490+0000","last_clean":"2024-04-25T05:36:35.976490+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:36:35.976490+0000","last_undegraded":"2024-04-25T05:36:35.976490+0000","last_fullsized":"2024-04-25T05:36:35.976490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:41.385356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"124'6807","reported_seq":6888,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.785670+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:36:37.785670+0000","last_peered":"2024-04-25T05:36:37.785670+0000","last_clean":"2024-04-25T05:36:37.785670+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:36:37.785670+0000","last_undegraded":"2024-04-25T05:36:37.785670+0000","last_fullsized":"2024-04-25T05:36:37.785670+0000","mapping_epoch":88,"log_start":"96'2100","ondisk_log_start":"96'2100","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":20421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974459+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:36:35.974459+0000","last_peered":"2024-04-25T05:36:35.974459+0000","last_clean":"2024-04-25T05:36:35.974459+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:36:35.974459+0000","last_undegraded":"2024-04-25T05:36:35.974459+0000","last_fullsized":"2024-04-25T05:36:35.974459+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.977188+0000","last_change":"2024-04-25T05:35:02.907524+0000","last_active":"2024-04-25T05:36:35.977188+0000","last_peered":"2024-04-25T05:36:35.977188+0000","last_clean":"2024-04-25T05:36:35.977188+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:36:35.977188+0000","last_undegraded":"2024-04-25T05:36:35.977188+0000","last_fullsized":"2024-04-25T05:36:35.977188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:02.907405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.907405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.386050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":176,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945459+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:36:35.945459+0000","last_peered":"2024-04-25T05:36:35.945459+0000","last_clean":"2024-04-25T05:36:35.945459+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:36:35.945459+0000","last_undegraded":"2024-04-25T05:36:35.945459+0000","last_fullsized":"2024-04-25T05:36:35.945459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.139950+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.967703+0000","last_change":"2024-04-25T05:35:31.652458+0000","last_active":"2024-04-25T05:36:35.967703+0000","last_peered":"2024-04-25T05:36:35.967703+0000","last_clean":"2024-04-25T05:36:35.967703+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:36:35.967703+0000","last_undegraded":"2024-04-25T05:36:35.967703+0000","last_fullsized":"2024-04-25T05:36:35.967703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:31.652381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:31.652381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.460018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"124'6795","reported_seq":6877,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.815581+0000","last_change":"2024-04-25T05:35:09.681598+0000","last_active":"2024-04-25T05:36:37.815581+0000","last_peered":"2024-04-25T05:36:37.815581+0000","last_clean":"2024-04-25T05:36:37.815581+0000","last_became_active":"2024-04-25T05:35:09.681316+0000","last_became_peered":"2024-04-25T05:35:09.681316+0000","last_unstale":"2024-04-25T05:36:37.815581+0000","last_undegraded":"2024-04-25T05:36:37.815581+0000","last_fullsized":"2024-04-25T05:36:37.815581+0000","mapping_epoch":88,"log_start":"95'1200","ondisk_log_start":"95'1200","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5595,"log_dups_size":0,"ondisk_log_size":5595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.149133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445317120,"num_objects":6795,"num_object_clones":0,"num_object_copies":20385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975581+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:36:35.975581+0000","last_peered":"2024-04-25T05:36:35.975581+0000","last_clean":"2024-04-25T05:36:35.975581+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:36:35.975581+0000","last_undegraded":"2024-04-25T05:36:35.975581+0000","last_fullsized":"2024-04-25T05:36:35.975581+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.897023+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.970153+0000","last_change":"2024-04-25T05:35:27.565332+0000","last_active":"2024-04-25T05:36:35.970153+0000","last_peered":"2024-04-25T05:36:35.970153+0000","last_clean":"2024-04-25T05:36:35.970153+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:36:35.970153+0000","last_undegraded":"2024-04-25T05:36:35.970153+0000","last_fullsized":"2024-04-25T05:36:35.970153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.565209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.565209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.931166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":148,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.977238+0000","last_change":"2024-04-25T05:36:30.395761+0000","last_active":"2024-04-25T05:36:35.977238+0000","last_peered":"2024-04-25T05:36:35.977238+0000","last_clean":"2024-04-25T05:36:35.977238+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:36:35.977238+0000","last_undegraded":"2024-04-25T05:36:35.977238+0000","last_fullsized":"2024-04-25T05:36:35.977238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:38.353889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":173,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974582+0000","last_change":"2024-04-25T05:35:32.693459+0000","last_active":"2024-04-25T05:36:35.974582+0000","last_peered":"2024-04-25T05:36:35.974582+0000","last_clean":"2024-04-25T05:36:35.974582+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:36:35.974582+0000","last_undegraded":"2024-04-25T05:36:35.974582+0000","last_fullsized":"2024-04-25T05:36:35.974582+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.919493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"124'6688","reported_seq":6840,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.681136+0000","last_change":"2024-04-25T05:36:31.742804+0000","last_active":"2024-04-25T05:36:37.681136+0000","last_peered":"2024-04-25T05:36:37.681136+0000","last_clean":"2024-04-25T05:36:37.681136+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:36:37.681136+0000","last_undegraded":"2024-04-25T05:36:37.681136+0000","last_fullsized":"2024-04-25T05:36:37.681136+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6352","last_scrub_stamp":"2024-04-25T05:36:31.742451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.742451+0000","objects_scrubbed":6298,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:58.047202+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438304768,"num_objects":6688,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":428032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972096+0000","last_change":"2024-04-25T05:36:27.172930+0000","last_active":"2024-04-25T05:36:35.972096+0000","last_peered":"2024-04-25T05:36:35.972096+0000","last_clean":"2024-04-25T05:36:35.972096+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:36:35.972096+0000","last_undegraded":"2024-04-25T05:36:35.972096+0000","last_fullsized":"2024-04-25T05:36:35.972096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:27.172809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.172809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.542287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.968322+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:36:35.968322+0000","last_peered":"2024-04-25T05:36:35.968322+0000","last_clean":"2024-04-25T05:36:35.968322+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:36:35.968322+0000","last_undegraded":"2024-04-25T05:36:35.968322+0000","last_fullsized":"2024-04-25T05:36:35.968322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.188154+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971439+0000","last_change":"2024-04-25T05:35:34.751017+0000","last_active":"2024-04-25T05:36:35.971439+0000","last_peered":"2024-04-25T05:36:35.971439+0000","last_clean":"2024-04-25T05:36:35.971439+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:36:35.971439+0000","last_undegraded":"2024-04-25T05:36:35.971439+0000","last_fullsized":"2024-04-25T05:36:35.971439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.750887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.750887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.517305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973138+0000","last_change":"2024-04-25T05:35:40.352567+0000","last_active":"2024-04-25T05:36:35.973138+0000","last_peered":"2024-04-25T05:36:35.973138+0000","last_clean":"2024-04-25T05:36:35.973138+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:36:35.973138+0000","last_undegraded":"2024-04-25T05:36:35.973138+0000","last_fullsized":"2024-04-25T05:36:35.973138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:01.266360+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"124'6855","reported_seq":6937,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.713182+0000","last_change":"2024-04-25T05:35:09.684480+0000","last_active":"2024-04-25T05:36:37.713182+0000","last_peered":"2024-04-25T05:36:37.713182+0000","last_clean":"2024-04-25T05:36:37.713182+0000","last_became_active":"2024-04-25T05:35:09.683860+0000","last_became_peered":"2024-04-25T05:35:09.683860+0000","last_unstale":"2024-04-25T05:36:37.713182+0000","last_undegraded":"2024-04-25T05:36:37.713182+0000","last_fullsized":"2024-04-25T05:36:37.713182+0000","mapping_epoch":88,"log_start":"93'600","ondisk_log_start":"93'600","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.916776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449249280,"num_objects":6855,"num_object_clones":0,"num_object_copies":20565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975068+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:36:35.975068+0000","last_peered":"2024-04-25T05:36:35.975068+0000","last_clean":"2024-04-25T05:36:35.975068+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:36:35.975068+0000","last_undegraded":"2024-04-25T05:36:35.975068+0000","last_fullsized":"2024-04-25T05:36:35.975068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.505234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":144,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971000+0000","last_change":"2024-04-25T05:36:21.033702+0000","last_active":"2024-04-25T05:36:35.971000+0000","last_peered":"2024-04-25T05:36:35.971000+0000","last_clean":"2024-04-25T05:36:35.971000+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:36:35.971000+0000","last_undegraded":"2024-04-25T05:36:35.971000+0000","last_fullsized":"2024-04-25T05:36:35.971000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:45.885604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945362+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:36:35.945362+0000","last_peered":"2024-04-25T05:36:35.945362+0000","last_clean":"2024-04-25T05:36:35.945362+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:36:35.945362+0000","last_undegraded":"2024-04-25T05:36:35.945362+0000","last_fullsized":"2024-04-25T05:36:35.945362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":233,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976851+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:36:35.976851+0000","last_peered":"2024-04-25T05:36:35.976851+0000","last_clean":"2024-04-25T05:36:35.976851+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:36:35.976851+0000","last_undegraded":"2024-04-25T05:36:35.976851+0000","last_fullsized":"2024-04-25T05:36:35.976851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.463394+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972975+0000","last_change":"2024-04-25T05:36:24.214089+0000","last_active":"2024-04-25T05:36:35.972975+0000","last_peered":"2024-04-25T05:36:35.972975+0000","last_clean":"2024-04-25T05:36:35.972975+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:36:35.972975+0000","last_undegraded":"2024-04-25T05:36:35.972975+0000","last_fullsized":"2024-04-25T05:36:35.972975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.446616+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"124'6768","reported_seq":6850,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.760818+0000","last_change":"2024-04-25T05:35:09.679881+0000","last_active":"2024-04-25T05:36:37.760818+0000","last_peered":"2024-04-25T05:36:37.760818+0000","last_clean":"2024-04-25T05:36:37.760818+0000","last_became_active":"2024-04-25T05:35:09.679580+0000","last_became_peered":"2024-04-25T05:35:09.679580+0000","last_unstale":"2024-04-25T05:36:37.760818+0000","last_undegraded":"2024-04-25T05:36:37.760818+0000","last_fullsized":"2024-04-25T05:36:37.760818+0000","mapping_epoch":88,"log_start":"95'1200","ondisk_log_start":"95'1200","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.149195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443547648,"num_objects":6768,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.977392+0000","last_change":"2024-04-25T05:35:34.717443+0000","last_active":"2024-04-25T05:36:35.977392+0000","last_peered":"2024-04-25T05:36:35.977392+0000","last_clean":"2024-04-25T05:36:35.977392+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:36:35.977392+0000","last_undegraded":"2024-04-25T05:36:35.977392+0000","last_fullsized":"2024-04-25T05:36:35.977392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.717213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.717213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.918493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.979227+0000","last_change":"2024-04-25T05:35:40.340205+0000","last_active":"2024-04-25T05:36:35.979227+0000","last_peered":"2024-04-25T05:36:35.979227+0000","last_clean":"2024-04-25T05:36:35.979227+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:36:35.979227+0000","last_undegraded":"2024-04-25T05:36:35.979227+0000","last_fullsized":"2024-04-25T05:36:35.979227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:40.339655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.339655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:01.210259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":149,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.967875+0000","last_change":"2024-04-25T05:36:16.674171+0000","last_active":"2024-04-25T05:36:35.967875+0000","last_peered":"2024-04-25T05:36:35.967875+0000","last_clean":"2024-04-25T05:36:35.967875+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:36:35.967875+0000","last_undegraded":"2024-04-25T05:36:35.967875+0000","last_fullsized":"2024-04-25T05:36:35.967875+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:27.824707+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":175,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976301+0000","last_change":"2024-04-25T05:35:39.312001+0000","last_active":"2024-04-25T05:36:35.976301+0000","last_peered":"2024-04-25T05:36:35.976301+0000","last_clean":"2024-04-25T05:36:35.976301+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:36:35.976301+0000","last_undegraded":"2024-04-25T05:36:35.976301+0000","last_fullsized":"2024-04-25T05:36:35.976301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:06.200939+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.969278+0000","last_change":"2024-04-25T05:35:42.672247+0000","last_active":"2024-04-25T05:36:35.969278+0000","last_peered":"2024-04-25T05:36:35.969278+0000","last_clean":"2024-04-25T05:36:35.969278+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:36:35.969278+0000","last_undegraded":"2024-04-25T05:36:35.969278+0000","last_fullsized":"2024-04-25T05:36:35.969278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.672114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.672114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:51.402270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"124'6730","reported_seq":6812,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.809814+0000","last_change":"2024-04-25T05:35:09.683951+0000","last_active":"2024-04-25T05:36:37.809814+0000","last_peered":"2024-04-25T05:36:37.809814+0000","last_clean":"2024-04-25T05:36:37.809814+0000","last_became_active":"2024-04-25T05:35:09.682815+0000","last_became_peered":"2024-04-25T05:35:09.682815+0000","last_unstale":"2024-04-25T05:36:37.809814+0000","last_undegraded":"2024-04-25T05:36:37.809814+0000","last_fullsized":"2024-04-25T05:36:37.809814+0000","mapping_epoch":88,"log_start":"94'1100","ondisk_log_start":"94'1100","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441057280,"num_objects":6730,"num_object_clones":0,"num_object_copies":20190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":430720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971864+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:36:35.971864+0000","last_peered":"2024-04-25T05:36:35.971864+0000","last_clean":"2024-04-25T05:36:35.971864+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:36:35.971864+0000","last_undegraded":"2024-04-25T05:36:35.971864+0000","last_fullsized":"2024-04-25T05:36:35.971864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.990922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":385,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976496+0000","last_change":"2024-04-25T05:36:28.364094+0000","last_active":"2024-04-25T05:36:35.976496+0000","last_peered":"2024-04-25T05:36:35.976496+0000","last_clean":"2024-04-25T05:36:35.976496+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:36:35.976496+0000","last_undegraded":"2024-04-25T05:36:35.976496+0000","last_fullsized":"2024-04-25T05:36:35.976496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:36:28.364006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.364006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:47.701186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974361+0000","last_change":"2024-04-25T05:36:26.336148+0000","last_active":"2024-04-25T05:36:35.974361+0000","last_peered":"2024-04-25T05:36:35.974361+0000","last_clean":"2024-04-25T05:36:35.974361+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:36:35.974361+0000","last_undegraded":"2024-04-25T05:36:35.974361+0000","last_fullsized":"2024-04-25T05:36:35.974361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:40.326541+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":184,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975192+0000","last_change":"2024-04-25T05:35:27.678182+0000","last_active":"2024-04-25T05:36:35.975192+0000","last_peered":"2024-04-25T05:36:35.975192+0000","last_clean":"2024-04-25T05:36:35.975192+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:36:35.975192+0000","last_undegraded":"2024-04-25T05:36:35.975192+0000","last_fullsized":"2024-04-25T05:36:35.975192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:27.678020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.678020+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.087083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976842+0000","last_change":"2024-04-25T05:35:29.726834+0000","last_active":"2024-04-25T05:36:35.976842+0000","last_peered":"2024-04-25T05:36:35.976842+0000","last_clean":"2024-04-25T05:36:35.976842+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:36:35.976842+0000","last_undegraded":"2024-04-25T05:36:35.976842+0000","last_fullsized":"2024-04-25T05:36:35.976842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:29.726669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.726669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:52.548355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"124'6771","reported_seq":6853,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.640375+0000","last_change":"2024-04-25T05:35:09.683621+0000","last_active":"2024-04-25T05:36:37.640375+0000","last_peered":"2024-04-25T05:36:37.640375+0000","last_clean":"2024-04-25T05:36:37.640375+0000","last_became_active":"2024-04-25T05:35:09.683250+0000","last_became_peered":"2024-04-25T05:35:09.683250+0000","last_unstale":"2024-04-25T05:36:37.640375+0000","last_undegraded":"2024-04-25T05:36:37.640375+0000","last_fullsized":"2024-04-25T05:36:37.640375+0000","mapping_epoch":88,"log_start":"95'1500","ondisk_log_start":"95'1500","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.387069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443744256,"num_objects":6771,"num_object_clones":0,"num_object_copies":20313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":433344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976472+0000","last_change":"2024-04-25T05:35:38.262254+0000","last_active":"2024-04-25T05:36:35.976472+0000","last_peered":"2024-04-25T05:36:35.976472+0000","last_clean":"2024-04-25T05:36:35.976472+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:36:35.976472+0000","last_undegraded":"2024-04-25T05:36:35.976472+0000","last_fullsized":"2024-04-25T05:36:35.976472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.261996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.261996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.492794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972328+0000","last_change":"2024-04-25T05:36:26.153204+0000","last_active":"2024-04-25T05:36:35.972328+0000","last_peered":"2024-04-25T05:36:35.972328+0000","last_clean":"2024-04-25T05:36:35.972328+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:36:35.972328+0000","last_undegraded":"2024-04-25T05:36:35.972328+0000","last_fullsized":"2024-04-25T05:36:35.972328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.991917+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":298,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975733+0000","last_change":"2024-04-25T05:36:20.496553+0000","last_active":"2024-04-25T05:36:35.975733+0000","last_peered":"2024-04-25T05:36:35.975733+0000","last_clean":"2024-04-25T05:36:35.975733+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:36:35.975733+0000","last_undegraded":"2024-04-25T05:36:35.975733+0000","last_fullsized":"2024-04-25T05:36:35.975733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:36:20.496280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.496280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:30.974274+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976033+0000","last_change":"2024-04-25T05:36:32.152204+0000","last_active":"2024-04-25T05:36:35.976033+0000","last_peered":"2024-04-25T05:36:35.976033+0000","last_clean":"2024-04-25T05:36:35.976033+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:36:35.976033+0000","last_undegraded":"2024-04-25T05:36:35.976033+0000","last_fullsized":"2024-04-25T05:36:35.976033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.408462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972275+0000","last_change":"2024-04-25T05:35:35.758618+0000","last_active":"2024-04-25T05:36:35.972275+0000","last_peered":"2024-04-25T05:36:35.972275+0000","last_clean":"2024-04-25T05:36:35.972275+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:36:35.972275+0000","last_undegraded":"2024-04-25T05:36:35.972275+0000","last_fullsized":"2024-04-25T05:36:35.972275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.758501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.758501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:37.780937+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"124'6779","reported_seq":6861,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.669387+0000","last_change":"2024-04-25T05:35:09.676826+0000","last_active":"2024-04-25T05:36:37.669387+0000","last_peered":"2024-04-25T05:36:37.669387+0000","last_clean":"2024-04-25T05:36:37.669387+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:36:37.669387+0000","last_undegraded":"2024-04-25T05:36:37.669387+0000","last_fullsized":"2024-04-25T05:36:37.669387+0000","mapping_epoch":88,"log_start":"94'1000","ondisk_log_start":"94'1000","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.987912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444268544,"num_objects":6779,"num_object_clones":0,"num_object_copies":20337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":433856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974453+0000","last_change":"2024-04-25T05:35:35.237306+0000","last_active":"2024-04-25T05:36:35.974453+0000","last_peered":"2024-04-25T05:36:35.974453+0000","last_clean":"2024-04-25T05:36:35.974453+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:36:35.974453+0000","last_undegraded":"2024-04-25T05:36:35.974453+0000","last_fullsized":"2024-04-25T05:36:35.974453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.341015+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976737+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:36:35.976737+0000","last_peered":"2024-04-25T05:36:35.976737+0000","last_clean":"2024-04-25T05:36:35.976737+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:36:35.976737+0000","last_undegraded":"2024-04-25T05:36:35.976737+0000","last_fullsized":"2024-04-25T05:36:35.976737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.664326+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":178,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972324+0000","last_change":"2024-04-25T05:35:35.249157+0000","last_active":"2024-04-25T05:36:35.972324+0000","last_peered":"2024-04-25T05:36:35.972324+0000","last_clean":"2024-04-25T05:36:35.972324+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:36:35.972324+0000","last_undegraded":"2024-04-25T05:36:35.972324+0000","last_fullsized":"2024-04-25T05:36:35.972324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:35.248443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.248443+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.985228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974671+0000","last_change":"2024-04-25T05:36:21.964578+0000","last_active":"2024-04-25T05:36:35.974671+0000","last_peered":"2024-04-25T05:36:35.974671+0000","last_clean":"2024-04-25T05:36:35.974671+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:36:35.974671+0000","last_undegraded":"2024-04-25T05:36:35.974671+0000","last_fullsized":"2024-04-25T05:36:35.974671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:48.818196+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"124'6880","reported_seq":6961,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.674497+0000","last_change":"2024-04-25T05:35:09.680036+0000","last_active":"2024-04-25T05:36:37.674497+0000","last_peered":"2024-04-25T05:36:37.674497+0000","last_clean":"2024-04-25T05:36:37.674497+0000","last_became_active":"2024-04-25T05:35:09.679390+0000","last_became_peered":"2024-04-25T05:35:09.679390+0000","last_unstale":"2024-04-25T05:36:37.674497+0000","last_undegraded":"2024-04-25T05:36:37.674497+0000","last_fullsized":"2024-04-25T05:36:37.674497+0000","mapping_epoch":88,"log_start":"96'2100","ondisk_log_start":"96'2100","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450887680,"num_objects":6880,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":440320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972384+0000","last_change":"2024-04-25T05:35:44.317463+0000","last_active":"2024-04-25T05:36:35.972384+0000","last_peered":"2024-04-25T05:36:35.972384+0000","last_clean":"2024-04-25T05:36:35.972384+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:36:35.972384+0000","last_undegraded":"2024-04-25T05:36:35.972384+0000","last_fullsized":"2024-04-25T05:36:35.972384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:09.893502+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971926+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:36:35.971926+0000","last_peered":"2024-04-25T05:36:35.971926+0000","last_clean":"2024-04-25T05:36:35.971926+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:36:35.971926+0000","last_undegraded":"2024-04-25T05:36:35.971926+0000","last_fullsized":"2024-04-25T05:36:35.971926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:35.016206+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":87,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.968039+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:36:35.968039+0000","last_peered":"2024-04-25T05:36:35.968039+0000","last_clean":"2024-04-25T05:36:35.968039+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:36:35.968039+0000","last_undegraded":"2024-04-25T05:36:35.968039+0000","last_fullsized":"2024-04-25T05:36:35.968039+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:55.713346+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973248+0000","last_change":"2024-04-25T05:35:42.311228+0000","last_active":"2024-04-25T05:36:35.973248+0000","last_peered":"2024-04-25T05:36:35.973248+0000","last_clean":"2024-04-25T05:36:35.973248+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:36:35.973248+0000","last_undegraded":"2024-04-25T05:36:35.973248+0000","last_fullsized":"2024-04-25T05:36:35.973248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:42.311077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.311077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.843077+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"124'6845","reported_seq":6927,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.730210+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:36:37.730210+0000","last_peered":"2024-04-25T05:36:37.730210+0000","last_clean":"2024-04-25T05:36:37.730210+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:36:37.730210+0000","last_undegraded":"2024-04-25T05:36:37.730210+0000","last_fullsized":"2024-04-25T05:36:37.730210+0000","mapping_epoch":88,"log_start":"97'2200","ondisk_log_start":"97'2200","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448593920,"num_objects":6845,"num_object_clones":0,"num_object_copies":20535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":438080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976361+0000","last_change":"2024-04-25T05:35:35.701192+0000","last_active":"2024-04-25T05:36:35.976361+0000","last_peered":"2024-04-25T05:36:35.976361+0000","last_clean":"2024-04-25T05:36:35.976361+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:36:35.976361+0000","last_undegraded":"2024-04-25T05:36:35.976361+0000","last_fullsized":"2024-04-25T05:36:35.976361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.701074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.701074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.222127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":136,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976119+0000","last_change":"2024-04-25T05:35:36.294561+0000","last_active":"2024-04-25T05:36:35.976119+0000","last_peered":"2024-04-25T05:36:35.976119+0000","last_clean":"2024-04-25T05:36:35.976119+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:36:35.976119+0000","last_undegraded":"2024-04-25T05:36:35.976119+0000","last_fullsized":"2024-04-25T05:36:35.976119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.239247+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945312+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:36:35.945312+0000","last_peered":"2024-04-25T05:36:35.945312+0000","last_clean":"2024-04-25T05:36:35.945312+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:36:35.945312+0000","last_undegraded":"2024-04-25T05:36:35.945312+0000","last_fullsized":"2024-04-25T05:36:35.945312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972574+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:36:35.972574+0000","last_peered":"2024-04-25T05:36:35.972574+0000","last_clean":"2024-04-25T05:36:35.972574+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:36:35.972574+0000","last_undegraded":"2024-04-25T05:36:35.972574+0000","last_fullsized":"2024-04-25T05:36:35.972574+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:43.089867+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"124'6788","reported_seq":6870,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.770029+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:36:37.770029+0000","last_peered":"2024-04-25T05:36:37.770029+0000","last_clean":"2024-04-25T05:36:37.770029+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:36:37.770029+0000","last_undegraded":"2024-04-25T05:36:37.770029+0000","last_fullsized":"2024-04-25T05:36:37.770029+0000","mapping_epoch":88,"log_start":"97'2100","ondisk_log_start":"97'2100","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444858368,"num_objects":6788,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945360+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:36:35.945360+0000","last_peered":"2024-04-25T05:36:35.945360+0000","last_clean":"2024-04-25T05:36:35.945360+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:36:35.945360+0000","last_undegraded":"2024-04-25T05:36:35.945360+0000","last_fullsized":"2024-04-25T05:36:35.945360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.155991+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.975502+0000","last_change":"2024-04-25T05:35:06.377537+0000","last_active":"2024-04-25T05:36:35.975502+0000","last_peered":"2024-04-25T05:36:35.975502+0000","last_clean":"2024-04-25T05:36:35.975502+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:36:35.975502+0000","last_undegraded":"2024-04-25T05:36:35.975502+0000","last_fullsized":"2024-04-25T05:36:35.975502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":186,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972437+0000","last_change":"2024-04-25T05:35:30.243679+0000","last_active":"2024-04-25T05:36:35.972437+0000","last_peered":"2024-04-25T05:36:35.972437+0000","last_clean":"2024-04-25T05:36:35.972437+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:36:35.972437+0000","last_undegraded":"2024-04-25T05:36:35.972437+0000","last_fullsized":"2024-04-25T05:36:35.972437+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.227823+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945145+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:36:35.945145+0000","last_peered":"2024-04-25T05:36:35.945145+0000","last_clean":"2024-04-25T05:36:35.945145+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:36:35.945145+0000","last_undegraded":"2024-04-25T05:36:35.945145+0000","last_fullsized":"2024-04-25T05:36:35.945145+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:37.962428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"124'6723","reported_seq":6805,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.697208+0000","last_change":"2024-04-25T05:35:09.678603+0000","last_active":"2024-04-25T05:36:37.697208+0000","last_peered":"2024-04-25T05:36:37.697208+0000","last_clean":"2024-04-25T05:36:37.697208+0000","last_became_active":"2024-04-25T05:35:09.678239+0000","last_became_peered":"2024-04-25T05:35:09.678239+0000","last_unstale":"2024-04-25T05:36:37.697208+0000","last_undegraded":"2024-04-25T05:36:37.697208+0000","last_fullsized":"2024-04-25T05:36:37.697208+0000","mapping_epoch":88,"log_start":"93'400","ondisk_log_start":"93'400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440598528,"num_objects":6723,"num_object_clones":0,"num_object_copies":20169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":430272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.971645+0000","last_change":"2024-04-25T05:35:27.774757+0000","last_active":"2024-04-25T05:36:35.971645+0000","last_peered":"2024-04-25T05:36:35.971645+0000","last_clean":"2024-04-25T05:36:35.971645+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:36:35.971645+0000","last_undegraded":"2024-04-25T05:36:35.971645+0000","last_fullsized":"2024-04-25T05:36:35.971645+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.774595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.774595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946166+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:36:35.946166+0000","last_peered":"2024-04-25T05:36:35.946166+0000","last_clean":"2024-04-25T05:36:35.946166+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:36:35.946166+0000","last_undegraded":"2024-04-25T05:36:35.946166+0000","last_fullsized":"2024-04-25T05:36:35.946166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.630225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972362+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:36:35.972362+0000","last_peered":"2024-04-25T05:36:35.972362+0000","last_clean":"2024-04-25T05:36:35.972362+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:36:35.972362+0000","last_undegraded":"2024-04-25T05:36:35.972362+0000","last_fullsized":"2024-04-25T05:36:35.972362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.338804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":155,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976211+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:36:35.976211+0000","last_peered":"2024-04-25T05:36:35.976211+0000","last_clean":"2024-04-25T05:36:35.976211+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:36:35.976211+0000","last_undegraded":"2024-04-25T05:36:35.976211+0000","last_fullsized":"2024-04-25T05:36:35.976211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:55.776165+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"124'6839","reported_seq":6921,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.717436+0000","last_change":"2024-04-25T05:35:09.684061+0000","last_active":"2024-04-25T05:36:37.717436+0000","last_peered":"2024-04-25T05:36:37.717436+0000","last_clean":"2024-04-25T05:36:37.717436+0000","last_became_active":"2024-04-25T05:35:09.683699+0000","last_became_peered":"2024-04-25T05:35:09.683699+0000","last_unstale":"2024-04-25T05:36:37.717436+0000","last_undegraded":"2024-04-25T05:36:37.717436+0000","last_fullsized":"2024-04-25T05:36:37.717436+0000","mapping_epoch":88,"log_start":"93'500","ondisk_log_start":"93'500","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6339,"log_dups_size":0,"ondisk_log_size":6339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:39.916689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448200704,"num_objects":6839,"num_object_clones":0,"num_object_copies":20517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6839,"num_write_kb":437696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973671+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:36:35.973671+0000","last_peered":"2024-04-25T05:36:35.973671+0000","last_clean":"2024-04-25T05:36:35.973671+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:36:35.973671+0000","last_undegraded":"2024-04-25T05:36:35.973671+0000","last_fullsized":"2024-04-25T05:36:35.973671+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.972198+0000","last_change":"2024-04-25T05:35:25.709565+0000","last_active":"2024-04-25T05:36:35.972198+0000","last_peered":"2024-04-25T05:36:35.972198+0000","last_clean":"2024-04-25T05:36:35.972198+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:36:35.972198+0000","last_undegraded":"2024-04-25T05:36:35.972198+0000","last_fullsized":"2024-04-25T05:36:35.972198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:25.709444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.709444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:37.833046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945180+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:36:35.945180+0000","last_peered":"2024-04-25T05:36:35.945180+0000","last_clean":"2024-04-25T05:36:35.945180+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:36:35.945180+0000","last_undegraded":"2024-04-25T05:36:35.945180+0000","last_fullsized":"2024-04-25T05:36:35.945180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:48.743618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":160,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946074+0000","last_change":"2024-04-25T05:36:15.740901+0000","last_active":"2024-04-25T05:36:35.946074+0000","last_peered":"2024-04-25T05:36:35.946074+0000","last_clean":"2024-04-25T05:36:35.946074+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:36:35.946074+0000","last_undegraded":"2024-04-25T05:36:35.946074+0000","last_fullsized":"2024-04-25T05:36:35.946074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:23.810446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"124'6775","reported_seq":6918,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.708685+0000","last_change":"2024-04-25T05:36:32.972424+0000","last_active":"2024-04-25T05:36:37.708685+0000","last_peered":"2024-04-25T05:36:37.708685+0000","last_clean":"2024-04-25T05:36:37.708685+0000","last_became_active":"2024-04-25T05:35:09.678971+0000","last_became_peered":"2024-04-25T05:35:09.678971+0000","last_unstale":"2024-04-25T05:36:37.708685+0000","last_undegraded":"2024-04-25T05:36:37.708685+0000","last_fullsized":"2024-04-25T05:36:37.708685+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":5549,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:37.944442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444006400,"num_objects":6775,"num_object_clones":0,"num_object_copies":20325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.969197+0000","last_change":"2024-04-25T05:35:39.752489+0000","last_active":"2024-04-25T05:36:35.969197+0000","last_peered":"2024-04-25T05:36:35.969197+0000","last_clean":"2024-04-25T05:36:35.969197+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:36:35.969197+0000","last_undegraded":"2024-04-25T05:36:35.969197+0000","last_fullsized":"2024-04-25T05:36:35.969197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:39.752319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.752319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:54.553556+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.976996+0000","last_change":"2024-04-25T05:35:33.279820+0000","last_active":"2024-04-25T05:36:35.976996+0000","last_peered":"2024-04-25T05:36:35.976996+0000","last_clean":"2024-04-25T05:36:35.976996+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:36:35.976996+0000","last_undegraded":"2024-04-25T05:36:35.976996+0000","last_fullsized":"2024-04-25T05:36:35.976996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:33.279557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:33.279557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.974330+0000","last_change":"2024-04-25T05:36:31.976928+0000","last_active":"2024-04-25T05:36:35.974330+0000","last_peered":"2024-04-25T05:36:35.974330+0000","last_clean":"2024-04-25T05:36:35.974330+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:36:35.974330+0000","last_undegraded":"2024-04-25T05:36:35.974330+0000","last_fullsized":"2024-04-25T05:36:35.974330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:37.473760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946541+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:36:35.946541+0000","last_peered":"2024-04-25T05:36:35.946541+0000","last_clean":"2024-04-25T05:36:35.946541+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:36:35.946541+0000","last_undegraded":"2024-04-25T05:36:35.946541+0000","last_fullsized":"2024-04-25T05:36:35.946541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.955851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"124'6999","reported_seq":7081,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:37.822692+0000","last_change":"2024-04-25T05:35:09.682779+0000","last_active":"2024-04-25T05:36:37.822692+0000","last_peered":"2024-04-25T05:36:37.822692+0000","last_clean":"2024-04-25T05:36:37.822692+0000","last_became_active":"2024-04-25T05:35:09.681887+0000","last_became_peered":"2024-04-25T05:35:09.681887+0000","last_unstale":"2024-04-25T05:36:37.822692+0000","last_undegraded":"2024-04-25T05:36:37.822692+0000","last_fullsized":"2024-04-25T05:36:37.822692+0000","mapping_epoch":88,"log_start":"95'1400","ondisk_log_start":"95'1400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458686464,"num_objects":6999,"num_object_clones":0,"num_object_copies":20997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.946229+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:36:35.946229+0000","last_peered":"2024-04-25T05:36:35.946229+0000","last_clean":"2024-04-25T05:36:35.946229+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:36:35.946229+0000","last_undegraded":"2024-04-25T05:36:35.946229+0000","last_fullsized":"2024-04-25T05:36:35.946229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.276359+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":169,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.968964+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:36:35.968964+0000","last_peered":"2024-04-25T05:36:35.968964+0000","last_clean":"2024-04-25T05:36:35.968964+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:36:35.968964+0000","last_undegraded":"2024-04-25T05:36:35.968964+0000","last_fullsized":"2024-04-25T05:36:35.968964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.813928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.973227+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:36:35.973227+0000","last_peered":"2024-04-25T05:36:35.973227+0000","last_clean":"2024-04-25T05:36:35.973227+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:36:35.973227+0000","last_undegraded":"2024-04-25T05:36:35.973227+0000","last_fullsized":"2024-04-25T05:36:35.973227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.450067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":144,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-04-25T05:36:35.945958+0000","last_change":"2024-04-25T05:36:30.640999+0000","last_active":"2024-04-25T05:36:35.945958+0000","last_peered":"2024-04-25T05:36:35.945958+0000","last_clean":"2024-04-25T05:36:35.945958+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:36:35.945958+0000","last_undegraded":"2024-04-25T05:36:35.945958+0000","last_fullsized":"2024-04-25T05:36:35.945958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.312587+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7126646784,"num_objects":108744,"num_object_clones":0,"num_object_copies":326232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108744,"num_write_kb":6959616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21378891776,"data_stored":21378891776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89644,"ondisk_log_size":89644,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070501,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230280,"kb_used_data":2163004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91514872,"statfs":{"total":95995035648,"available":93711228928,"internally_reserved":0,"allocated":2214916096,"data_stored":2214267785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364841,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228660,"kb_used_data":2161380,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91516492,"statfs":{"total":95995035648,"available":93712887808,"internally_reserved":0,"allocated":2213253120,"data_stored":2212628287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2659992,"kb_used_data":2584400,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91085160,"statfs":{"total":95995035648,"available":93271203840,"internally_reserved":0,"allocated":2646425600,"data_stored":2645755303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3560964,"kb_used_data":3471476,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":90184188,"statfs":{"total":95995035648,"available":92348608512,"internally_reserved":0,"allocated":3554791424,"data_stored":3554149572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2672484,"kb_used_data":2596892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91072668,"statfs":{"total":95995035648,"available":93258412032,"internally_reserved":0,"allocated":2659217408,"data_stored":2658599065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4026824,"kb_used_data":3919292,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":89718328,"statfs":{"total":95995035648,"available":91871567872,"internally_reserved":0,"allocated":4013355008,"data_stored":4012704517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2235716,"kb_used_data":2168444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91509436,"statfs":{"total":95995035648,"available":93705662464,"internally_reserved":0,"allocated":2220486656,"data_stored":2219836885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691886,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773676,"kb_used_data":1720292,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":91971476,"statfs":{"total":95995035648,"available":94178791424,"internally_reserved":0,"allocated":1761579008,"data_stored":1760953107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3575578624,"data_stored":3575578624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2668822528,"data_stored":2668822528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2665086976,"data_stored":2665086976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4025876480,"data_stored":4025876480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2221801472,"data_stored":2221801472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1772224512,"data_stored":1772224512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2228879360,"data_stored":2228879360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2220621824,"data_stored":2220621824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:36:44.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:36:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:44.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:44.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:44.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[27010]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:44.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:36:44.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:36:44.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[37492]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:44.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:36:44.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:44 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:36:44.414 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:44.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:44.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:44 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:44.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:44 smithi060 ceph-mon[30898]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:44 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:36:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:44 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:36:44.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:44.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:44.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:44.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 115110 115094 81.7178 51.75 0.0195934 0.0122292 2024-04-25T05:36:44.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:44.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:44.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:44.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:45.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:45.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:45.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:45.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:45 smithi043 ceph-mon[27010]: pgmap v117: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:45.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:45 smithi043 ceph-mon[27010]: osdmap e127: 8 total, 8 up, 8 in 2024-04-25T05:36:45.188 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:45 smithi043 ceph-mon[37492]: pgmap v117: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:45.188 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:45 smithi043 ceph-mon[37492]: osdmap e127: 8 total, 8 up, 8 in 2024-04-25T05:36:45.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:45.240 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:45 smithi060 ceph-mon[30898]: pgmap v117: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-25T05:36:45.240 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:45 smithi060 ceph-mon[30898]: osdmap e127: 8 total, 8 up, 8 in 2024-04-25T05:36:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:45.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:36:45.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:45.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:45.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:45.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:45.650 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:45.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:45.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:45.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 115975 115959 81.4071 54.0625 0.0140425 0.0122782 2024-04-25T05:36:45.784 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:45.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:45.829 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:45.852 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:45.872 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:45.913 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:45.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:45.955 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:45.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:45.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:45.999 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:46.043 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:46.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:46.109 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:46.123 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:46.124 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:46.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:46.148 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:36:46.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:46.271 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:46.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/564977030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.308 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/303112212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:46.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:46.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/564977030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/303112212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:46.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/564977030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:36:46.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/303112212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:46.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:46.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:46.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:46.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:46.673 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:46.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0098 2024-04-25T05:36:46.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 116866 2024-04-25T05:36:46.748 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 81.1481 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 30.1771 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 182.375 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 49.625 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1298 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 482.834 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 2918 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 794 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0123186 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.00906694 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 0.140459 2024-04-25T05:36:46.749 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00120772 2024-04-25T05:36:46.752 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-04-25T05:36:46.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:46.823 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:46.854 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:36:46.860 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:36:46.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:46.944 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:46.945 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:36:46.945 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:36:46.945 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:46.952 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:47.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:47.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:47.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.218 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:47.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:47.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:47.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:47.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:47.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:47.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: pgmap v119: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-04-25T05:36:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: osdmap e128: 8 total, 8 up, 8 in 2024-04-25T05:36:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:47.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:47.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: pgmap v119: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: osdmap e128: 8 total, 8 up, 8 in 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:47.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: pgmap v119: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 786 op/s 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: osdmap e128: 8 total, 8 up, 8 in 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:36:47.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:47.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:47.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:47.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:47.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:48.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:48.052 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:48.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:48.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:48 smithi060 ceph-mon[30898]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:48 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:36:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:48 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:36:48.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:48.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:36:48.318 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[27010]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:36:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:36:48.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[37492]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:36:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:36:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:48 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:36:48.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:48.472 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:48.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:48.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:48.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:48.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:48.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:48.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:48.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:49.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:49.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:49.208 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:49.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:49.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:49.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.431 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:49.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: pgmap v121: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 804 op/s 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:49.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: pgmap v121: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 804 op/s 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: pgmap v121: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 804 op/s 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:49.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:49.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:49.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:36:49.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:49.830 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-04-25T05:36:46.134839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7436311067,"num_objects":113690,"num_object_clones":0,"num_object_copies":341070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113690,"num_whiteouts":0,"num_read":1380,"num_read_kb":1194,"num_write":114267,"num_write_kb":7263901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90878,"ondisk_log_size":90878,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22315068,"kb_used_data":21707484,"kb_used_omap":12,"kb_used_meta":607475,"kb_avail":727646148,"statfs":{"total":767960285184,"available":745109655552,"internally_reserved":0,"allocated":22228463616,"data_stored":22223246393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":622054960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":309067776,"num_objects":4716,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4716,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4716,"num_write_kb":301824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.429137"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":117,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714466+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:36:40.714466+0000","last_peered":"2024-04-25T05:36:40.714466+0000","last_clean":"2024-04-25T05:36:40.714466+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:36:40.714466+0000","last_undegraded":"2024-04-25T05:36:40.714466+0000","last_fullsized":"2024-04-25T05:36:40.714466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733631+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:36:40.733631+0000","last_peered":"2024-04-25T05:36:40.733631+0000","last_clean":"2024-04-25T05:36:40.733631+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:36:40.733631+0000","last_undegraded":"2024-04-25T05:36:40.733631+0000","last_fullsized":"2024-04-25T05:36:40.733631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.859796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.718537+0000","last_change":"2024-04-25T05:36:18.176520+0000","last_active":"2024-04-25T05:36:40.718537+0000","last_peered":"2024-04-25T05:36:40.718537+0000","last_clean":"2024-04-25T05:36:40.718537+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:36:40.718537+0000","last_undegraded":"2024-04-25T05:36:40.718537+0000","last_fullsized":"2024-04-25T05:36:40.718537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.176394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.176394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:23.464747+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714438+0000","last_change":"2024-04-25T05:36:13.708900+0000","last_active":"2024-04-25T05:36:40.714438+0000","last_peered":"2024-04-25T05:36:40.714438+0000","last_clean":"2024-04-25T05:36:40.714438+0000","last_became_active":"2024-04-25T05:34:00.692268+0000","last_became_peered":"2024-04-25T05:34:00.692268+0000","last_unstale":"2024-04-25T05:36:40.714438+0000","last_undegraded":"2024-04-25T05:36:40.714438+0000","last_fullsized":"2024-04-25T05:36:40.714438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.170106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725034+0000","last_change":"2024-04-25T05:34:04.762220+0000","last_active":"2024-04-25T05:36:40.725034+0000","last_peered":"2024-04-25T05:36:40.725034+0000","last_clean":"2024-04-25T05:36:40.725034+0000","last_became_active":"2024-04-25T05:34:04.761285+0000","last_became_peered":"2024-04-25T05:34:04.761285+0000","last_unstale":"2024-04-25T05:36:40.725034+0000","last_undegraded":"2024-04-25T05:36:40.725034+0000","last_fullsized":"2024-04-25T05:36:40.725034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":121,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716897+0000","last_change":"2024-04-25T05:34:02.704030+0000","last_active":"2024-04-25T05:36:40.716897+0000","last_peered":"2024-04-25T05:36:40.716897+0000","last_clean":"2024-04-25T05:36:40.716897+0000","last_became_active":"2024-04-25T05:34:02.703661+0000","last_became_peered":"2024-04-25T05:34:02.703661+0000","last_unstale":"2024-04-25T05:36:40.716897+0000","last_undegraded":"2024-04-25T05:36:40.716897+0000","last_fullsized":"2024-04-25T05:36:40.716897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725867+0000","last_change":"2024-04-25T05:35:27.201991+0000","last_active":"2024-04-25T05:36:40.725867+0000","last_peered":"2024-04-25T05:36:40.725867+0000","last_clean":"2024-04-25T05:36:40.725867+0000","last_became_active":"2024-04-25T05:33:58.679443+0000","last_became_peered":"2024-04-25T05:33:58.679443+0000","last_unstale":"2024-04-25T05:36:40.725867+0000","last_undegraded":"2024-04-25T05:36:40.725867+0000","last_fullsized":"2024-04-25T05:36:40.725867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":199,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724374+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:36:40.724374+0000","last_peered":"2024-04-25T05:36:40.724374+0000","last_clean":"2024-04-25T05:36:40.724374+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:36:40.724374+0000","last_undegraded":"2024-04-25T05:36:40.724374+0000","last_fullsized":"2024-04-25T05:36:40.724374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716677+0000","last_change":"2024-04-25T05:35:32.284750+0000","last_active":"2024-04-25T05:36:40.716677+0000","last_peered":"2024-04-25T05:36:40.716677+0000","last_clean":"2024-04-25T05:36:40.716677+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:36:40.716677+0000","last_undegraded":"2024-04-25T05:36:40.716677+0000","last_fullsized":"2024-04-25T05:36:40.716677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.284637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.284637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:40.825789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728885+0000","last_change":"2024-04-25T05:36:29.981966+0000","last_active":"2024-04-25T05:36:40.728885+0000","last_peered":"2024-04-25T05:36:40.728885+0000","last_clean":"2024-04-25T05:36:40.728885+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:36:40.728885+0000","last_undegraded":"2024-04-25T05:36:40.728885+0000","last_fullsized":"2024-04-25T05:36:40.728885+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.093687+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728979+0000","last_change":"2024-04-25T05:36:31.998727+0000","last_active":"2024-04-25T05:36:40.728979+0000","last_peered":"2024-04-25T05:36:40.728979+0000","last_clean":"2024-04-25T05:36:40.728979+0000","last_became_active":"2024-04-25T05:33:58.687312+0000","last_became_peered":"2024-04-25T05:33:58.687312+0000","last_unstale":"2024-04-25T05:36:40.728979+0000","last_undegraded":"2024-04-25T05:36:40.728979+0000","last_fullsized":"2024-04-25T05:36:40.728979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.071661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":166,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714645+0000","last_change":"2024-04-25T05:34:00.689765+0000","last_active":"2024-04-25T05:36:40.714645+0000","last_peered":"2024-04-25T05:36:40.714645+0000","last_clean":"2024-04-25T05:36:40.714645+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:36:40.714645+0000","last_undegraded":"2024-04-25T05:36:40.714645+0000","last_fullsized":"2024-04-25T05:36:40.714645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.672983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728348+0000","last_change":"2024-04-25T05:35:32.732758+0000","last_active":"2024-04-25T05:36:40.728348+0000","last_peered":"2024-04-25T05:36:40.728348+0000","last_clean":"2024-04-25T05:36:40.728348+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:36:40.728348+0000","last_undegraded":"2024-04-25T05:36:40.728348+0000","last_fullsized":"2024-04-25T05:36:40.728348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.732662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.732662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:49.439622+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731500+0000","last_change":"2024-04-25T05:35:44.725188+0000","last_active":"2024-04-25T05:36:40.731500+0000","last_peered":"2024-04-25T05:36:40.731500+0000","last_clean":"2024-04-25T05:36:40.731500+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:36:40.731500+0000","last_undegraded":"2024-04-25T05:36:40.731500+0000","last_fullsized":"2024-04-25T05:36:40.731500+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.725019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.725019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:07.191021+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728883+0000","last_change":"2024-04-25T05:35:06.788446+0000","last_active":"2024-04-25T05:36:40.728883+0000","last_peered":"2024-04-25T05:36:40.728883+0000","last_clean":"2024-04-25T05:36:40.728883+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:36:40.728883+0000","last_undegraded":"2024-04-25T05:36:40.728883+0000","last_fullsized":"2024-04-25T05:36:40.728883+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.788358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.987932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":169,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715127+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:36:40.715127+0000","last_peered":"2024-04-25T05:36:40.715127+0000","last_clean":"2024-04-25T05:36:40.715127+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:36:40.715127+0000","last_undegraded":"2024-04-25T05:36:40.715127+0000","last_fullsized":"2024-04-25T05:36:40.715127+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.256093+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":117,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715192+0000","last_change":"2024-04-25T05:34:04.759052+0000","last_active":"2024-04-25T05:36:40.715192+0000","last_peered":"2024-04-25T05:36:40.715192+0000","last_clean":"2024-04-25T05:36:40.715192+0000","last_became_active":"2024-04-25T05:34:04.758718+0000","last_became_peered":"2024-04-25T05:34:04.758718+0000","last_unstale":"2024-04-25T05:36:40.715192+0000","last_undegraded":"2024-04-25T05:36:40.715192+0000","last_fullsized":"2024-04-25T05:36:40.715192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731762+0000","last_change":"2024-04-25T05:35:27.229584+0000","last_active":"2024-04-25T05:36:40.731762+0000","last_peered":"2024-04-25T05:36:40.731762+0000","last_clean":"2024-04-25T05:36:40.731762+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:36:40.731762+0000","last_undegraded":"2024-04-25T05:36:40.731762+0000","last_fullsized":"2024-04-25T05:36:40.731762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.229461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.229461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716704+0000","last_change":"2024-04-25T05:35:07.169298+0000","last_active":"2024-04-25T05:36:40.716704+0000","last_peered":"2024-04-25T05:36:40.716704+0000","last_clean":"2024-04-25T05:36:40.716704+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:36:40.716704+0000","last_undegraded":"2024-04-25T05:36:40.716704+0000","last_fullsized":"2024-04-25T05:36:40.716704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:07.169214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.169214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.722627+0000","last_change":"2024-04-25T05:35:28.590848+0000","last_active":"2024-04-25T05:36:40.722627+0000","last_peered":"2024-04-25T05:36:40.722627+0000","last_clean":"2024-04-25T05:36:40.722627+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:36:40.722627+0000","last_undegraded":"2024-04-25T05:36:40.722627+0000","last_fullsized":"2024-04-25T05:36:40.722627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:28.590759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.590759+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.469480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726923+0000","last_change":"2024-04-25T05:34:04.761591+0000","last_active":"2024-04-25T05:36:40.726923+0000","last_peered":"2024-04-25T05:36:40.726923+0000","last_clean":"2024-04-25T05:36:40.726923+0000","last_became_active":"2024-04-25T05:34:04.761155+0000","last_became_peered":"2024-04-25T05:34:04.761155+0000","last_unstale":"2024-04-25T05:36:40.726923+0000","last_undegraded":"2024-04-25T05:36:40.726923+0000","last_fullsized":"2024-04-25T05:36:40.726923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":121,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715962+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:36:40.715962+0000","last_peered":"2024-04-25T05:36:40.715962+0000","last_clean":"2024-04-25T05:36:40.715962+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:36:40.715962+0000","last_undegraded":"2024-04-25T05:36:40.715962+0000","last_fullsized":"2024-04-25T05:36:40.715962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714651+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:36:40.714651+0000","last_peered":"2024-04-25T05:36:40.714651+0000","last_clean":"2024-04-25T05:36:40.714651+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:36:40.714651+0000","last_undegraded":"2024-04-25T05:36:40.714651+0000","last_fullsized":"2024-04-25T05:36:40.714651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.515357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725469+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:36:40.725469+0000","last_peered":"2024-04-25T05:36:40.725469+0000","last_clean":"2024-04-25T05:36:40.725469+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:36:40.725469+0000","last_undegraded":"2024-04-25T05:36:40.725469+0000","last_fullsized":"2024-04-25T05:36:40.725469+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":127,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733271+0000","last_change":"2024-04-25T05:35:24.795037+0000","last_active":"2024-04-25T05:36:40.733271+0000","last_peered":"2024-04-25T05:36:40.733271+0000","last_clean":"2024-04-25T05:36:40.733271+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:36:40.733271+0000","last_undegraded":"2024-04-25T05:36:40.733271+0000","last_fullsized":"2024-04-25T05:36:40.733271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:35:24.794880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.794880+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:48.363218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711912+0000","last_change":"2024-04-25T05:36:22.989438+0000","last_active":"2024-04-25T05:36:40.711912+0000","last_peered":"2024-04-25T05:36:40.711912+0000","last_clean":"2024-04-25T05:36:40.711912+0000","last_became_active":"2024-04-25T05:34:02.700339+0000","last_became_peered":"2024-04-25T05:34:02.700339+0000","last_unstale":"2024-04-25T05:36:40.711912+0000","last_undegraded":"2024-04-25T05:36:40.711912+0000","last_fullsized":"2024-04-25T05:36:40.711912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.372787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":139,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733719+0000","last_change":"2024-04-25T05:35:25.303821+0000","last_active":"2024-04-25T05:36:40.733719+0000","last_peered":"2024-04-25T05:36:40.733719+0000","last_clean":"2024-04-25T05:36:40.733719+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:36:40.733719+0000","last_undegraded":"2024-04-25T05:36:40.733719+0000","last_fullsized":"2024-04-25T05:36:40.733719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:25.303682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.303682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717962+0000","last_change":"2024-04-25T05:36:15.129016+0000","last_active":"2024-04-25T05:36:40.717962+0000","last_peered":"2024-04-25T05:36:40.717962+0000","last_clean":"2024-04-25T05:36:40.717962+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:36:40.717962+0000","last_undegraded":"2024-04-25T05:36:40.717962+0000","last_fullsized":"2024-04-25T05:36:40.717962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:36:15.128887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.128887+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:26.080685+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731512+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:36:40.731512+0000","last_peered":"2024-04-25T05:36:40.731512+0000","last_clean":"2024-04-25T05:36:40.731512+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:36:40.731512+0000","last_undegraded":"2024-04-25T05:36:40.731512+0000","last_fullsized":"2024-04-25T05:36:40.731512+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:03.143088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.718322+0000","last_change":"2024-04-25T05:36:14.171466+0000","last_active":"2024-04-25T05:36:40.718322+0000","last_peered":"2024-04-25T05:36:40.718322+0000","last_clean":"2024-04-25T05:36:40.718322+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:36:40.718322+0000","last_undegraded":"2024-04-25T05:36:40.718322+0000","last_fullsized":"2024-04-25T05:36:40.718322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.171349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.171349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:24.814418+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.723823+0000","last_change":"2024-04-25T05:35:36.324777+0000","last_active":"2024-04-25T05:36:40.723823+0000","last_peered":"2024-04-25T05:36:40.723823+0000","last_clean":"2024-04-25T05:36:40.723823+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:36:40.723823+0000","last_undegraded":"2024-04-25T05:36:40.723823+0000","last_fullsized":"2024-04-25T05:36:40.723823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.688505+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":165,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711738+0000","last_change":"2024-04-25T05:35:25.674485+0000","last_active":"2024-04-25T05:36:40.711738+0000","last_peered":"2024-04-25T05:36:40.711738+0000","last_clean":"2024-04-25T05:36:40.711738+0000","last_became_active":"2024-04-25T05:34:00.690855+0000","last_became_peered":"2024-04-25T05:34:00.690855+0000","last_unstale":"2024-04-25T05:36:40.711738+0000","last_undegraded":"2024-04-25T05:36:40.711738+0000","last_fullsized":"2024-04-25T05:36:40.711738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:25.674276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.674276+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.871064+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":127,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731397+0000","last_change":"2024-04-25T05:36:22.428077+0000","last_active":"2024-04-25T05:36:40.731397+0000","last_peered":"2024-04-25T05:36:40.731397+0000","last_clean":"2024-04-25T05:36:40.731397+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:36:40.731397+0000","last_undegraded":"2024-04-25T05:36:40.731397+0000","last_fullsized":"2024-04-25T05:36:40.731397+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:22.427930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.427930+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:25.727609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711162+0000","last_change":"2024-04-25T05:35:28.647997+0000","last_active":"2024-04-25T05:36:40.711162+0000","last_peered":"2024-04-25T05:36:40.711162+0000","last_clean":"2024-04-25T05:36:40.711162+0000","last_became_active":"2024-04-25T05:34:02.700955+0000","last_became_peered":"2024-04-25T05:34:02.700955+0000","last_unstale":"2024-04-25T05:36:40.711162+0000","last_undegraded":"2024-04-25T05:36:40.711162+0000","last_fullsized":"2024-04-25T05:36:40.711162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.718782+0000","last_change":"2024-04-25T05:36:21.217686+0000","last_active":"2024-04-25T05:36:40.718782+0000","last_peered":"2024-04-25T05:36:40.718782+0000","last_clean":"2024-04-25T05:36:40.718782+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:36:40.718782+0000","last_undegraded":"2024-04-25T05:36:40.718782+0000","last_fullsized":"2024-04-25T05:36:40.718782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.483356+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.723926+0000","last_change":"2024-04-25T05:36:19.405542+0000","last_active":"2024-04-25T05:36:40.723926+0000","last_peered":"2024-04-25T05:36:40.723926+0000","last_clean":"2024-04-25T05:36:40.723926+0000","last_became_active":"2024-04-25T05:34:00.692242+0000","last_became_peered":"2024-04-25T05:34:00.692242+0000","last_unstale":"2024-04-25T05:36:40.723926+0000","last_undegraded":"2024-04-25T05:36:40.723926+0000","last_fullsized":"2024-04-25T05:36:40.723926+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:48.487037+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725280+0000","last_change":"2024-04-25T05:34:04.762202+0000","last_active":"2024-04-25T05:36:40.725280+0000","last_peered":"2024-04-25T05:36:40.725280+0000","last_clean":"2024-04-25T05:36:40.725280+0000","last_became_active":"2024-04-25T05:34:04.761141+0000","last_became_peered":"2024-04-25T05:34:04.761141+0000","last_unstale":"2024-04-25T05:36:40.725280+0000","last_undegraded":"2024-04-25T05:36:40.725280+0000","last_fullsized":"2024-04-25T05:36:40.725280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731392+0000","last_change":"2024-04-25T05:35:45.748116+0000","last_active":"2024-04-25T05:36:40.731392+0000","last_peered":"2024-04-25T05:36:40.731392+0000","last_clean":"2024-04-25T05:36:40.731392+0000","last_became_active":"2024-04-25T05:34:02.701458+0000","last_became_peered":"2024-04-25T05:34:02.701458+0000","last_unstale":"2024-04-25T05:36:40.731392+0000","last_undegraded":"2024-04-25T05:36:40.731392+0000","last_fullsized":"2024-04-25T05:36:40.731392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:45.748002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:45.748002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.126478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729750+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:36:40.729750+0000","last_peered":"2024-04-25T05:36:40.729750+0000","last_clean":"2024-04-25T05:36:40.729750+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:36:40.729750+0000","last_undegraded":"2024-04-25T05:36:40.729750+0000","last_fullsized":"2024-04-25T05:36:40.729750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:38.198166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":165,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724271+0000","last_change":"2024-04-25T05:35:38.338139+0000","last_active":"2024-04-25T05:36:40.724271+0000","last_peered":"2024-04-25T05:36:40.724271+0000","last_clean":"2024-04-25T05:36:40.724271+0000","last_became_active":"2024-04-25T05:34:00.999214+0000","last_became_peered":"2024-04-25T05:34:00.999214+0000","last_unstale":"2024-04-25T05:36:40.724271+0000","last_undegraded":"2024-04-25T05:36:40.724271+0000","last_fullsized":"2024-04-25T05:36:40.724271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.229898+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716372+0000","last_change":"2024-04-25T05:35:34.276800+0000","last_active":"2024-04-25T05:36:40.716372+0000","last_peered":"2024-04-25T05:36:40.716372+0000","last_clean":"2024-04-25T05:36:40.716372+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:36:40.716372+0000","last_undegraded":"2024-04-25T05:36:40.716372+0000","last_fullsized":"2024-04-25T05:36:40.716372+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.276673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.276673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":122,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724957+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:36:40.724957+0000","last_peered":"2024-04-25T05:36:40.724957+0000","last_clean":"2024-04-25T05:36:40.724957+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:36:40.724957+0000","last_undegraded":"2024-04-25T05:36:40.724957+0000","last_fullsized":"2024-04-25T05:36:40.724957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733140+0000","last_change":"2024-04-25T05:35:42.725828+0000","last_active":"2024-04-25T05:36:40.733140+0000","last_peered":"2024-04-25T05:36:40.733140+0000","last_clean":"2024-04-25T05:36:40.733140+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:36:40.733140+0000","last_undegraded":"2024-04-25T05:36:40.733140+0000","last_fullsized":"2024-04-25T05:36:40.733140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.725737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.103408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":87,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728536+0000","last_change":"2024-04-25T05:35:33.708929+0000","last_active":"2024-04-25T05:36:40.728536+0000","last_peered":"2024-04-25T05:36:40.728536+0000","last_clean":"2024-04-25T05:36:40.728536+0000","last_became_active":"2024-04-25T05:35:32.892093+0000","last_became_peered":"2024-04-25T05:35:32.892093+0000","last_unstale":"2024-04-25T05:36:40.728536+0000","last_undegraded":"2024-04-25T05:36:40.728536+0000","last_fullsized":"2024-04-25T05:36:40.728536+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:35:33.708629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:33.708629+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.641362+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.723941+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:36:40.723941+0000","last_peered":"2024-04-25T05:36:40.723941+0000","last_clean":"2024-04-25T05:36:40.723941+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:36:40.723941+0000","last_undegraded":"2024-04-25T05:36:40.723941+0000","last_fullsized":"2024-04-25T05:36:40.723941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732538+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:36:40.732538+0000","last_peered":"2024-04-25T05:36:40.732538+0000","last_clean":"2024-04-25T05:36:40.732538+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:36:40.732538+0000","last_undegraded":"2024-04-25T05:36:40.732538+0000","last_fullsized":"2024-04-25T05:36:40.732538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.932366+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":143,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715715+0000","last_change":"2024-04-25T05:35:02.707425+0000","last_active":"2024-04-25T05:36:40.715715+0000","last_peered":"2024-04-25T05:36:40.715715+0000","last_clean":"2024-04-25T05:36:40.715715+0000","last_became_active":"2024-04-25T05:33:58.679825+0000","last_became_peered":"2024-04-25T05:33:58.679825+0000","last_unstale":"2024-04-25T05:36:40.715715+0000","last_undegraded":"2024-04-25T05:36:40.715715+0000","last_fullsized":"2024-04-25T05:36:40.715715+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":152,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715365+0000","last_change":"2024-04-25T05:35:07.728633+0000","last_active":"2024-04-25T05:36:40.715365+0000","last_peered":"2024-04-25T05:36:40.715365+0000","last_clean":"2024-04-25T05:36:40.715365+0000","last_became_active":"2024-04-25T05:34:00.692881+0000","last_became_peered":"2024-04-25T05:34:00.692881+0000","last_unstale":"2024-04-25T05:36:40.715365+0000","last_undegraded":"2024-04-25T05:36:40.715365+0000","last_fullsized":"2024-04-25T05:36:40.715365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717278+0000","last_change":"2024-04-25T05:36:30.109005+0000","last_active":"2024-04-25T05:36:40.717278+0000","last_peered":"2024-04-25T05:36:40.717278+0000","last_clean":"2024-04-25T05:36:40.717278+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:36:40.717278+0000","last_undegraded":"2024-04-25T05:36:40.717278+0000","last_fullsized":"2024-04-25T05:36:40.717278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.212177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":171,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729032+0000","last_change":"2024-04-25T05:35:39.823582+0000","last_active":"2024-04-25T05:36:40.729032+0000","last_peered":"2024-04-25T05:36:40.729032+0000","last_clean":"2024-04-25T05:36:40.729032+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:36:40.729032+0000","last_undegraded":"2024-04-25T05:36:40.729032+0000","last_fullsized":"2024-04-25T05:36:40.729032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:39.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.823449+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.097815+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731693+0000","last_change":"2024-04-25T05:36:15.504640+0000","last_active":"2024-04-25T05:36:40.731693+0000","last_peered":"2024-04-25T05:36:40.731693+0000","last_clean":"2024-04-25T05:36:40.731693+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:36:40.731693+0000","last_undegraded":"2024-04-25T05:36:40.731693+0000","last_fullsized":"2024-04-25T05:36:40.731693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:15.504395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.504395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:20.668339+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":173,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726064+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:36:40.726064+0000","last_peered":"2024-04-25T05:36:40.726064+0000","last_clean":"2024-04-25T05:36:40.726064+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:36:40.726064+0000","last_undegraded":"2024-04-25T05:36:40.726064+0000","last_fullsized":"2024-04-25T05:36:40.726064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.719798+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715840+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:36:40.715840+0000","last_peered":"2024-04-25T05:36:40.715840+0000","last_clean":"2024-04-25T05:36:40.715840+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:36:40.715840+0000","last_undegraded":"2024-04-25T05:36:40.715840+0000","last_fullsized":"2024-04-25T05:36:40.715840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.179078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":163,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726401+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:36:40.726401+0000","last_peered":"2024-04-25T05:36:40.726401+0000","last_clean":"2024-04-25T05:36:40.726401+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:36:40.726401+0000","last_undegraded":"2024-04-25T05:36:40.726401+0000","last_fullsized":"2024-04-25T05:36:40.726401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715766+0000","last_change":"2024-04-25T05:36:31.118006+0000","last_active":"2024-04-25T05:36:40.715766+0000","last_peered":"2024-04-25T05:36:40.715766+0000","last_clean":"2024-04-25T05:36:40.715766+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:36:40.715766+0000","last_undegraded":"2024-04-25T05:36:40.715766+0000","last_fullsized":"2024-04-25T05:36:40.715766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.339192+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":167,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729176+0000","last_change":"2024-04-25T05:35:40.845196+0000","last_active":"2024-04-25T05:36:40.729176+0000","last_peered":"2024-04-25T05:36:40.729176+0000","last_clean":"2024-04-25T05:36:40.729176+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:36:40.729176+0000","last_undegraded":"2024-04-25T05:36:40.729176+0000","last_fullsized":"2024-04-25T05:36:40.729176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.845036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:08.541350+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.722916+0000","last_change":"2024-04-25T05:35:30.673162+0000","last_active":"2024-04-25T05:36:40.722916+0000","last_peered":"2024-04-25T05:36:40.722916+0000","last_clean":"2024-04-25T05:36:40.722916+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:36:40.722916+0000","last_undegraded":"2024-04-25T05:36:40.722916+0000","last_fullsized":"2024-04-25T05:36:40.722916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:30.673041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.673041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.187518+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":122,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726732+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:36:40.726732+0000","last_peered":"2024-04-25T05:36:40.726732+0000","last_clean":"2024-04-25T05:36:40.726732+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:36:40.726732+0000","last_undegraded":"2024-04-25T05:36:40.726732+0000","last_fullsized":"2024-04-25T05:36:40.726732+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732230+0000","last_change":"2024-04-25T05:35:28.761829+0000","last_active":"2024-04-25T05:36:40.732230+0000","last_peered":"2024-04-25T05:36:40.732230+0000","last_clean":"2024-04-25T05:36:40.732230+0000","last_became_active":"2024-04-25T05:33:58.685441+0000","last_became_peered":"2024-04-25T05:33:58.685441+0000","last_unstale":"2024-04-25T05:36:40.732230+0000","last_undegraded":"2024-04-25T05:36:40.732230+0000","last_fullsized":"2024-04-25T05:36:40.732230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.761706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.761706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.404788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.718161+0000","last_change":"2024-04-25T05:36:20.155217+0000","last_active":"2024-04-25T05:36:40.718161+0000","last_peered":"2024-04-25T05:36:40.718161+0000","last_clean":"2024-04-25T05:36:40.718161+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:36:40.718161+0000","last_undegraded":"2024-04-25T05:36:40.718161+0000","last_fullsized":"2024-04-25T05:36:40.718161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:40.449029+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732903+0000","last_change":"2024-04-25T05:36:18.481176+0000","last_active":"2024-04-25T05:36:40.732903+0000","last_peered":"2024-04-25T05:36:40.732903+0000","last_clean":"2024-04-25T05:36:40.732903+0000","last_became_active":"2024-04-25T05:34:04.760512+0000","last_became_peered":"2024-04-25T05:34:04.760512+0000","last_unstale":"2024-04-25T05:36:40.732903+0000","last_undegraded":"2024-04-25T05:36:40.732903+0000","last_fullsized":"2024-04-25T05:36:40.732903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.481007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.481007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.599925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729836+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:36:40.729836+0000","last_peered":"2024-04-25T05:36:40.729836+0000","last_clean":"2024-04-25T05:36:40.729836+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:36:40.729836+0000","last_undegraded":"2024-04-25T05:36:40.729836+0000","last_fullsized":"2024-04-25T05:36:40.729836+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.303715+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.710721+0000","last_change":"2024-04-25T05:35:44.715464+0000","last_active":"2024-04-25T05:36:40.710721+0000","last_peered":"2024-04-25T05:36:40.710721+0000","last_clean":"2024-04-25T05:36:40.710721+0000","last_became_active":"2024-04-25T05:33:58.682505+0000","last_became_peered":"2024-04-25T05:33:58.682505+0000","last_unstale":"2024-04-25T05:36:40.710721+0000","last_undegraded":"2024-04-25T05:36:40.710721+0000","last_fullsized":"2024-04-25T05:36:40.710721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.715360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.715360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:49.881138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.727010+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:36:40.727010+0000","last_peered":"2024-04-25T05:36:40.727010+0000","last_clean":"2024-04-25T05:36:40.727010+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:36:40.727010+0000","last_undegraded":"2024-04-25T05:36:40.727010+0000","last_fullsized":"2024-04-25T05:36:40.727010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"126'6983","reported_seq":7133,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.734973+0000","last_change":"2024-04-25T05:36:18.461712+0000","last_active":"2024-04-25T05:36:42.734973+0000","last_peered":"2024-04-25T05:36:42.734973+0000","last_clean":"2024-04-25T05:36:42.734973+0000","last_became_active":"2024-04-25T05:35:09.679411+0000","last_became_peered":"2024-04-25T05:35:09.679411+0000","last_unstale":"2024-04-25T05:36:42.734973+0000","last_undegraded":"2024-04-25T05:36:42.734973+0000","last_fullsized":"2024-04-25T05:36:42.734973+0000","mapping_epoch":88,"log_start":"95'1900","ondisk_log_start":"95'1900","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'5609","last_scrub_stamp":"2024-04-25T05:36:18.461345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.461345+0000","objects_scrubbed":5551,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:42.618808+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457637888,"num_objects":6983,"num_object_clones":0,"num_object_copies":20949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6983,"num_write_kb":446912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.730005+0000","last_change":"2024-04-25T05:35:24.680523+0000","last_active":"2024-04-25T05:36:40.730005+0000","last_peered":"2024-04-25T05:36:40.730005+0000","last_clean":"2024-04-25T05:36:40.730005+0000","last_became_active":"2024-04-25T05:34:04.759074+0000","last_became_peered":"2024-04-25T05:34:04.759074+0000","last_unstale":"2024-04-25T05:36:40.730005+0000","last_undegraded":"2024-04-25T05:36:40.730005+0000","last_fullsized":"2024-04-25T05:36:40.730005+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.680437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.414013+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":150,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.735067+0000","last_change":"2024-04-25T05:36:14.472050+0000","last_active":"2024-04-25T05:36:40.735067+0000","last_peered":"2024-04-25T05:36:40.735067+0000","last_clean":"2024-04-25T05:36:40.735067+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:36:40.735067+0000","last_undegraded":"2024-04-25T05:36:40.735067+0000","last_fullsized":"2024-04-25T05:36:40.735067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:36:14.471895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.471895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:17.098974+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":183,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728361+0000","last_change":"2024-04-25T05:36:17.062531+0000","last_active":"2024-04-25T05:36:40.728361+0000","last_peered":"2024-04-25T05:36:40.728361+0000","last_clean":"2024-04-25T05:36:40.728361+0000","last_became_active":"2024-04-25T05:34:01.000481+0000","last_became_peered":"2024-04-25T05:34:01.000481+0000","last_unstale":"2024-04-25T05:36:40.728361+0000","last_undegraded":"2024-04-25T05:36:40.728361+0000","last_fullsized":"2024-04-25T05:36:40.728361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:33.927079+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"118'9","reported_seq":180,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732359+0000","last_change":"2024-04-25T05:36:29.349021+0000","last_active":"2024-04-25T05:36:40.732359+0000","last_peered":"2024-04-25T05:36:40.732359+0000","last_clean":"2024-04-25T05:36:40.732359+0000","last_became_active":"2024-04-25T05:34:02.701105+0000","last_became_peered":"2024-04-25T05:34:02.701105+0000","last_unstale":"2024-04-25T05:36:40.732359+0000","last_undegraded":"2024-04-25T05:36:40.732359+0000","last_fullsized":"2024-04-25T05:36:40.732359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:41.385356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"126'7118","reported_seq":7204,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.692212+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:36:42.692212+0000","last_peered":"2024-04-25T05:36:42.692212+0000","last_clean":"2024-04-25T05:36:42.692212+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:36:42.692212+0000","last_undegraded":"2024-04-25T05:36:42.692212+0000","last_fullsized":"2024-04-25T05:36:42.692212+0000","mapping_epoch":88,"log_start":"98'2500","ondisk_log_start":"98'2500","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.541369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466485248,"num_objects":7118,"num_object_clones":0,"num_object_copies":21354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":455552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725115+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:36:40.725115+0000","last_peered":"2024-04-25T05:36:40.725115+0000","last_clean":"2024-04-25T05:36:40.725115+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:36:40.725115+0000","last_undegraded":"2024-04-25T05:36:40.725115+0000","last_fullsized":"2024-04-25T05:36:40.725115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732855+0000","last_change":"2024-04-25T05:35:02.907524+0000","last_active":"2024-04-25T05:36:40.732855+0000","last_peered":"2024-04-25T05:36:40.732855+0000","last_clean":"2024-04-25T05:36:40.732855+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:36:40.732855+0000","last_undegraded":"2024-04-25T05:36:40.732855+0000","last_fullsized":"2024-04-25T05:36:40.732855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:02.907405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.907405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":180,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716704+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:36:40.716704+0000","last_peered":"2024-04-25T05:36:40.716704+0000","last_clean":"2024-04-25T05:36:40.716704+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:36:40.716704+0000","last_undegraded":"2024-04-25T05:36:40.716704+0000","last_fullsized":"2024-04-25T05:36:40.716704+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.139950+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.721883+0000","last_change":"2024-04-25T05:35:31.652458+0000","last_active":"2024-04-25T05:36:40.721883+0000","last_peered":"2024-04-25T05:36:40.721883+0000","last_clean":"2024-04-25T05:36:40.721883+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:36:40.721883+0000","last_undegraded":"2024-04-25T05:36:40.721883+0000","last_fullsized":"2024-04-25T05:36:40.721883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:31.652381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:31.652381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"126'7071","reported_seq":7157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.678125+0000","last_change":"2024-04-25T05:35:09.681598+0000","last_active":"2024-04-25T05:36:42.678125+0000","last_peered":"2024-04-25T05:36:42.678125+0000","last_clean":"2024-04-25T05:36:42.678125+0000","last_became_active":"2024-04-25T05:35:09.681316+0000","last_became_peered":"2024-04-25T05:35:09.681316+0000","last_unstale":"2024-04-25T05:36:42.678125+0000","last_undegraded":"2024-04-25T05:36:42.678125+0000","last_fullsized":"2024-04-25T05:36:42.678125+0000","mapping_epoch":88,"log_start":"95'1500","ondisk_log_start":"95'1500","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.236230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463405056,"num_objects":7071,"num_object_clones":0,"num_object_copies":21213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7071,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731106+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:36:40.731106+0000","last_peered":"2024-04-25T05:36:40.731106+0000","last_clean":"2024-04-25T05:36:40.731106+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:36:40.731106+0000","last_undegraded":"2024-04-25T05:36:40.731106+0000","last_fullsized":"2024-04-25T05:36:40.731106+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.897023+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.722196+0000","last_change":"2024-04-25T05:35:27.565332+0000","last_active":"2024-04-25T05:36:40.722196+0000","last_peered":"2024-04-25T05:36:40.722196+0000","last_clean":"2024-04-25T05:36:40.722196+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:36:40.722196+0000","last_undegraded":"2024-04-25T05:36:40.722196+0000","last_fullsized":"2024-04-25T05:36:40.722196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.565209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.565209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.931166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":152,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733387+0000","last_change":"2024-04-25T05:36:30.395761+0000","last_active":"2024-04-25T05:36:40.733387+0000","last_peered":"2024-04-25T05:36:40.733387+0000","last_clean":"2024-04-25T05:36:40.733387+0000","last_became_active":"2024-04-25T05:33:58.685793+0000","last_became_peered":"2024-04-25T05:33:58.685793+0000","last_unstale":"2024-04-25T05:36:40.733387+0000","last_undegraded":"2024-04-25T05:36:40.733387+0000","last_fullsized":"2024-04-25T05:36:40.733387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:38.353889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":177,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711341+0000","last_change":"2024-04-25T05:35:32.693459+0000","last_active":"2024-04-25T05:36:40.711341+0000","last_peered":"2024-04-25T05:36:40.711341+0000","last_clean":"2024-04-25T05:36:40.711341+0000","last_became_active":"2024-04-25T05:34:00.999498+0000","last_became_peered":"2024-04-25T05:34:00.999498+0000","last_unstale":"2024-04-25T05:36:40.711341+0000","last_undegraded":"2024-04-25T05:36:40.711341+0000","last_fullsized":"2024-04-25T05:36:40.711341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.919493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"126'7007","reported_seq":7163,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.713451+0000","last_change":"2024-04-25T05:36:31.742804+0000","last_active":"2024-04-25T05:36:42.713451+0000","last_peered":"2024-04-25T05:36:42.713451+0000","last_clean":"2024-04-25T05:36:42.713451+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:36:42.713451+0000","last_undegraded":"2024-04-25T05:36:42.713451+0000","last_fullsized":"2024-04-25T05:36:42.713451+0000","mapping_epoch":88,"log_start":"93'100","ondisk_log_start":"93'100","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6352","last_scrub_stamp":"2024-04-25T05:36:31.742451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.742451+0000","objects_scrubbed":6298,"log_size":6907,"log_dups_size":0,"ondisk_log_size":6907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:58.047202+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459210752,"num_objects":7007,"num_object_clones":0,"num_object_copies":21021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7007,"num_write_kb":448448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717740+0000","last_change":"2024-04-25T05:36:27.172930+0000","last_active":"2024-04-25T05:36:40.717740+0000","last_peered":"2024-04-25T05:36:40.717740+0000","last_clean":"2024-04-25T05:36:40.717740+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:36:40.717740+0000","last_undegraded":"2024-04-25T05:36:40.717740+0000","last_fullsized":"2024-04-25T05:36:40.717740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:27.172809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.172809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.542287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.722188+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:36:40.722188+0000","last_peered":"2024-04-25T05:36:40.722188+0000","last_clean":"2024-04-25T05:36:40.722188+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:36:40.722188+0000","last_undegraded":"2024-04-25T05:36:40.722188+0000","last_fullsized":"2024-04-25T05:36:40.722188+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.188154+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728971+0000","last_change":"2024-04-25T05:35:34.751017+0000","last_active":"2024-04-25T05:36:40.728971+0000","last_peered":"2024-04-25T05:36:40.728971+0000","last_clean":"2024-04-25T05:36:40.728971+0000","last_became_active":"2024-04-25T05:33:58.685596+0000","last_became_peered":"2024-04-25T05:33:58.685596+0000","last_unstale":"2024-04-25T05:36:40.728971+0000","last_undegraded":"2024-04-25T05:36:40.728971+0000","last_fullsized":"2024-04-25T05:36:40.728971+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.750887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.750887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.517305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716840+0000","last_change":"2024-04-25T05:35:40.352567+0000","last_active":"2024-04-25T05:36:40.716840+0000","last_peered":"2024-04-25T05:36:40.716840+0000","last_clean":"2024-04-25T05:36:40.716840+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:36:40.716840+0000","last_undegraded":"2024-04-25T05:36:40.716840+0000","last_fullsized":"2024-04-25T05:36:40.716840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:01.266360+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"126'7181","reported_seq":7267,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.701100+0000","last_change":"2024-04-25T05:35:09.684480+0000","last_active":"2024-04-25T05:36:42.701100+0000","last_peered":"2024-04-25T05:36:42.701100+0000","last_clean":"2024-04-25T05:36:42.701100+0000","last_became_active":"2024-04-25T05:35:09.683860+0000","last_became_peered":"2024-04-25T05:35:09.683860+0000","last_unstale":"2024-04-25T05:36:42.701100+0000","last_undegraded":"2024-04-25T05:36:42.701100+0000","last_fullsized":"2024-04-25T05:36:42.701100+0000","mapping_epoch":88,"log_start":"93'900","ondisk_log_start":"93'900","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470614016,"num_objects":7181,"num_object_clones":0,"num_object_copies":21543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":459584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726193+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:36:40.726193+0000","last_peered":"2024-04-25T05:36:40.726193+0000","last_clean":"2024-04-25T05:36:40.726193+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:36:40.726193+0000","last_undegraded":"2024-04-25T05:36:40.726193+0000","last_fullsized":"2024-04-25T05:36:40.726193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.505234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729359+0000","last_change":"2024-04-25T05:36:21.033702+0000","last_active":"2024-04-25T05:36:40.729359+0000","last_peered":"2024-04-25T05:36:40.729359+0000","last_clean":"2024-04-25T05:36:40.729359+0000","last_became_active":"2024-04-25T05:33:57.668894+0000","last_became_peered":"2024-04-25T05:33:57.668894+0000","last_unstale":"2024-04-25T05:36:40.729359+0000","last_undegraded":"2024-04-25T05:36:40.729359+0000","last_fullsized":"2024-04-25T05:36:40.729359+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:45.885604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714917+0000","last_change":"2024-04-25T05:33:58.683662+0000","last_active":"2024-04-25T05:36:40.714917+0000","last_peered":"2024-04-25T05:36:40.714917+0000","last_clean":"2024-04-25T05:36:40.714917+0000","last_became_active":"2024-04-25T05:33:58.682153+0000","last_became_peered":"2024-04-25T05:33:58.682153+0000","last_unstale":"2024-04-25T05:36:40.714917+0000","last_undegraded":"2024-04-25T05:36:40.714917+0000","last_fullsized":"2024-04-25T05:36:40.714917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":238,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.736735+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:36:40.736735+0000","last_peered":"2024-04-25T05:36:40.736735+0000","last_clean":"2024-04-25T05:36:40.736735+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:36:40.736735+0000","last_undegraded":"2024-04-25T05:36:40.736735+0000","last_fullsized":"2024-04-25T05:36:40.736735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.463394+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716213+0000","last_change":"2024-04-25T05:36:24.214089+0000","last_active":"2024-04-25T05:36:40.716213+0000","last_peered":"2024-04-25T05:36:40.716213+0000","last_clean":"2024-04-25T05:36:40.716213+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:36:40.716213+0000","last_undegraded":"2024-04-25T05:36:40.716213+0000","last_fullsized":"2024-04-25T05:36:40.716213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.446616+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"126'7049","reported_seq":7135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.769119+0000","last_change":"2024-04-25T05:35:09.679881+0000","last_active":"2024-04-25T05:36:42.769119+0000","last_peered":"2024-04-25T05:36:42.769119+0000","last_clean":"2024-04-25T05:36:42.769119+0000","last_became_active":"2024-04-25T05:35:09.679580+0000","last_became_peered":"2024-04-25T05:35:09.679580+0000","last_unstale":"2024-04-25T05:36:42.769119+0000","last_undegraded":"2024-04-25T05:36:42.769119+0000","last_fullsized":"2024-04-25T05:36:42.769119+0000","mapping_epoch":88,"log_start":"95'1400","ondisk_log_start":"95'1400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5649,"log_dups_size":0,"ondisk_log_size":5649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.239431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461963264,"num_objects":7049,"num_object_clones":0,"num_object_copies":21147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":451136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733552+0000","last_change":"2024-04-25T05:35:34.717443+0000","last_active":"2024-04-25T05:36:40.733552+0000","last_peered":"2024-04-25T05:36:40.733552+0000","last_clean":"2024-04-25T05:36:40.733552+0000","last_became_active":"2024-04-25T05:34:04.760422+0000","last_became_peered":"2024-04-25T05:34:04.760422+0000","last_unstale":"2024-04-25T05:36:40.733552+0000","last_undegraded":"2024-04-25T05:36:40.733552+0000","last_fullsized":"2024-04-25T05:36:40.733552+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.717213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.717213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.918493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.730579+0000","last_change":"2024-04-25T05:35:40.340205+0000","last_active":"2024-04-25T05:36:40.730579+0000","last_peered":"2024-04-25T05:36:40.730579+0000","last_clean":"2024-04-25T05:36:40.730579+0000","last_became_active":"2024-04-25T05:33:58.679610+0000","last_became_peered":"2024-04-25T05:33:58.679610+0000","last_unstale":"2024-04-25T05:36:40.730579+0000","last_undegraded":"2024-04-25T05:36:40.730579+0000","last_fullsized":"2024-04-25T05:36:40.730579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:40.339655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.339655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:01.210259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.721882+0000","last_change":"2024-04-25T05:36:16.674171+0000","last_active":"2024-04-25T05:36:40.721882+0000","last_peered":"2024-04-25T05:36:40.721882+0000","last_clean":"2024-04-25T05:36:40.721882+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:36:40.721882+0000","last_undegraded":"2024-04-25T05:36:40.721882+0000","last_fullsized":"2024-04-25T05:36:40.721882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:27.824707+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":179,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.730745+0000","last_change":"2024-04-25T05:35:39.312001+0000","last_active":"2024-04-25T05:36:40.730745+0000","last_peered":"2024-04-25T05:36:40.730745+0000","last_clean":"2024-04-25T05:36:40.730745+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:36:40.730745+0000","last_undegraded":"2024-04-25T05:36:40.730745+0000","last_fullsized":"2024-04-25T05:36:40.730745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:06.200939+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.723233+0000","last_change":"2024-04-25T05:35:42.672247+0000","last_active":"2024-04-25T05:36:40.723233+0000","last_peered":"2024-04-25T05:36:40.723233+0000","last_clean":"2024-04-25T05:36:40.723233+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:36:40.723233+0000","last_undegraded":"2024-04-25T05:36:40.723233+0000","last_fullsized":"2024-04-25T05:36:40.723233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.672114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.672114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:51.402270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"126'7026","reported_seq":7112,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.782765+0000","last_change":"2024-04-25T05:35:09.683951+0000","last_active":"2024-04-25T05:36:42.782765+0000","last_peered":"2024-04-25T05:36:42.782765+0000","last_clean":"2024-04-25T05:36:42.782765+0000","last_became_active":"2024-04-25T05:35:09.682815+0000","last_became_peered":"2024-04-25T05:35:09.682815+0000","last_unstale":"2024-04-25T05:36:42.782765+0000","last_undegraded":"2024-04-25T05:36:42.782765+0000","last_fullsized":"2024-04-25T05:36:42.782765+0000","mapping_epoch":88,"log_start":"95'1400","ondisk_log_start":"95'1400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.239499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460455936,"num_objects":7026,"num_object_clones":0,"num_object_copies":21078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7026,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729409+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:36:40.729409+0000","last_peered":"2024-04-25T05:36:40.729409+0000","last_clean":"2024-04-25T05:36:40.729409+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:36:40.729409+0000","last_undegraded":"2024-04-25T05:36:40.729409+0000","last_fullsized":"2024-04-25T05:36:40.729409+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.990922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":389,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731173+0000","last_change":"2024-04-25T05:36:28.364094+0000","last_active":"2024-04-25T05:36:40.731173+0000","last_peered":"2024-04-25T05:36:40.731173+0000","last_clean":"2024-04-25T05:36:40.731173+0000","last_became_active":"2024-04-25T05:33:50.653235+0000","last_became_peered":"2024-04-25T05:33:50.653235+0000","last_unstale":"2024-04-25T05:36:40.731173+0000","last_undegraded":"2024-04-25T05:36:40.731173+0000","last_fullsized":"2024-04-25T05:36:40.731173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:36:28.364006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.364006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:47.701186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724357+0000","last_change":"2024-04-25T05:36:26.336148+0000","last_active":"2024-04-25T05:36:40.724357+0000","last_peered":"2024-04-25T05:36:40.724357+0000","last_clean":"2024-04-25T05:36:40.724357+0000","last_became_active":"2024-04-25T05:33:58.678931+0000","last_became_peered":"2024-04-25T05:33:58.678931+0000","last_unstale":"2024-04-25T05:36:40.724357+0000","last_undegraded":"2024-04-25T05:36:40.724357+0000","last_fullsized":"2024-04-25T05:36:40.724357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:40.326541+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":189,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711605+0000","last_change":"2024-04-25T05:35:27.678182+0000","last_active":"2024-04-25T05:36:40.711605+0000","last_peered":"2024-04-25T05:36:40.711605+0000","last_clean":"2024-04-25T05:36:40.711605+0000","last_became_active":"2024-04-25T05:34:01.001778+0000","last_became_peered":"2024-04-25T05:34:01.001778+0000","last_unstale":"2024-04-25T05:36:40.711605+0000","last_undegraded":"2024-04-25T05:36:40.711605+0000","last_fullsized":"2024-04-25T05:36:40.711605+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:27.678020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.678020+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.087083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.732618+0000","last_change":"2024-04-25T05:35:29.726834+0000","last_active":"2024-04-25T05:36:40.732618+0000","last_peered":"2024-04-25T05:36:40.732618+0000","last_clean":"2024-04-25T05:36:40.732618+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:36:40.732618+0000","last_undegraded":"2024-04-25T05:36:40.732618+0000","last_fullsized":"2024-04-25T05:36:40.732618+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:29.726669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.726669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:52.548355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"126'7056","reported_seq":7142,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.593848+0000","last_change":"2024-04-25T05:35:09.683621+0000","last_active":"2024-04-25T05:36:42.593848+0000","last_peered":"2024-04-25T05:36:42.593848+0000","last_clean":"2024-04-25T05:36:42.593848+0000","last_became_active":"2024-04-25T05:35:09.683250+0000","last_became_peered":"2024-04-25T05:35:09.683250+0000","last_unstale":"2024-04-25T05:36:42.593848+0000","last_undegraded":"2024-04-25T05:36:42.593848+0000","last_fullsized":"2024-04-25T05:36:42.593848+0000","mapping_epoch":88,"log_start":"95'1800","ondisk_log_start":"95'1800","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462422016,"num_objects":7056,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.736332+0000","last_change":"2024-04-25T05:35:38.262254+0000","last_active":"2024-04-25T05:36:40.736332+0000","last_peered":"2024-04-25T05:36:40.736332+0000","last_clean":"2024-04-25T05:36:40.736332+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:36:40.736332+0000","last_undegraded":"2024-04-25T05:36:40.736332+0000","last_fullsized":"2024-04-25T05:36:40.736332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.261996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.261996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.492794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717683+0000","last_change":"2024-04-25T05:36:26.153204+0000","last_active":"2024-04-25T05:36:40.717683+0000","last_peered":"2024-04-25T05:36:40.717683+0000","last_clean":"2024-04-25T05:36:40.717683+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:36:40.717683+0000","last_undegraded":"2024-04-25T05:36:40.717683+0000","last_fullsized":"2024-04-25T05:36:40.717683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.991917+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":307,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.943362+0000","last_change":"2024-04-25T05:36:20.496553+0000","last_active":"2024-04-25T05:36:40.943362+0000","last_peered":"2024-04-25T05:36:40.943362+0000","last_clean":"2024-04-25T05:36:40.943362+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:36:40.943362+0000","last_undegraded":"2024-04-25T05:36:40.943362+0000","last_fullsized":"2024-04-25T05:36:40.943362+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:36:20.496280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.496280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:30.974274+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":160,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717583+0000","last_change":"2024-04-25T05:36:32.152204+0000","last_active":"2024-04-25T05:36:40.717583+0000","last_peered":"2024-04-25T05:36:40.717583+0000","last_clean":"2024-04-25T05:36:40.717583+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:36:40.717583+0000","last_undegraded":"2024-04-25T05:36:40.717583+0000","last_fullsized":"2024-04-25T05:36:40.717583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.408462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729649+0000","last_change":"2024-04-25T05:35:35.758618+0000","last_active":"2024-04-25T05:36:40.729649+0000","last_peered":"2024-04-25T05:36:40.729649+0000","last_clean":"2024-04-25T05:36:40.729649+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:36:40.729649+0000","last_undegraded":"2024-04-25T05:36:40.729649+0000","last_fullsized":"2024-04-25T05:36:40.729649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.758501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.758501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:42.994906+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"126'7062","reported_seq":7148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.613908+0000","last_change":"2024-04-25T05:35:09.676826+0000","last_active":"2024-04-25T05:36:42.613908+0000","last_peered":"2024-04-25T05:36:42.613908+0000","last_clean":"2024-04-25T05:36:42.613908+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:36:42.613908+0000","last_undegraded":"2024-04-25T05:36:42.613908+0000","last_fullsized":"2024-04-25T05:36:42.613908+0000","mapping_epoch":88,"log_start":"95'1200","ondisk_log_start":"95'1200","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5862,"log_dups_size":0,"ondisk_log_size":5862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462815232,"num_objects":7062,"num_object_clones":0,"num_object_copies":21186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":451968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.733535+0000","last_change":"2024-04-25T05:35:35.237306+0000","last_active":"2024-04-25T05:36:40.733535+0000","last_peered":"2024-04-25T05:36:40.733535+0000","last_clean":"2024-04-25T05:36:40.733535+0000","last_became_active":"2024-04-25T05:34:04.758741+0000","last_became_peered":"2024-04-25T05:34:04.758741+0000","last_unstale":"2024-04-25T05:36:40.733535+0000","last_undegraded":"2024-04-25T05:36:40.733535+0000","last_fullsized":"2024-04-25T05:36:40.733535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.341015+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.736600+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:36:40.736600+0000","last_peered":"2024-04-25T05:36:40.736600+0000","last_clean":"2024-04-25T05:36:40.736600+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:36:40.736600+0000","last_undegraded":"2024-04-25T05:36:40.736600+0000","last_fullsized":"2024-04-25T05:36:40.736600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.664326+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":182,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716598+0000","last_change":"2024-04-25T05:35:35.249157+0000","last_active":"2024-04-25T05:36:40.716598+0000","last_peered":"2024-04-25T05:36:40.716598+0000","last_clean":"2024-04-25T05:36:40.716598+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:36:40.716598+0000","last_undegraded":"2024-04-25T05:36:40.716598+0000","last_fullsized":"2024-04-25T05:36:40.716598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:35.248443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.248443+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:56.985228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711405+0000","last_change":"2024-04-25T05:36:21.964578+0000","last_active":"2024-04-25T05:36:40.711405+0000","last_peered":"2024-04-25T05:36:40.711405+0000","last_clean":"2024-04-25T05:36:40.711405+0000","last_became_active":"2024-04-25T05:34:02.708133+0000","last_became_peered":"2024-04-25T05:34:02.708133+0000","last_unstale":"2024-04-25T05:36:40.711405+0000","last_undegraded":"2024-04-25T05:36:40.711405+0000","last_fullsized":"2024-04-25T05:36:40.711405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:48.818196+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"126'7173","reported_seq":7258,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.671956+0000","last_change":"2024-04-25T05:35:09.680036+0000","last_active":"2024-04-25T05:36:42.671956+0000","last_peered":"2024-04-25T05:36:42.671956+0000","last_clean":"2024-04-25T05:36:42.671956+0000","last_became_active":"2024-04-25T05:35:09.679390+0000","last_became_peered":"2024-04-25T05:35:09.679390+0000","last_unstale":"2024-04-25T05:36:42.671956+0000","last_undegraded":"2024-04-25T05:36:42.671956+0000","last_fullsized":"2024-04-25T05:36:42.671956+0000","mapping_epoch":88,"log_start":"97'2400","ondisk_log_start":"97'2400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470089728,"num_objects":7173,"num_object_clones":0,"num_object_copies":21519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":459072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716479+0000","last_change":"2024-04-25T05:35:44.317463+0000","last_active":"2024-04-25T05:36:40.716479+0000","last_peered":"2024-04-25T05:36:40.716479+0000","last_clean":"2024-04-25T05:36:40.716479+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:36:40.716479+0000","last_undegraded":"2024-04-25T05:36:40.716479+0000","last_fullsized":"2024-04-25T05:36:40.716479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:09.893502+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729089+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:36:40.729089+0000","last_peered":"2024-04-25T05:36:40.729089+0000","last_clean":"2024-04-25T05:36:40.729089+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:36:40.729089+0000","last_undegraded":"2024-04-25T05:36:40.729089+0000","last_fullsized":"2024-04-25T05:36:40.729089+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:35.016206+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.721931+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:36:40.721931+0000","last_peered":"2024-04-25T05:36:40.721931+0000","last_clean":"2024-04-25T05:36:40.721931+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:36:40.721931+0000","last_undegraded":"2024-04-25T05:36:40.721931+0000","last_fullsized":"2024-04-25T05:36:40.721931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:55.713346+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":171,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716940+0000","last_change":"2024-04-25T05:35:42.311228+0000","last_active":"2024-04-25T05:36:40.716940+0000","last_peered":"2024-04-25T05:36:40.716940+0000","last_clean":"2024-04-25T05:36:40.716940+0000","last_became_active":"2024-04-25T05:34:02.707840+0000","last_became_peered":"2024-04-25T05:34:02.707840+0000","last_unstale":"2024-04-25T05:36:40.716940+0000","last_undegraded":"2024-04-25T05:36:40.716940+0000","last_fullsized":"2024-04-25T05:36:40.716940+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:42.311077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.311077+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.843077+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"126'7123","reported_seq":7209,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.787582+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:36:42.787582+0000","last_peered":"2024-04-25T05:36:42.787582+0000","last_clean":"2024-04-25T05:36:42.787582+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:36:42.787582+0000","last_undegraded":"2024-04-25T05:36:42.787582+0000","last_fullsized":"2024-04-25T05:36:42.787582+0000","mapping_epoch":88,"log_start":"98'2500","ondisk_log_start":"98'2500","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.541300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466812928,"num_objects":7123,"num_object_clones":0,"num_object_copies":21369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7123,"num_write_kb":455872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.731954+0000","last_change":"2024-04-25T05:35:35.701192+0000","last_active":"2024-04-25T05:36:40.731954+0000","last_peered":"2024-04-25T05:36:40.731954+0000","last_clean":"2024-04-25T05:36:40.731954+0000","last_became_active":"2024-04-25T05:34:04.760855+0000","last_became_peered":"2024-04-25T05:34:04.760855+0000","last_unstale":"2024-04-25T05:36:40.731954+0000","last_undegraded":"2024-04-25T05:36:40.731954+0000","last_fullsized":"2024-04-25T05:36:40.731954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.701074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.701074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:57.222127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.717770+0000","last_change":"2024-04-25T05:35:36.294561+0000","last_active":"2024-04-25T05:36:40.717770+0000","last_peered":"2024-04-25T05:36:40.717770+0000","last_clean":"2024-04-25T05:36:40.717770+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:36:40.717770+0000","last_undegraded":"2024-04-25T05:36:40.717770+0000","last_fullsized":"2024-04-25T05:36:40.717770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.239247+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716563+0000","last_change":"2024-04-25T05:34:00.686225+0000","last_active":"2024-04-25T05:36:40.716563+0000","last_peered":"2024-04-25T05:36:40.716563+0000","last_clean":"2024-04-25T05:36:40.716563+0000","last_became_active":"2024-04-25T05:34:00.685854+0000","last_became_peered":"2024-04-25T05:34:00.685854+0000","last_unstale":"2024-04-25T05:36:40.716563+0000","last_undegraded":"2024-04-25T05:36:40.716563+0000","last_fullsized":"2024-04-25T05:36:40.716563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.672943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":171,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724195+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:36:40.724195+0000","last_peered":"2024-04-25T05:36:40.724195+0000","last_clean":"2024-04-25T05:36:40.724195+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:36:40.724195+0000","last_undegraded":"2024-04-25T05:36:40.724195+0000","last_fullsized":"2024-04-25T05:36:40.724195+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:43.089867+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"126'7065","reported_seq":7151,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.779490+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:36:42.779490+0000","last_peered":"2024-04-25T05:36:42.779490+0000","last_clean":"2024-04-25T05:36:42.779490+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:36:42.779490+0000","last_undegraded":"2024-04-25T05:36:42.779490+0000","last_fullsized":"2024-04-25T05:36:42.779490+0000","mapping_epoch":88,"log_start":"98'2400","ondisk_log_start":"98'2400","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:46.541252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463011840,"num_objects":7065,"num_object_clones":0,"num_object_copies":21195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":452160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.714527+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:36:40.714527+0000","last_peered":"2024-04-25T05:36:40.714527+0000","last_clean":"2024-04-25T05:36:40.714527+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:36:40.714527+0000","last_undegraded":"2024-04-25T05:36:40.714527+0000","last_fullsized":"2024-04-25T05:36:40.714527+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.155991+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.726524+0000","last_change":"2024-04-25T05:35:06.377537+0000","last_active":"2024-04-25T05:36:40.726524+0000","last_peered":"2024-04-25T05:36:40.726524+0000","last_clean":"2024-04-25T05:36:40.726524+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:36:40.726524+0000","last_undegraded":"2024-04-25T05:36:40.726524+0000","last_fullsized":"2024-04-25T05:36:40.726524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":190,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724099+0000","last_change":"2024-04-25T05:35:30.243679+0000","last_active":"2024-04-25T05:36:40.724099+0000","last_peered":"2024-04-25T05:36:40.724099+0000","last_clean":"2024-04-25T05:36:40.724099+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:36:40.724099+0000","last_undegraded":"2024-04-25T05:36:40.724099+0000","last_fullsized":"2024-04-25T05:36:40.724099+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716258+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:36:40.716258+0000","last_peered":"2024-04-25T05:36:40.716258+0000","last_clean":"2024-04-25T05:36:40.716258+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:36:40.716258+0000","last_undegraded":"2024-04-25T05:36:40.716258+0000","last_fullsized":"2024-04-25T05:36:40.716258+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:37.962428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"126'7043","reported_seq":7129,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.686816+0000","last_change":"2024-04-25T05:35:09.678603+0000","last_active":"2024-04-25T05:36:42.686816+0000","last_peered":"2024-04-25T05:36:42.686816+0000","last_clean":"2024-04-25T05:36:42.686816+0000","last_became_active":"2024-04-25T05:35:09.678239+0000","last_became_peered":"2024-04-25T05:35:09.678239+0000","last_unstale":"2024-04-25T05:36:42.686816+0000","last_undegraded":"2024-04-25T05:36:42.686816+0000","last_fullsized":"2024-04-25T05:36:42.686816+0000","mapping_epoch":88,"log_start":"93'700","ondisk_log_start":"93'700","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:43.871030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461570048,"num_objects":7043,"num_object_clones":0,"num_object_copies":21129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":126,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.728873+0000","last_change":"2024-04-25T05:35:27.774757+0000","last_active":"2024-04-25T05:36:40.728873+0000","last_peered":"2024-04-25T05:36:40.728873+0000","last_clean":"2024-04-25T05:36:40.728873+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:36:40.728873+0000","last_undegraded":"2024-04-25T05:36:40.728873+0000","last_fullsized":"2024-04-25T05:36:40.728873+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.774595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.774595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.987948+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":162,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716120+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:36:40.716120+0000","last_peered":"2024-04-25T05:36:40.716120+0000","last_clean":"2024-04-25T05:36:40.716120+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:36:40.716120+0000","last_undegraded":"2024-04-25T05:36:40.716120+0000","last_fullsized":"2024-04-25T05:36:40.716120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:45.673066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":138,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724018+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:36:40.724018+0000","last_peered":"2024-04-25T05:36:40.724018+0000","last_clean":"2024-04-25T05:36:40.724018+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:36:40.724018+0000","last_undegraded":"2024-04-25T05:36:40.724018+0000","last_fullsized":"2024-04-25T05:36:40.724018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:04.338804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":159,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.730652+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:36:40.730652+0000","last_peered":"2024-04-25T05:36:40.730652+0000","last_clean":"2024-04-25T05:36:40.730652+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:36:40.730652+0000","last_undegraded":"2024-04-25T05:36:40.730652+0000","last_fullsized":"2024-04-25T05:36:40.730652+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:55.776165+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"126'7139","reported_seq":7225,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.639831+0000","last_change":"2024-04-25T05:35:09.684061+0000","last_active":"2024-04-25T05:36:42.639831+0000","last_peered":"2024-04-25T05:36:42.639831+0000","last_clean":"2024-04-25T05:36:42.639831+0000","last_became_active":"2024-04-25T05:35:09.683699+0000","last_became_peered":"2024-04-25T05:35:09.683699+0000","last_unstale":"2024-04-25T05:36:42.639831+0000","last_undegraded":"2024-04-25T05:36:42.639831+0000","last_fullsized":"2024-04-25T05:36:42.639831+0000","mapping_epoch":88,"log_start":"93'800","ondisk_log_start":"93'800","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":6339,"log_dups_size":0,"ondisk_log_size":6339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467861504,"num_objects":7139,"num_object_clones":0,"num_object_copies":21417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.725976+0000","last_change":"2024-04-25T05:34:04.761846+0000","last_active":"2024-04-25T05:36:40.725976+0000","last_peered":"2024-04-25T05:36:40.725976+0000","last_clean":"2024-04-25T05:36:40.725976+0000","last_became_active":"2024-04-25T05:34:04.761529+0000","last_became_peered":"2024-04-25T05:34:04.761529+0000","last_unstale":"2024-04-25T05:36:40.725976+0000","last_undegraded":"2024-04-25T05:36:40.725976+0000","last_fullsized":"2024-04-25T05:36:40.725976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.729648+0000","last_change":"2024-04-25T05:35:25.709565+0000","last_active":"2024-04-25T05:36:40.729648+0000","last_peered":"2024-04-25T05:36:40.729648+0000","last_clean":"2024-04-25T05:36:40.729648+0000","last_became_active":"2024-04-25T05:34:02.698945+0000","last_became_peered":"2024-04-25T05:34:02.698945+0000","last_unstale":"2024-04-25T05:36:40.729648+0000","last_undegraded":"2024-04-25T05:36:40.729648+0000","last_fullsized":"2024-04-25T05:36:40.729648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:25.709444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.709444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:42.994941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":137,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716343+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:36:40.716343+0000","last_peered":"2024-04-25T05:36:40.716343+0000","last_clean":"2024-04-25T05:36:40.716343+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:36:40.716343+0000","last_undegraded":"2024-04-25T05:36:40.716343+0000","last_fullsized":"2024-04-25T05:36:40.716343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:48.743618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":164,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715091+0000","last_change":"2024-04-25T05:36:15.740901+0000","last_active":"2024-04-25T05:36:40.715091+0000","last_peered":"2024-04-25T05:36:40.715091+0000","last_clean":"2024-04-25T05:36:40.715091+0000","last_became_active":"2024-04-25T05:34:00.692269+0000","last_became_peered":"2024-04-25T05:34:00.692269+0000","last_unstale":"2024-04-25T05:36:40.715091+0000","last_undegraded":"2024-04-25T05:36:40.715091+0000","last_fullsized":"2024-04-25T05:36:40.715091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:23.810446+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"126'7063","reported_seq":7210,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.688903+0000","last_change":"2024-04-25T05:36:32.972424+0000","last_active":"2024-04-25T05:36:42.688903+0000","last_peered":"2024-04-25T05:36:42.688903+0000","last_clean":"2024-04-25T05:36:42.688903+0000","last_became_active":"2024-04-25T05:35:09.678971+0000","last_became_peered":"2024-04-25T05:35:09.678971+0000","last_unstale":"2024-04-25T05:36:42.688903+0000","last_undegraded":"2024-04-25T05:36:42.688903+0000","last_fullsized":"2024-04-25T05:36:42.688903+0000","mapping_epoch":88,"log_start":"93'200","ondisk_log_start":"93'200","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":5549,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:44.421910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462880768,"num_objects":7063,"num_object_clones":0,"num_object_copies":21189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":452032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.722293+0000","last_change":"2024-04-25T05:35:39.752489+0000","last_active":"2024-04-25T05:36:40.722293+0000","last_peered":"2024-04-25T05:36:40.722293+0000","last_clean":"2024-04-25T05:36:40.722293+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:36:40.722293+0000","last_undegraded":"2024-04-25T05:36:40.722293+0000","last_fullsized":"2024-04-25T05:36:40.722293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:39.752319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.752319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:54.553556+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.736926+0000","last_change":"2024-04-25T05:35:33.279820+0000","last_active":"2024-04-25T05:36:40.736926+0000","last_peered":"2024-04-25T05:36:40.736926+0000","last_clean":"2024-04-25T05:36:40.736926+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:36:40.736926+0000","last_undegraded":"2024-04-25T05:36:40.736926+0000","last_fullsized":"2024-04-25T05:36:40.736926+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:33.279557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:33.279557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:41.815997+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":146,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.711157+0000","last_change":"2024-04-25T05:36:31.976928+0000","last_active":"2024-04-25T05:36:40.711157+0000","last_peered":"2024-04-25T05:36:40.711157+0000","last_clean":"2024-04-25T05:36:40.711157+0000","last_became_active":"2024-04-25T05:33:58.682482+0000","last_became_peered":"2024-04-25T05:33:58.682482+0000","last_unstale":"2024-04-25T05:36:40.711157+0000","last_undegraded":"2024-04-25T05:36:40.711157+0000","last_fullsized":"2024-04-25T05:36:40.711157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:37.473760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":161,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.716285+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:36:40.716285+0000","last_peered":"2024-04-25T05:36:40.716285+0000","last_clean":"2024-04-25T05:36:40.716285+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:36:40.716285+0000","last_undegraded":"2024-04-25T05:36:40.716285+0000","last_fullsized":"2024-04-25T05:36:40.716285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:53.955851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"126'7301","reported_seq":7387,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:42.820945+0000","last_change":"2024-04-25T05:35:09.682779+0000","last_active":"2024-04-25T05:36:42.820945+0000","last_peered":"2024-04-25T05:36:42.820945+0000","last_clean":"2024-04-25T05:36:42.820945+0000","last_became_active":"2024-04-25T05:35:09.681887+0000","last_became_peered":"2024-04-25T05:35:09.681887+0000","last_unstale":"2024-04-25T05:36:42.820945+0000","last_undegraded":"2024-04-25T05:36:42.820945+0000","last_fullsized":"2024-04-25T05:36:42.820945+0000","mapping_epoch":88,"log_start":"95'1700","ondisk_log_start":"95'1700","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:36:47.239469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478478336,"num_objects":7301,"num_object_clones":0,"num_object_copies":21903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7301,"num_write_kb":467264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715380+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:36:40.715380+0000","last_peered":"2024-04-25T05:36:40.715380+0000","last_clean":"2024-04-25T05:36:40.715380+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:36:40.715380+0000","last_undegraded":"2024-04-25T05:36:40.715380+0000","last_fullsized":"2024-04-25T05:36:40.715380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.276359+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":174,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.727520+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:36:40.727520+0000","last_peered":"2024-04-25T05:36:40.727520+0000","last_clean":"2024-04-25T05:36:40.727520+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:36:40.727520+0000","last_undegraded":"2024-04-25T05:36:40.727520+0000","last_fullsized":"2024-04-25T05:36:40.727520+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:00.813928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.724869+0000","last_change":"2024-04-25T05:33:58.674980+0000","last_active":"2024-04-25T05:36:40.724869+0000","last_peered":"2024-04-25T05:36:40.724869+0000","last_clean":"2024-04-25T05:36:40.724869+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:36:40.724869+0000","last_undegraded":"2024-04-25T05:36:40.724869+0000","last_fullsized":"2024-04-25T05:36:40.724869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-04-25T05:36:40.715858+0000","last_change":"2024-04-25T05:36:30.640999+0000","last_active":"2024-04-25T05:36:40.715858+0000","last_peered":"2024-04-25T05:36:40.715858+0000","last_clean":"2024-04-25T05:36:40.715858+0000","last_became_active":"2024-04-25T05:34:00.686085+0000","last_became_peered":"2024-04-25T05:34:00.686085+0000","last_unstale":"2024-04-25T05:36:40.715858+0000","last_undegraded":"2024-04-25T05:36:40.715858+0000","last_fullsized":"2024-04-25T05:36:40.715858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.312587+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7435714560,"num_objects":113460,"num_object_clones":0,"num_object_copies":340380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113460,"num_write_kb":7261440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22308782080,"data_stored":22308782080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90060,"ondisk_log_size":90060,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070502,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2336476,"kb_used_data":2265044,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91408676,"statfs":{"total":95995035648,"available":93602484224,"internally_reserved":0,"allocated":2319405056,"data_stored":2318745817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364842,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2336840,"kb_used_data":2261244,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91408312,"statfs":{"total":95995035648,"available":93602111488,"internally_reserved":0,"allocated":2315513856,"data_stored":2314878095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2765108,"kb_used_data":2697832,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90980044,"statfs":{"total":95995035648,"available":93163565056,"internally_reserved":0,"allocated":2762579968,"data_stored":2761898743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3710552,"kb_used_data":3629388,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":90034600,"statfs":{"total":95995035648,"available":92195430400,"internally_reserved":0,"allocated":3716493312,"data_stored":3715840532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2779264,"kb_used_data":2711988,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90965888,"statfs":{"total":95995035648,"available":93149069312,"internally_reserved":0,"allocated":2777075712,"data_stored":2776446441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4213980,"kb_used_data":4098132,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":89531172,"statfs":{"total":95995035648,"available":91679920128,"internally_reserved":0,"allocated":4196487168,"data_stored":4195825749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":118618567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2316836,"kb_used_data":2245396,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91428316,"statfs":{"total":95995035648,"available":93622595584,"internally_reserved":0,"allocated":2299285504,"data_stored":2298624805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691887,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856012,"kb_used_data":1798460,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91889140,"statfs":{"total":95995035648,"available":94094479360,"internally_reserved":0,"allocated":1841623040,"data_stored":1840986211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3729850368,"data_stored":3729850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2780495872,"data_stored":2780495872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2780889088,"data_stored":2780889088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4201381888,"data_stored":4201381888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2316959744,"data_stored":2316959744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1849688064,"data_stored":1849688064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2328231936,"data_stored":2328231936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2321285120,"data_stored":2321285120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:36:49.836 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:36:49.836 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:36:49.836 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:36:49.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd out 6 2024-04-25T05:36:49.881 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:49.881 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:36:49.881 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:36:49.881 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:36:49.881 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:49.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:49.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:49.919 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:36:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:36:50.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.035 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:50.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:50.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:50.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:50.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:50.314 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:50.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:50.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:50.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:50 smithi060 ceph-mon[30898]: 7.5 scrub starts 2024-04-25T05:36:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:50 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:36:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:50 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:36:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:50 smithi060 ceph-mon[30898]: 7.0 scrub starts 2024-04-25T05:36:50.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:50.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:36:50.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[37492]: 7.5 scrub starts 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[37492]: 7.0 scrub starts 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[27010]: 7.5 scrub starts 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:36:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:50 smithi043 ceph-mon[27010]: 7.0 scrub starts 2024-04-25T05:36:50.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:36:50.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:50.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:50.816 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:50.853 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:50.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:50.943 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:36:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:50.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:50.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:51.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:51.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:51.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:51.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:51.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.16k op/s 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: 4.1a scrub starts 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: 4.1a scrub ok 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.16k op/s 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: 4.1a scrub starts 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: 4.1a scrub ok 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:51 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:36:51.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.392 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:51.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:51.398 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.16k op/s 2024-04-25T05:36:51.398 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: 4.1a scrub starts 2024-04-25T05:36:51.398 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: 4.1a scrub ok 2024-04-25T05:36:51.398 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:36:51.399 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:51 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:36:51.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:51.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:45776 7.5 7:a01b19d2:::benchmark_data_smithi043_74356_object45775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:40.462135+0000", 2024-04-25T05:36:51.592 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.127254871999995, 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079471440000000004, 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 45776 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462135+0000", 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462135+0000", 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462136+0000", 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2330000000000001e-06 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462177+0000", 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1331999999999997e-05 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462186+0000", 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1009999999999998e-06 2024-04-25T05:36:51.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462194+0000", 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7090000000000002e-06 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462301+0000", 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010639799999999999 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462465+0000", 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016489 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.462632+0000", 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000166119 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.466934+0000", 2024-04-25T05:36:51.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043022429999999999 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467847+0000", 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00091285699999999999 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541509+0000", 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073662796000000003 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541570+0000", 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0826999999999999e-05 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541606+0000", 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5935000000000001e-05 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.599 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:45783 7.5 7:a61f2677:::benchmark_data_smithi043_74356_object45782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:40.464967+0000", 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.124422577000004, 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.076799237000000006, 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 45783 2024-04-25T05:36:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.464967+0000", 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.464967+0000", 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.464969+0000", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5370000000000001e-06 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465122+0000", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015344600000000001 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465135+0000", 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2775000000000001e-05 2024-04-25T05:36:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465143+0000", 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7999999999999999e-06 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465222+0000", 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9604999999999994e-05 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465357+0000", 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013433999999999999 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.465504+0000", 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014715299999999999 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467082+0000", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015786820000000001 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.467896+0000", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00081346800000000003 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541656+0000", 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073760168000000001 2024-04-25T05:36:51.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541713+0000", 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7355000000000002e-05 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.541766+0000", 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2908000000000002e-05 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:58637 7.5 7:ace55749:::benchmark_data_smithi043_74356_object58636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:49.780850+0000", 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.808540059999999, 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073732107000000005, 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.609 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 58637 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780850+0000", 2024-04-25T05:36:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780850+0000", 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780851+0000", 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4160000000000001e-06 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780923+0000", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2278000000000002e-05 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780936+0000", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3063e-05 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.780945+0000", 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.259e-06 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781011+0000", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5963000000000005e-05 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781139+0000", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128014 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781272+0000", 2024-04-25T05:36:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013349400000000001 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784563+0000", 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003291389 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.786222+0000", 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001658784 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854476+0000", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.068253505000000006 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854535+0000", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8879999999999999e-05 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854582+0000", 2024-04-25T05:36:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7061999999999998e-05 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:58639 7.5 7:aa417560:::benchmark_data_smithi043_74356_object58638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:49.781196+0000", 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.808194118000003, 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073550090999999998, 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 58639 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781196+0000", 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781196+0000", 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781196+0000", 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0100000000000004e-07 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781224+0000", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7943e-05 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781233+0000", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0410000000000003e-06 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.781241+0000", 2024-04-25T05:36:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9880000000000004e-06 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.783124+0000", 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001883149 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.783243+0000", 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011821500000000001 2024-04-25T05:36:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.783385+0000", 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014227 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.784713+0000", 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013283850000000001 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.786270+0000", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00155715 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854635+0000", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.068364528999999993 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854704+0000", 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8801999999999996e-05 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:49.854746+0000", 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1817999999999998e-05 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:59421 7.5 7:a7f3e364:::benchmark_data_smithi043_74356_object59420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:50.453178+0000", 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.136211936999999, 2024-04-25T05:36:51.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073435393000000002, 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59421 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453178+0000", 2024-04-25T05:36:51.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453178+0000", 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453178+0000", 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6300000000000005e-07 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453246+0000", 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7711000000000003e-05 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453255+0000", 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3149999999999998e-06 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.453269+0000", 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3342e-05 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.454724+0000", 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014555390000000001 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.454854+0000", 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012968699999999999 2024-04-25T05:36:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455009+0000", 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015473799999999999 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.456839+0000", 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001830466 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.457710+0000", 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00087096999999999999 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.526472+0000", 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.068762161000000002 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.526563+0000", 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0227000000000002e-05 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.526613+0000", 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0574000000000001e-05 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:59422 7.5 7:a48a7a3b:::benchmark_data_smithi043_74356_object59421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:50.455461+0000", 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 61.133928775999998, 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072149147999999996, 2024-04-25T05:36:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59422 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455461+0000", 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455461+0000", 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455462+0000", 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1800000000000004e-07 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455511+0000", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8918999999999997e-05 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455523+0000", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.252e-05 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455531+0000", 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6839999999999996e-06 2024-04-25T05:36:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.455986+0000", 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00045523200000000002 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.456072+0000", 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5532999999999996e-05 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.456156+0000", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3945999999999999e-05 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.456986+0000", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00083054500000000002 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.457768+0000", 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00078146699999999999 2024-04-25T05:36:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.527497+0000", 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.069729399999999997 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.527569+0000", 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1978999999999996e-05 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.527610+0000", 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1004999999999998e-05 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:59950 7.5 7:a4b1a940:::benchmark_data_smithi043_74356_object59949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:50.910380+0000", 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 60.679009675000003, 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.069215181000000001, 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.637 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 59950 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910380+0000", 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910380+0000", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910382+0000", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1210000000000001e-06 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910420+0000", 2024-04-25T05:36:51.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7722999999999998e-05 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910441+0000", 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1084999999999999e-05 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910452+0000", 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1523e-05 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910520+0000", 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7906999999999999e-05 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910683+0000", 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016237099999999999 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.910823+0000", 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000140629 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.913528+0000", 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027042590000000001 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.914432+0000", 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00090394 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.979479+0000", 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.065047736999999994 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.979550+0000", 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0340999999999996e-05 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:50.979595+0000", 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5544999999999998e-05 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.643 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61441 7.5 7:a916e294:::benchmark_data_smithi043_74356_object61440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.414893472999999, 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098210427000000003, 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.644 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61441 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174498+0000", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6130000000000001e-06 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174543+0000", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5485999999999998e-05 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174552+0000", 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174557+0000", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2220000000000001e-06 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175099+0000", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000542155 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175220+0000", 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121154 2024-04-25T05:36:51.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175330+0000", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109385 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176611+0000", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001281199 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.182295+0000", 2024-04-25T05:36:51.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056836250000000003 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185068+0000", 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027733480000000001 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185119+0000", 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1099e-05 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272707+0000", 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087587571000000003 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61442 7.5 7:af9252c3:::benchmark_data_smithi043_74356_object61441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.412577192999997, 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096090660999999994, 2024-04-25T05:36:51.650 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61442 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:36:51.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176814+0000", 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.533e-06 2024-04-25T05:36:51.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176873+0000", 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8678000000000003e-05 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176883+0000", 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0701e-05 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176892+0000", 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4479999999999994e-06 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176940+0000", 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8111000000000001e-05 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:51.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177024+0000", 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4243e-05 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177104+0000", 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9800000000000002e-05 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.178395+0000", 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001291489 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184162+0000", 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005766658 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272787+0000", 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088625132999999995 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272844+0000", 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209000000000001e-05 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272903+0000", 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8658e-05 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:61447 7.6 e104/88 7:60edaceb:::benchmark_data_smithi043_74356_object61446:head v 104'3778, mlcod=104'3775)", 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.183869+0000", 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.405520871, 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088966628000000006, 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183869+0000", 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183869+0000", 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183869+0000", 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0999999999999998e-07 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183940+0000", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0496999999999999e-05 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183947+0000", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8630000000000002e-06 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183959+0000", 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1762e-05 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184017+0000", 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8774999999999999e-05 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184037+0000", 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9785000000000001e-05 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272778+0000", 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088740747999999994 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272835+0000", 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7587999999999998e-05 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:61449 7.f e104/88 7:f5689256:::benchmark_data_smithi043_74356_object61448:head v 104'3919, mlcod=104'3917)", 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.183972+0000", 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.405418142999999, 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.089044631999999999, 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183972+0000", 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183972+0000", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.183972+0000", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3099999999999999e-07 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184014+0000", 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2370999999999998e-05 2024-04-25T05:36:51.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184021+0000", 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3369999999999996e-06 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184026+0000", 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7060000000000001e-06 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184073+0000", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7142999999999999e-05 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184092+0000", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8995999999999999e-05 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272969+0000", 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088876137999999993 2024-04-25T05:36:51.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.273016+0000", 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.761e-05 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:61441 7.5 e104/88 ondisk, result = 0)", 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.184902+0000", 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.404487185999997, 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087824674000000005, 2024-04-25T05:36:51.668 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184902+0000", 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184902+0000", 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184903+0000", 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.17e-07 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184905+0000", 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1900000000000002e-06 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184910+0000", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6280000000000004e-06 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184914+0000", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.391e-06 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185058+0000", 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000143833 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185064+0000", 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.404e-06 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272727+0000", 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087663010999999999 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.672 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:61442 7.5 e104/88 ondisk, result = 0)", 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.186868+0000", 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.402521655999998, 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.086041589000000002, 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.186868+0000", 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.186868+0000", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.186869+0000", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0900000000000001e-07 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.186873+0000", 2024-04-25T05:36:51.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4390000000000002e-06 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272638+0000", 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.085765327000000002 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272663+0000", 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4156e-05 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272773+0000", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110572 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272781+0000", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2220000000000002e-06 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272910+0000", 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128264 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.676 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:61460 7.c e104/88 7:3ee2d5b3:::benchmark_data_smithi043_74356_object61459:head v 104'3830, mlcod=104'3828)", 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.187729+0000", 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.401660274000001, 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.089878471000000001, 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.187729+0000", 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.187729+0000", 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.187730+0000", 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2099999999999996e-07 2024-04-25T05:36:51.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.187752+0000", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1610000000000001e-05 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272623+0000", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.084871584999999999 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272650+0000", 2024-04-25T05:36:51.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7039999999999999e-05 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272767+0000", 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116817 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272792+0000", 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4791999999999999e-05 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.277541+0000", 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047495419999999998 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.277608+0000", 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6464000000000006e-05 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:61465 7.8 e104/88 7:13ebdc7c:::benchmark_data_smithi043_74356_object61464:head v 104'3813, mlcod=104'3811)", 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.192541+0000", 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 59.396849060000001, 2024-04-25T05:36:51.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087869322, 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.192541+0000", 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.192541+0000", 2024-04-25T05:36:51.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.192542+0000", 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.569999999999999e-07 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.192581+0000", 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8962999999999999e-05 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272620+0000", 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.080039939000000004 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272647+0000", 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6665999999999998e-05 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272763+0000", 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011618899999999999 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272785+0000", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2137000000000002e-05 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.280374+0000", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075890020000000001 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.280410+0000", 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5469000000000002e-05 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.685 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:63838 7.5 7:aa8bfa8d:::benchmark_data_smithi043_74356_object63837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:54.191996+0000", 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 57.397393616000002, 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072901715000000006, 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.686 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 63838 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.191996+0000", 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.191996+0000", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.191997+0000", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.034e-06 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192051+0000", 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3749999999999999e-05 2024-04-25T05:36:51.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192061+0000", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8279999999999998e-06 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192066+0000", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6710000000000004e-06 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192107+0000", 2024-04-25T05:36:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1124999999999997e-05 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192207+0000", 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9461000000000002e-05 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.192301+0000", 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3608000000000002e-05 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.193828+0000", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015270769999999999 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.201285+0000", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0074575680000000004 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.264771+0000", 2024-04-25T05:36:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.063485733000000003 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.264848+0000", 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7521999999999994e-05 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.264898+0000", 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9338000000000001e-05 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:63850 7.5 7:a92ea53d:::benchmark_data_smithi043_74356_object63849:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:54.197665+0000", 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 57.391724721999999, 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.070288259000000006, 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 63850 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197665+0000", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197665+0000", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197665+0000", 2024-04-25T05:36:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0999999999999999e-07 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197694+0000", 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8750000000000001e-05 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197700+0000", 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1469999999999998e-06 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.197706+0000", 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4240000000000001e-06 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.201327+0000", 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036210690000000002 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.201468+0000", 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014082400000000001 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.201612+0000", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014406900000000001 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.202967+0000", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013550680000000001 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.252201+0000", 2024-04-25T05:36:51.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.049234689999999998 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.267876+0000", 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015674873999999998 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.267922+0000", 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5838999999999999e-05 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:54.267953+0000", 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0994999999999997e-05 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:99680 7.5 7:a6da5147:::benchmark_data_smithi043_74356_object99679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:27.931333+0000", 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 23.658057064000001, 2024-04-25T05:36:51.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073670490000000005, 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 99680 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931333+0000", 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931333+0000", 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931334+0000", 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.108e-06 2024-04-25T05:36:51.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931399+0000", 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5080999999999996e-05 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931425+0000", 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6055e-05 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931435+0000", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6760000000000002e-06 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931499+0000", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4168e-05 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931665+0000", 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016585499999999999 2024-04-25T05:36:51.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.931827+0000", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162776 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.935108+0000", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003280289 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:27.935338+0000", 2024-04-25T05:36:51.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023051300000000001 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.004925+0000", 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.069586736999999996 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.004963+0000", 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7669999999999997e-05 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:28.005003+0000", 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0562000000000002e-05 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:101550 7.5 7:abb10cfa:::benchmark_data_smithi043_74356_object101549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:29.905136+0000", 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 21.684253171999998, 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079165868, 2024-04-25T05:36:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 101550 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905136+0000", 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905136+0000", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905141+0000", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8250000000000004e-06 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905183+0000", 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1372000000000003e-05 2024-04-25T05:36:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905200+0000", 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7014999999999999e-05 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905208+0000", 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4540000000000007e-06 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905635+0000", 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00042685100000000001 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905768+0000", 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132631 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.905915+0000", 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147102 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.909817+0000", 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039025930000000002 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.914749+0000", 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049312849999999997 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.984176+0000", 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.069427830999999995 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.984241+0000", 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4487999999999996e-05 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:29.984302+0000", 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1420999999999994e-05 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:103735 7.5 7:afd07efd:::benchmark_data_smithi043_74356_object103734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:36:32.541375+0000", 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.048015084999999, 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.076794555, 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:36:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 103735 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541375+0000", 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541375+0000", 2024-04-25T05:36:51.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541375+0000", 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2099999999999995e-07 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541439+0000", 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3516000000000006e-05 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541448+0000", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.38e-06 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.541457+0000", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2719999999999997e-06 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.544075+0000", 2024-04-25T05:36:51.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026188180000000002 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.544220+0000", 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000144435 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.544391+0000", 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000171242 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.547099+0000", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027083279999999999 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.547681+0000", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00058133600000000001 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.618010+0000", 2024-04-25T05:36:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.070328821 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.618107+0000", 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7394999999999998e-05 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:36:32.618169+0000", 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2090999999999999e-05 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.719 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:36:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:36:51.720 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:36:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:51.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:51.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:51.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:51.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.038 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:52.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:52.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:52.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:36:52.183 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:36:52.184 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:36:52.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:52.294 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[37492]: 7.e scrub starts 2024-04-25T05:36:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[37492]: 7.5 scrub ok 2024-04-25T05:36:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1254162758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[37492]: 7.0 scrub ok 2024-04-25T05:36:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[27010]: 7.e scrub starts 2024-04-25T05:36:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[27010]: 7.5 scrub ok 2024-04-25T05:36:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1254162758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-mon[27010]: 7.0 scrub ok 2024-04-25T05:36:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:52 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:36:52.313+0000 7f173ec68700 -1 mon.a@0(leader).osd e128 definitely_dead 0 2024-04-25T05:36:52.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:52.458 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:52.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:52 smithi060 ceph-mon[30898]: 7.e scrub starts 2024-04-25T05:36:52.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:52 smithi060 ceph-mon[30898]: 7.5 scrub ok 2024-04-25T05:36:52.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1254162758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:36:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:52 smithi060 ceph-mon[30898]: 7.0 scrub ok 2024-04-25T05:36:52.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:36:52.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:52.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:52.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:52.769 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:52.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:52.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:52.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:53.023 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:53.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:53.112 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:53.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:53.154 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:53.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:36:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:36:53.219 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.6. 2024-04-25T05:36:53.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:53.280 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:53.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:53.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:53.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 27 MiB/s wr, 570 op/s 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 7.d scrub starts 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:36:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:53 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:36:53.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:53.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:53.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 27 MiB/s wr, 570 op/s 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 7.d scrub starts 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:36:53.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:36:53.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:36:53.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:36:53.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 27 MiB/s wr, 570 op/s 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 7.d scrub starts 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: Client client.admin marked osd.6 out, while it was still marked up 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:36:53.646 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:36:53.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:53 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:36:53.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:53.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:53.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:53.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:53.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:53.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:54.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:54.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:54.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:54.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:54.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:54.276 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:54.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:54.399 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 7.9 scrub starts 2024-04-25T05:36:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-25T05:36:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: osdmap e129: 8 total, 8 up, 7 in 2024-04-25T05:36:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 7.e scrub ok 2024-04-25T05:36:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:36:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:36:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:36:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:54 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:36:54.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:54.523 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:54.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:54.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 7.9 scrub starts 2024-04-25T05:36:54.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-25T05:36:54.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: osdmap e129: 8 total, 8 up, 7 in 2024-04-25T05:36:54.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 7.e scrub ok 2024-04-25T05:36:54.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:36:54.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:36:54.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:36:54.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 7.9 scrub starts 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3277421909' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: osdmap e129: 8 total, 8 up, 7 in 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 7.e scrub ok 2024-04-25T05:36:54.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:36:54.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:36:54.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:36:54.645 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:54 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:36:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:54.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:54.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:54.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:54.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:54.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:54.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:36:55.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:55.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:55.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:55.171 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:55.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: pgmap v125: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.17k op/s 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 7.9 scrub ok 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: osdmap e130: 8 total, 8 up, 7 in 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:36:55.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: pgmap v125: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.17k op/s 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 7.9 scrub ok 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: osdmap e130: 8 total, 8 up, 7 in 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:36:55.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:55 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:36:55.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:55.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:55.422 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:55.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: pgmap v125: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.17k op/s 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 7.9 scrub ok 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: osdmap e130: 8 total, 8 up, 7 in 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:36:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:55 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:36:55.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.549 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:55.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:55.676 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:55.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:55.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:55.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:55.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:56.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:56.153 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:56.177 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:56.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:56.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:56.321 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:36:56.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:56.445 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:56.452 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:36:56.452 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:36:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:36:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7812 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'7629 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'7610 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'7754 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: osdmap e131: 8 total, 8 up, 7 in 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7800 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2503482271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:56.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:36:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:36:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7812 2024-04-25T05:36:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'7629 2024-04-25T05:36:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'7610 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'7754 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: osdmap e131: 8 total, 8 up, 7 in 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7800 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2503482271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7812 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'7629 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 128'7610 2024-04-25T05:36:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 128'7754 2024-04-25T05:36:56.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: osdmap e131: 8 total, 8 up, 7 in 2024-04-25T05:36:56.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'7800 2024-04-25T05:36:56.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2503482271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:36:56.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:56.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:56.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:56.837 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:56.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:56.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:57.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:57.122 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:57.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:36:57.249 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:36:57.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:57.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[27010]: pgmap v128: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[27010]: osdmap e132: 8 total, 8 up, 7 in 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[37492]: pgmap v128: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:36:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:36:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:57 smithi043 ceph-mon[37492]: osdmap e132: 8 total, 8 up, 7 in 2024-04-25T05:36:57.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:57 smithi060 ceph-mon[30898]: pgmap v128: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:36:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:57 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:36:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:57 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:36:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:36:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:57 smithi060 ceph-mon[30898]: osdmap e132: 8 total, 8 up, 7 in 2024-04-25T05:36:57.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:57.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:57.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:57.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:57.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:57.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:57.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:57.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:58.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:58.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:58.172 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:58.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:58.300 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:58.402 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:36:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:58.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:36:58.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:58.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:58.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:58 smithi043 ceph-mon[27010]: osdmap e133: 8 total, 8 up, 7 in 2024-04-25T05:36:58.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:36:58 smithi043 ceph-mon[27010]: pgmap v131: 148 pgs: 1 active+remapped+backfilling, 145 active+clean, 2 active+clean+scrubbing; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 119 op/s; 6574/332028 objects misplaced (1.980%); 4.5 MiB/s, 84 objects/s recovering 2024-04-25T05:36:58.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:58 smithi043 ceph-mon[37492]: osdmap e133: 8 total, 8 up, 7 in 2024-04-25T05:36:58.639 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:36:58 smithi043 ceph-mon[37492]: pgmap v131: 148 pgs: 1 active+remapped+backfilling, 145 active+clean, 2 active+clean+scrubbing; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 119 op/s; 6574/332028 objects misplaced (1.980%); 4.5 MiB/s, 84 objects/s recovering 2024-04-25T05:36:58.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:58.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:58.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:58 smithi060 ceph-mon[30898]: osdmap e133: 8 total, 8 up, 7 in 2024-04-25T05:36:58.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:36:58 smithi060 ceph-mon[30898]: pgmap v131: 148 pgs: 1 active+remapped+backfilling, 145 active+clean, 2 active+clean+scrubbing; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 119 op/s; 6574/332028 objects misplaced (1.980%); 4.5 MiB/s, 84 objects/s recovering 2024-04-25T05:36:58.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:36:58.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:36:58.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:36:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:58.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:36:59.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:59.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:59.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:36:59.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:36:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.263 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:59.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:36:59.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:36:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:36:59.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:59.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:59.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:36:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:36:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:36:59.740 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:36:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:36:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:36:59.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:59.874 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:36:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:36:59.920 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:36:59.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:36:59.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:36:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:36:59.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:36:59.965 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:00.022 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:00.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:00.064 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:00.103 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:00.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:00.144 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:00.192 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:00.197 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:00.201 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:37:00.201 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:37:00.202 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T05:37:00.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:37:00.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:00.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:37:00.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:00.417 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:00.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:00.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:00.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:00.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: osdmap e134: 8 total, 8 up, 7 in 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: pgmap v133: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 564 op/s; 32915/325860 objects misplaced (10.101%); 14 MiB/s, 247 objects/s recovering 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/19306388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: osdmap e134: 8 total, 8 up, 7 in 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: pgmap v133: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 564 op/s; 32915/325860 objects misplaced (10.101%); 14 MiB/s, 247 objects/s recovering 2024-04-25T05:37:00.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/19306388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:00.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:00.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:00.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:00.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: osdmap e134: 8 total, 8 up, 7 in 2024-04-25T05:37:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:37:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:37:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: pgmap v133: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 456 B/s rd, 0 B/s wr, 564 op/s; 32915/325860 objects misplaced (10.101%); 14 MiB/s, 247 objects/s recovering 2024-04-25T05:37:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/19306388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:00.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:00.790 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:37:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:37:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:37:00.790 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:00.793 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:00.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:00.921 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:00.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:01.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:01.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:01.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.176 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:01.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:01.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:01.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:01.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:01.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:01.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:01 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:37:01.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:01 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:37:01.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:01 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:37:01.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:01 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:37:01.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:01.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:01.627 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:01 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:37:01.627 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:01 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:37:01.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:01.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:01.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:01.890 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:01.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:01.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:02.017 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:02.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:02.154 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:02.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:02.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:02.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:02.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:02.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:02.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[27010]: pgmap v134: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 422 op/s; 32915/325860 objects misplaced (10.101%); 11 MiB/s, 184 objects/s recovering 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[37492]: pgmap v134: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 422 op/s; 32915/325860 objects misplaced (10.101%); 11 MiB/s, 184 objects/s recovering 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:02.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:02.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:37:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:37:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-mon[30898]: pgmap v134: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 422 op/s; 32915/325860 objects misplaced (10.101%); 11 MiB/s, 184 objects/s recovering 2024-04-25T05:37:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:02.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:02.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:02.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:02.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:02.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:03.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:03.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:03.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.175 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:03.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:03.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:03.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.297 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:03.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:03.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:37:03.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:03.423 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:03.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:03.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:03.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:03.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:03.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:03.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:03.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:03.857 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:03.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:03.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:03.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:04.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:04.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:04.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:04.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:04.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:04.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:04.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:04.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[27010]: 5.4 deep-scrub starts 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[27010]: 5.4 deep-scrub ok 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[27010]: pgmap v135: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 905 op/s; 29449/313329 objects misplaced (9.399%); 27 MiB/s, 445 objects/s recovering 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[37492]: 5.4 deep-scrub starts 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[37492]: 5.4 deep-scrub ok 2024-04-25T05:37:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:04 smithi043 ceph-mon[37492]: pgmap v135: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 905 op/s; 29449/313329 objects misplaced (9.399%); 27 MiB/s, 445 objects/s recovering 2024-04-25T05:37:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:04.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:04 smithi060 ceph-mon[30898]: 5.4 deep-scrub starts 2024-04-25T05:37:04.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:04 smithi060 ceph-mon[30898]: 5.4 deep-scrub ok 2024-04-25T05:37:04.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:04 smithi060 ceph-mon[30898]: pgmap v135: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 905 op/s; 29449/313329 objects misplaced (9.399%); 27 MiB/s, 445 objects/s recovering 2024-04-25T05:37:04.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:04.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:04.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:04.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:05.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:05.053 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:05.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:05.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:05.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:05.313 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:05.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:05.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:05.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[27010]: osdmap e135: 8 total, 8 up, 7 in 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:05.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:05 smithi043 ceph-mon[37492]: osdmap e135: 8 total, 8 up, 7 in 2024-04-25T05:37:05.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:05.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:05.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:05 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:05.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:05 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:05.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:05 smithi060 ceph-mon[30898]: osdmap e135: 8 total, 8 up, 7 in 2024-04-25T05:37:05.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:05.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:05.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:05.951 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:05.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:06.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:06.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:06.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:06.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:06.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:06.364 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:06.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:06.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:37:06.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:06.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:06.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:06.614 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:06 smithi043 ceph-mon[27010]: pgmap v137: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 779 op/s; 29449/313329 objects misplaced (9.399%); 23 MiB/s, 370 objects/s recovering 2024-04-25T05:37:06.647 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:06 smithi043 ceph-mon[37492]: pgmap v137: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 779 op/s; 29449/313329 objects misplaced (9.399%); 23 MiB/s, 370 objects/s recovering 2024-04-25T05:37:06.656 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:06.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:06.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:06.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:06 smithi060 ceph-mon[30898]: pgmap v137: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 779 op/s; 29449/313329 objects misplaced (9.399%); 23 MiB/s, 370 objects/s recovering 2024-04-25T05:37:06.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:06.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:06.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:07.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:07.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:07.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:07.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:07.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:07.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:07.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:07 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:37:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:07 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:37:07.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:07 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:37:07.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:07 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:37:07.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:07.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:07.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:07 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:37:07.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:07 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:37:07.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:07.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:07.915 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:07.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:07.975 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:07.988 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:07.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:07.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:08.025 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:08.078 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:08.089 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:08.124 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:08.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:08.210 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:08.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:08.254 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:08.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:08.307 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:08.327 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:08.328 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:08.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:08.391 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:37:08.403 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:08.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:08.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:08.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[27010]: pgmap v138: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s; 28665/311205 objects misplaced (9.211%); 21 MiB/s, 328 objects/s recovering 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1530434722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[37492]: pgmap v138: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s; 28665/311205 objects misplaced (9.211%); 21 MiB/s, 328 objects/s recovering 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1530434722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:08.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:08.669 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:08.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:08 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:37:08.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:08 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:37:08.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:08 smithi060 ceph-mon[30898]: pgmap v138: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 582 op/s; 28665/311205 objects misplaced (9.211%); 21 MiB/s, 328 objects/s recovering 2024-04-25T05:37:08.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1530434722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:08.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:08.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:08.846 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 136, 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:37:08.367425+0000", 2024-04-25T05:37:08.847 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:36:53.196620+0000", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5801984, 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "noscrub", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:37:08.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 7, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 1, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 1, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 1, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 1, 2024-04-25T05:37:08.850 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 1, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "21", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.851 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.852 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.853 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 7.8899998664855957, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 7.8899998664855957, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.37999999523162842, 2024-04-25T05:37:08.854 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 3, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 3, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:37:08.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:37:08.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.857 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.858 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.859 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:37:08.860 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.861 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.863 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.863 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.863 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.863 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:37:08.866 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.866 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.866 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.867 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.868 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:37:08.869 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.870 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.871 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.872 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.873 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.874 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4900000095367432, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4900000095367432, 2024-04-25T05:37:08.875 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.876 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.877 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.878 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.4900000095367432, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.4900000095367432, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 7, 2024-04-25T05:37:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_0", 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:35:08.463631+0000", 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:37:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "91", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4900000095367432, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4900000095367432, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.901 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 96, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.902 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:08.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.905 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.906 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.909 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:37:08.910 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.912 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.913 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 129, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.914 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:37:08.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:08.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:37:08.918 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 88, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:37:08.919 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.920 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.921 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:37:08.922 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 132, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.923 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.925 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.926 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:37:04.225827+0000", 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:35.267959+0000", 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:59.003964+0000", 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.928 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:35:58.020438+0000", 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.929 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:33.086906+0000", 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:29.844812+0000", 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:43.196834+0000", 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:08.931 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:29.659289+0000", 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.2", 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.12", 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:08.933 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.1", 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.7", 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.b", 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.935 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.c", 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.d", 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.936 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:08.937 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:37:08.938 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:08.939 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:08.940 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:08.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:09.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:09.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:09.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:09.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:09.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:09.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: osdmap e136: 8 total, 8 up, 7 in 2024-04-25T05:37:09.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/754049031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:09.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:09.456 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:09.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:09.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:37:09.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:37:09.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:09.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:09.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: osdmap e136: 8 total, 8 up, 7 in 2024-04-25T05:37:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/754049031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:09.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:09.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:09.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:37:09.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:37:09.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:09.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/362083667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: osdmap e136: 8 total, 8 up, 7 in 2024-04-25T05:37:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/754049031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:09.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:09.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:09.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:09.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:09.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:10.084 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:10.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:10.123 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:10.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:10.251 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:10.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:10.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:10.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:10.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:10.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:10.629 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:10.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:10.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:10 smithi060 ceph-mon[30898]: pgmap v140: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 26175/302046 objects misplaced (8.666%); 35 MiB/s, 552 objects/s recovering 2024-04-25T05:37:10.756 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:10.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:10.880 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:10 smithi043 ceph-mon[27010]: pgmap v140: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 26175/302046 objects misplaced (8.666%); 35 MiB/s, 552 objects/s recovering 2024-04-25T05:37:10.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:10 smithi043 ceph-mon[37492]: pgmap v140: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 26175/302046 objects misplaced (8.666%); 35 MiB/s, 552 objects/s recovering 2024-04-25T05:37:10.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:10.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:10.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:11.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:11.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:11.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:11.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:11.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:11.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:11.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:11.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:11.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:11.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:11.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:11.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:11.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:11.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:11.969 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:11.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:12.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:12.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:12.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:12.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:12.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:12.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:12.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:12.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:12.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:37:12.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:12.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:12.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:12.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:12.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:13.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:13.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:12 smithi043 ceph-mon[27010]: pgmap v141: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 470 op/s; 26175/302046 objects misplaced (8.666%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:37:13.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:12 smithi043 ceph-mon[37492]: pgmap v141: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 470 op/s; 26175/302046 objects misplaced (8.666%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:37:13.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:13.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:37:13.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:13.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:13.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:12 smithi060 ceph-mon[30898]: pgmap v141: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 6.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 470 op/s; 26175/302046 objects misplaced (8.666%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:37:13.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:12 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:13.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.348 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:13.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:13.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.474 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:13.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:13.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:13.599 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:13.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:13.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:13.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:13.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:13.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:14.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:14.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:14.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:13 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:13 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:14.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:13 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:14.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:14.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:14.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:14.393 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:14.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:14.521 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:14.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:14.644 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:14.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:14.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:14.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:14.909 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:14.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:15.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:15.053 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:15.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:14 smithi043 ceph-mon[27010]: pgmap v142: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 837 op/s; 23170/289851 objects misplaced (7.994%); 28 MiB/s, 453 objects/s recovering 2024-04-25T05:37:15.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:14 smithi043 ceph-mon[37492]: pgmap v142: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 837 op/s; 23170/289851 objects misplaced (7.994%); 28 MiB/s, 453 objects/s recovering 2024-04-25T05:37:15.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:15.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:14 smithi060 ceph-mon[30898]: pgmap v142: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 837 op/s; 23170/289851 objects misplaced (7.994%); 28 MiB/s, 453 objects/s recovering 2024-04-25T05:37:15.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:15.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:15.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:15.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:15.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:15.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:15.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:15.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:15.825 INFO:tasks.thrashosds.thrasher:j is {'epoch': 136, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:37:08.367425+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:36:53.196620+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-25T05:35:08.463631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:37:04.225827+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:35.267959+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:59.003964+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:35:58.020438+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:33.086906+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:29.844812+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:36:43.196834+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:29.659289+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.1', 'osds': [4, 3, 6]}, {'pgid': '7.7', 'osds': [0, 3, 6]}, {'pgid': '7.b', 'osds': [1, 5, 6]}, {'pgid': '7.c', 'osds': [3, 7, 6]}, {'pgid': '7.d', 'osds': [1, 0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:37:15.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:37:15.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:15.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:15.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:16.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:16.089 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:16.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:16.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:16.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:16.417 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:16.418 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:16.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:16.527 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:37:16.527 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:37:16.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:16.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:16.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:16.748 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:16.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:16.875 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:16.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:16.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:17.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:17.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:17.146 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:16 smithi043 ceph-mon[27010]: pgmap v143: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 23170/289851 objects misplaced (7.994%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:37:17.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:16 smithi043 ceph-mon[37492]: pgmap v143: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 23170/289851 objects misplaced (7.994%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:37:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:16 smithi060 ceph-mon[30898]: pgmap v143: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 23170/289851 objects misplaced (7.994%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:37:17.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:17.276 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:17.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:17.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:17.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:17.557 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:17.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:17.666 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:17.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:17.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:17.843 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:17.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:17.967 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:17.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:18.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:18.091 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:18.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:18.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:18.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:18.092 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:18.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:18.152 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:18.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:18.197 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:18.218 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:18.246 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:18.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:18.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:18.291 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:18.320 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:18.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:18.360 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:18.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:18.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:18.447 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:18.454 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:18.455 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:18.455 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:18.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:18.614 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:18.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:18.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:18.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:37:18.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:18.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:18.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:18.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:19.033 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:37:19.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:19.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:19.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:19.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:19.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[27010]: pgmap v144: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 22416/287718 objects misplaced (7.791%); 27 MiB/s, 427 objects/s recovering 2024-04-25T05:37:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3126513783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2032336328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[37492]: pgmap v144: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 22416/287718 objects misplaced (7.791%); 27 MiB/s, 427 objects/s recovering 2024-04-25T05:37:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3126513783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2032336328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:19.432 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:19.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:19 smithi060 ceph-mon[30898]: pgmap v144: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 783 op/s; 22416/287718 objects misplaced (7.791%); 27 MiB/s, 427 objects/s recovering 2024-04-25T05:37:19.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3126513783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2032336328' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:19.511 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:37:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:37:19.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:19.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:19.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:19.696 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:19.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:19.824 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:19.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:19.949 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:19.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:20.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:20.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:20.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:20.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:20.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:20.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[27010]: osdmap e137: 8 total, 8 up, 7 in 2024-04-25T05:37:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:20 smithi043 ceph-mon[37492]: osdmap e137: 8 total, 8 up, 7 in 2024-04-25T05:37:20.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:20.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:20 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:20 smithi060 ceph-mon[30898]: osdmap e137: 8 total, 8 up, 7 in 2024-04-25T05:37:20.523 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-04-25T05:37:19.018341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6075193907,"num_objects":92922,"num_object_clones":0,"num_object_copies":278766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20058,"num_objects_unfound":0,"num_objects_dirty":92922,"num_whiteouts":0,"num_read":1413,"num_read_kb":1227,"num_write":141849,"num_write_kb":7481886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9846,"num_bytes_recovered":640946958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76123,"ondisk_log_size":76123,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17265924,"kb_used_data":16597900,"kb_used_omap":10,"kb_used_meta":667957,"kb_avail":638950140,"statfs":{"total":671965249536,"available":654284943360,"internally_reserved":0,"allocated":16996249600,"data_stored":16993355012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":683988074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-508559360,"num_objects":-7760,"num_object_clones":0,"num_object_copies":-23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6117,"num_objects_unfound":0,"num_objects_dirty":-7760,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4294,"num_bytes_recovered":281411584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.635720"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":137,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397205+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:37:08.397205+0000","last_peered":"2024-04-25T05:37:08.397205+0000","last_clean":"2024-04-25T05:37:08.397205+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:37:08.397205+0000","last_undegraded":"2024-04-25T05:37:08.397205+0000","last_fullsized":"2024-04-25T05:37:08.397205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399358+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:37:08.399358+0000","last_peered":"2024-04-25T05:37:08.399358+0000","last_clean":"2024-04-25T05:37:08.399358+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:37:08.399358+0000","last_undegraded":"2024-04-25T05:37:08.399358+0000","last_fullsized":"2024-04-25T05:37:08.399358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406056+0000","last_change":"2024-04-25T05:36:18.176520+0000","last_active":"2024-04-25T05:37:08.406056+0000","last_peered":"2024-04-25T05:37:08.406056+0000","last_clean":"2024-04-25T05:37:08.406056+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:37:08.406056+0000","last_undegraded":"2024-04-25T05:37:08.406056+0000","last_fullsized":"2024-04-25T05:37:08.406056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.176394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.176394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:23.464747+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":199,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399617+0000","last_change":"2024-04-25T05:36:54.265352+0000","last_active":"2024-04-25T05:37:08.399617+0000","last_peered":"2024-04-25T05:37:08.399617+0000","last_clean":"2024-04-25T05:37:08.399617+0000","last_became_active":"2024-04-25T05:36:54.243668+0000","last_became_peered":"2024-04-25T05:36:54.243668+0000","last_unstale":"2024-04-25T05:37:08.399617+0000","last_undegraded":"2024-04-25T05:37:08.399617+0000","last_fullsized":"2024-04-25T05:37:08.399617+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:29.288783+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406585+0000","last_change":"2024-04-25T05:36:54.236185+0000","last_active":"2024-04-25T05:37:08.406585+0000","last_peered":"2024-04-25T05:37:08.406585+0000","last_clean":"2024-04-25T05:37:08.406585+0000","last_became_active":"2024-04-25T05:36:54.235695+0000","last_became_peered":"2024-04-25T05:36:54.235695+0000","last_unstale":"2024-04-25T05:37:08.406585+0000","last_undegraded":"2024-04-25T05:37:08.406585+0000","last_fullsized":"2024-04-25T05:37:08.406585+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":147,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399825+0000","last_change":"2024-04-25T05:36:54.230604+0000","last_active":"2024-04-25T05:37:08.399825+0000","last_peered":"2024-04-25T05:37:08.399825+0000","last_clean":"2024-04-25T05:37:08.399825+0000","last_became_active":"2024-04-25T05:36:54.229992+0000","last_became_peered":"2024-04-25T05:36:54.229992+0000","last_unstale":"2024-04-25T05:37:08.399825+0000","last_undegraded":"2024-04-25T05:37:08.399825+0000","last_fullsized":"2024-04-25T05:37:08.399825+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.411656+0000","last_change":"2024-04-25T05:36:54.230655+0000","last_active":"2024-04-25T05:37:08.411656+0000","last_peered":"2024-04-25T05:37:08.411656+0000","last_clean":"2024-04-25T05:37:08.411656+0000","last_became_active":"2024-04-25T05:36:54.230169+0000","last_became_peered":"2024-04-25T05:36:54.230169+0000","last_unstale":"2024-04-25T05:37:08.411656+0000","last_undegraded":"2024-04-25T05:37:08.411656+0000","last_fullsized":"2024-04-25T05:37:08.411656+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":219,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.404443+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:37:08.404443+0000","last_peered":"2024-04-25T05:37:08.404443+0000","last_clean":"2024-04-25T05:37:08.404443+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:37:08.404443+0000","last_undegraded":"2024-04-25T05:37:08.404443+0000","last_fullsized":"2024-04-25T05:37:08.404443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403896+0000","last_change":"2024-04-25T05:35:32.284750+0000","last_active":"2024-04-25T05:37:08.403896+0000","last_peered":"2024-04-25T05:37:08.403896+0000","last_clean":"2024-04-25T05:37:08.403896+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:37:08.403896+0000","last_undegraded":"2024-04-25T05:37:08.403896+0000","last_fullsized":"2024-04-25T05:37:08.403896+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.284637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.284637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382254+0000","last_change":"2024-04-25T05:36:29.981966+0000","last_active":"2024-04-25T05:37:08.382254+0000","last_peered":"2024-04-25T05:37:08.382254+0000","last_clean":"2024-04-25T05:37:08.382254+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:37:08.382254+0000","last_undegraded":"2024-04-25T05:37:08.382254+0000","last_fullsized":"2024-04-25T05:37:08.382254+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.093687+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":172,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382550+0000","last_change":"2024-04-25T05:36:54.244217+0000","last_active":"2024-04-25T05:37:08.382550+0000","last_peered":"2024-04-25T05:37:08.382550+0000","last_clean":"2024-04-25T05:37:08.382550+0000","last_became_active":"2024-04-25T05:36:54.243924+0000","last_became_peered":"2024-04-25T05:36:54.243924+0000","last_unstale":"2024-04-25T05:37:08.382550+0000","last_undegraded":"2024-04-25T05:37:08.382550+0000","last_fullsized":"2024-04-25T05:37:08.382550+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.105851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":194,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397489+0000","last_change":"2024-04-25T05:36:49.748038+0000","last_active":"2024-04-25T05:37:08.397489+0000","last_peered":"2024-04-25T05:37:08.397489+0000","last_clean":"2024-04-25T05:37:08.397489+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:37:08.397489+0000","last_undegraded":"2024-04-25T05:37:08.397489+0000","last_fullsized":"2024-04-25T05:37:08.397489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:11.511083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.381931+0000","last_change":"2024-04-25T05:35:32.732758+0000","last_active":"2024-04-25T05:37:08.381931+0000","last_peered":"2024-04-25T05:37:08.381931+0000","last_clean":"2024-04-25T05:37:08.381931+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:37:08.381931+0000","last_undegraded":"2024-04-25T05:37:08.381931+0000","last_fullsized":"2024-04-25T05:37:08.381931+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.732662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.732662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397148+0000","last_change":"2024-04-25T05:35:44.725188+0000","last_active":"2024-04-25T05:37:08.397148+0000","last_peered":"2024-04-25T05:37:08.397148+0000","last_clean":"2024-04-25T05:37:08.397148+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:37:08.397148+0000","last_undegraded":"2024-04-25T05:37:08.397148+0000","last_fullsized":"2024-04-25T05:37:08.397148+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.725019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.725019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382427+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:37:08.382427+0000","last_peered":"2024-04-25T05:37:08.382427+0000","last_clean":"2024-04-25T05:37:08.382427+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:37:08.382427+0000","last_undegraded":"2024-04-25T05:37:08.382427+0000","last_fullsized":"2024-04-25T05:37:08.382427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.095746+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":189,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398824+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:37:08.398824+0000","last_peered":"2024-04-25T05:37:08.398824+0000","last_clean":"2024-04-25T05:37:08.398824+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:37:08.398824+0000","last_undegraded":"2024-04-25T05:37:08.398824+0000","last_fullsized":"2024-04-25T05:37:08.398824+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.256093+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399430+0000","last_change":"2024-04-25T05:36:54.245109+0000","last_active":"2024-04-25T05:37:08.399430+0000","last_peered":"2024-04-25T05:37:08.399430+0000","last_clean":"2024-04-25T05:37:08.399430+0000","last_became_active":"2024-04-25T05:36:54.244292+0000","last_became_peered":"2024-04-25T05:36:54.244292+0000","last_unstale":"2024-04-25T05:37:08.399430+0000","last_undegraded":"2024-04-25T05:37:08.399430+0000","last_fullsized":"2024-04-25T05:37:08.399430+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:19.691117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405665+0000","last_change":"2024-04-25T05:36:52.722150+0000","last_active":"2024-04-25T05:37:08.405665+0000","last_peered":"2024-04-25T05:37:08.405665+0000","last_clean":"2024-04-25T05:37:08.405665+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:37:08.405665+0000","last_undegraded":"2024-04-25T05:37:08.405665+0000","last_fullsized":"2024-04-25T05:37:08.405665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.475127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406361+0000","last_change":"2024-04-25T05:36:55.198379+0000","last_active":"2024-04-25T05:37:08.406361+0000","last_peered":"2024-04-25T05:37:08.406361+0000","last_clean":"2024-04-25T05:37:08.406361+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:37:08.406361+0000","last_undegraded":"2024-04-25T05:37:08.406361+0000","last_fullsized":"2024-04-25T05:37:08.406361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:11.778316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.393078+0000","last_change":"2024-04-25T05:35:28.590848+0000","last_active":"2024-04-25T05:37:08.393078+0000","last_peered":"2024-04-25T05:37:08.393078+0000","last_clean":"2024-04-25T05:37:08.393078+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:37:08.393078+0000","last_undegraded":"2024-04-25T05:37:08.393078+0000","last_fullsized":"2024-04-25T05:37:08.393078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:28.590759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.590759+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.410925+0000","last_change":"2024-04-25T05:36:54.237606+0000","last_active":"2024-04-25T05:37:08.410925+0000","last_peered":"2024-04-25T05:37:08.410925+0000","last_clean":"2024-04-25T05:37:08.410925+0000","last_became_active":"2024-04-25T05:36:54.237282+0000","last_became_peered":"2024-04-25T05:36:54.237282+0000","last_unstale":"2024-04-25T05:37:08.410925+0000","last_undegraded":"2024-04-25T05:37:08.410925+0000","last_fullsized":"2024-04-25T05:37:08.410925+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":141,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399060+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:37:08.399060+0000","last_peered":"2024-04-25T05:37:08.399060+0000","last_clean":"2024-04-25T05:37:08.399060+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:37:08.399060+0000","last_undegraded":"2024-04-25T05:37:08.399060+0000","last_fullsized":"2024-04-25T05:37:08.399060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397587+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:37:08.397587+0000","last_peered":"2024-04-25T05:37:08.397587+0000","last_clean":"2024-04-25T05:37:08.397587+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:37:08.397587+0000","last_undegraded":"2024-04-25T05:37:08.397587+0000","last_fullsized":"2024-04-25T05:37:08.397587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.515357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.411021+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:37:08.411021+0000","last_peered":"2024-04-25T05:37:08.411021+0000","last_clean":"2024-04-25T05:37:08.411021+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:37:08.411021+0000","last_undegraded":"2024-04-25T05:37:08.411021+0000","last_fullsized":"2024-04-25T05:37:08.411021+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398347+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:37:08.398347+0000","last_peered":"2024-04-25T05:37:08.398347+0000","last_clean":"2024-04-25T05:37:08.398347+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:37:08.398347+0000","last_undegraded":"2024-04-25T05:37:08.398347+0000","last_fullsized":"2024-04-25T05:37:08.398347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.648018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"130'9","reported_seq":61,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768386+0000","last_change":"2024-04-25T05:36:54.240381+0000","last_active":"2024-04-25T05:37:08.768386+0000","last_peered":"2024-04-25T05:37:08.768386+0000","last_clean":"2024-04-25T05:37:08.768386+0000","last_became_active":"2024-04-25T05:36:54.232031+0000","last_became_peered":"2024-04-25T05:36:54.232031+0000","last_unstale":"2024-04-25T05:37:08.768386+0000","last_undegraded":"2024-04-25T05:37:08.768386+0000","last_fullsized":"2024-04-25T05:37:08.768386+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:26.935418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399244+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:37:08.399244+0000","last_peered":"2024-04-25T05:37:08.399244+0000","last_clean":"2024-04-25T05:37:08.399244+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:37:08.399244+0000","last_undegraded":"2024-04-25T05:37:08.399244+0000","last_fullsized":"2024-04-25T05:37:08.399244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:10.328531+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405365+0000","last_change":"2024-04-25T05:36:15.129016+0000","last_active":"2024-04-25T05:37:08.405365+0000","last_peered":"2024-04-25T05:37:08.405365+0000","last_clean":"2024-04-25T05:37:08.405365+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:37:08.405365+0000","last_undegraded":"2024-04-25T05:37:08.405365+0000","last_fullsized":"2024-04-25T05:37:08.405365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:36:15.128887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.128887+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:26.080685+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405122+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:37:08.405122+0000","last_peered":"2024-04-25T05:37:08.405122+0000","last_clean":"2024-04-25T05:37:08.405122+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:37:08.405122+0000","last_undegraded":"2024-04-25T05:37:08.405122+0000","last_fullsized":"2024-04-25T05:37:08.405122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405754+0000","last_change":"2024-04-25T05:36:14.171466+0000","last_active":"2024-04-25T05:37:08.405754+0000","last_peered":"2024-04-25T05:37:08.405754+0000","last_clean":"2024-04-25T05:37:08.405754+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:37:08.405754+0000","last_undegraded":"2024-04-25T05:37:08.405754+0000","last_fullsized":"2024-04-25T05:37:08.405754+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.171349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.171349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:24.814418+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403331+0000","last_change":"2024-04-25T05:35:36.324777+0000","last_active":"2024-04-25T05:37:08.403331+0000","last_peered":"2024-04-25T05:37:08.403331+0000","last_clean":"2024-04-25T05:37:08.403331+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:37:08.403331+0000","last_undegraded":"2024-04-25T05:37:08.403331+0000","last_fullsized":"2024-04-25T05:37:08.403331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":88,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382862+0000","last_change":"2024-04-25T05:36:54.261299+0000","last_active":"2024-04-25T05:37:08.382862+0000","last_peered":"2024-04-25T05:37:08.382862+0000","last_clean":"2024-04-25T05:37:08.382862+0000","last_became_active":"2024-04-25T05:36:54.237414+0000","last_became_peered":"2024-04-25T05:36:54.237414+0000","last_unstale":"2024-04-25T05:37:08.382862+0000","last_undegraded":"2024-04-25T05:37:08.382862+0000","last_fullsized":"2024-04-25T05:37:08.382862+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:25.674276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:25.674276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":147,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397020+0000","last_change":"2024-04-25T05:36:22.428077+0000","last_active":"2024-04-25T05:37:08.397020+0000","last_peered":"2024-04-25T05:37:08.397020+0000","last_clean":"2024-04-25T05:37:08.397020+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:37:08.397020+0000","last_undegraded":"2024-04-25T05:37:08.397020+0000","last_fullsized":"2024-04-25T05:37:08.397020+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:22.427930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.427930+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:25.727609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397457+0000","last_change":"2024-04-25T05:36:54.248389+0000","last_active":"2024-04-25T05:37:08.397457+0000","last_peered":"2024-04-25T05:37:08.397457+0000","last_clean":"2024-04-25T05:37:08.397457+0000","last_became_active":"2024-04-25T05:36:54.247892+0000","last_became_peered":"2024-04-25T05:36:54.247892+0000","last_unstale":"2024-04-25T05:37:08.397457+0000","last_undegraded":"2024-04-25T05:37:08.397457+0000","last_fullsized":"2024-04-25T05:37:08.397457+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.408783+0000","last_change":"2024-04-25T05:36:21.217686+0000","last_active":"2024-04-25T05:37:08.408783+0000","last_peered":"2024-04-25T05:37:08.408783+0000","last_clean":"2024-04-25T05:37:08.408783+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:37:08.408783+0000","last_undegraded":"2024-04-25T05:37:08.408783+0000","last_fullsized":"2024-04-25T05:37:08.408783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.483356+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":172,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403434+0000","last_change":"2024-04-25T05:36:54.262135+0000","last_active":"2024-04-25T05:37:08.403434+0000","last_peered":"2024-04-25T05:37:08.403434+0000","last_clean":"2024-04-25T05:37:08.403434+0000","last_became_active":"2024-04-25T05:36:54.244002+0000","last_became_peered":"2024-04-25T05:36:54.244002+0000","last_unstale":"2024-04-25T05:37:08.403434+0000","last_undegraded":"2024-04-25T05:37:08.403434+0000","last_fullsized":"2024-04-25T05:37:08.403434+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:35.722656+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406847+0000","last_change":"2024-04-25T05:36:54.236715+0000","last_active":"2024-04-25T05:37:08.406847+0000","last_peered":"2024-04-25T05:37:08.406847+0000","last_clean":"2024-04-25T05:37:08.406847+0000","last_became_active":"2024-04-25T05:36:54.235240+0000","last_became_peered":"2024-04-25T05:36:54.235240+0000","last_unstale":"2024-04-25T05:37:08.406847+0000","last_undegraded":"2024-04-25T05:37:08.406847+0000","last_fullsized":"2024-04-25T05:37:08.406847+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397818+0000","last_change":"2024-04-25T05:37:00.575011+0000","last_active":"2024-04-25T05:37:08.397818+0000","last_peered":"2024-04-25T05:37:08.397818+0000","last_clean":"2024-04-25T05:37:08.397818+0000","last_became_active":"2024-04-25T05:36:54.239270+0000","last_became_peered":"2024-04-25T05:36:54.239270+0000","last_unstale":"2024-04-25T05:37:08.397818+0000","last_undegraded":"2024-04-25T05:37:08.397818+0000","last_fullsized":"2024-04-25T05:37:08.397818+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:24.324484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382982+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:37:08.382982+0000","last_peered":"2024-04-25T05:37:08.382982+0000","last_clean":"2024-04-25T05:37:08.382982+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:37:08.382982+0000","last_undegraded":"2024-04-25T05:37:08.382982+0000","last_fullsized":"2024-04-25T05:37:08.382982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:38.198166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.404347+0000","last_change":"2024-04-25T05:36:54.252887+0000","last_active":"2024-04-25T05:37:08.404347+0000","last_peered":"2024-04-25T05:37:08.404347+0000","last_clean":"2024-04-25T05:37:08.404347+0000","last_became_active":"2024-04-25T05:36:54.232522+0000","last_became_peered":"2024-04-25T05:36:54.232522+0000","last_unstale":"2024-04-25T05:37:08.404347+0000","last_undegraded":"2024-04-25T05:37:08.404347+0000","last_fullsized":"2024-04-25T05:37:08.404347+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403592+0000","last_change":"2024-04-25T05:35:34.276800+0000","last_active":"2024-04-25T05:37:08.403592+0000","last_peered":"2024-04-25T05:37:08.403592+0000","last_clean":"2024-04-25T05:37:08.403592+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:37:08.403592+0000","last_undegraded":"2024-04-25T05:37:08.403592+0000","last_fullsized":"2024-04-25T05:37:08.403592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.276673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.276673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406489+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:37:08.406489+0000","last_peered":"2024-04-25T05:37:08.406489+0000","last_clean":"2024-04-25T05:37:08.406489+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:37:08.406489+0000","last_undegraded":"2024-04-25T05:37:08.406489+0000","last_fullsized":"2024-04-25T05:37:08.406489+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398661+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:37:08.398661+0000","last_peered":"2024-04-25T05:37:08.398661+0000","last_clean":"2024-04-25T05:37:08.398661+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:37:08.398661+0000","last_undegraded":"2024-04-25T05:37:08.398661+0000","last_fullsized":"2024-04-25T05:37:08.398661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:13.840716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":123,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382357+0000","last_change":"2024-04-25T05:36:54.252111+0000","last_active":"2024-04-25T05:37:08.382357+0000","last_peered":"2024-04-25T05:37:08.382357+0000","last_clean":"2024-04-25T05:37:08.382357+0000","last_became_active":"2024-04-25T05:36:54.238351+0000","last_became_peered":"2024-04-25T05:36:54.238351+0000","last_unstale":"2024-04-25T05:37:08.382357+0000","last_undegraded":"2024-04-25T05:37:08.382357+0000","last_fullsized":"2024-04-25T05:37:08.382357+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:35:33.708629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:33.708629+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.404036+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:37:08.404036+0000","last_peered":"2024-04-25T05:37:08.404036+0000","last_clean":"2024-04-25T05:37:08.404036+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:37:08.404036+0000","last_undegraded":"2024-04-25T05:37:08.404036+0000","last_fullsized":"2024-04-25T05:37:08.404036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.400222+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:37:08.400222+0000","last_peered":"2024-04-25T05:37:08.400222+0000","last_clean":"2024-04-25T05:37:08.400222+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:37:08.400222+0000","last_undegraded":"2024-04-25T05:37:08.400222+0000","last_fullsized":"2024-04-25T05:37:08.400222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":173,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398784+0000","last_change":"2024-04-25T05:36:54.240953+0000","last_active":"2024-04-25T05:37:08.398784+0000","last_peered":"2024-04-25T05:37:08.398784+0000","last_clean":"2024-04-25T05:37:08.398784+0000","last_became_active":"2024-04-25T05:36:54.235617+0000","last_became_peered":"2024-04-25T05:36:54.235617+0000","last_unstale":"2024-04-25T05:37:08.398784+0000","last_undegraded":"2024-04-25T05:37:08.398784+0000","last_fullsized":"2024-04-25T05:37:08.398784+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399022+0000","last_change":"2024-04-25T05:36:54.258480+0000","last_active":"2024-04-25T05:37:08.399022+0000","last_peered":"2024-04-25T05:37:08.399022+0000","last_clean":"2024-04-25T05:37:08.399022+0000","last_became_active":"2024-04-25T05:36:54.236292+0000","last_became_peered":"2024-04-25T05:36:54.236292+0000","last_unstale":"2024-04-25T05:37:08.399022+0000","last_undegraded":"2024-04-25T05:37:08.399022+0000","last_fullsized":"2024-04-25T05:37:08.399022+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405003+0000","last_change":"2024-04-25T05:36:30.109005+0000","last_active":"2024-04-25T05:37:08.405003+0000","last_peered":"2024-04-25T05:37:08.405003+0000","last_clean":"2024-04-25T05:37:08.405003+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:37:08.405003+0000","last_undegraded":"2024-04-25T05:37:08.405003+0000","last_fullsized":"2024-04-25T05:37:08.405003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.212177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":198,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768392+0000","last_change":"2024-04-25T05:35:39.823582+0000","last_active":"2024-04-25T05:37:08.768392+0000","last_peered":"2024-04-25T05:37:08.768392+0000","last_clean":"2024-04-25T05:37:08.768392+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:37:08.768392+0000","last_undegraded":"2024-04-25T05:37:08.768392+0000","last_fullsized":"2024-04-25T05:37:08.768392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:39.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.823449+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:16.116461+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397760+0000","last_change":"2024-04-25T05:36:15.504640+0000","last_active":"2024-04-25T05:37:08.397760+0000","last_peered":"2024-04-25T05:37:08.397760+0000","last_clean":"2024-04-25T05:37:08.397760+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:37:08.397760+0000","last_undegraded":"2024-04-25T05:37:08.397760+0000","last_fullsized":"2024-04-25T05:37:08.397760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:15.504395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.504395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:20.668339+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":193,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.414367+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:37:08.414367+0000","last_peered":"2024-04-25T05:37:08.414367+0000","last_clean":"2024-04-25T05:37:08.414367+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:37:08.414367+0000","last_undegraded":"2024-04-25T05:37:08.414367+0000","last_fullsized":"2024-04-25T05:37:08.414367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.719798+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.400013+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:37:08.400013+0000","last_peered":"2024-04-25T05:37:08.400013+0000","last_clean":"2024-04-25T05:37:08.400013+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:37:08.400013+0000","last_undegraded":"2024-04-25T05:37:08.400013+0000","last_fullsized":"2024-04-25T05:37:08.400013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.179078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.798902+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:37:08.798902+0000","last_peered":"2024-04-25T05:37:08.798902+0000","last_clean":"2024-04-25T05:37:08.798902+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:37:08.798902+0000","last_undegraded":"2024-04-25T05:37:08.798902+0000","last_fullsized":"2024-04-25T05:37:08.798902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403421+0000","last_change":"2024-04-25T05:36:31.118006+0000","last_active":"2024-04-25T05:37:08.403421+0000","last_peered":"2024-04-25T05:37:08.403421+0000","last_clean":"2024-04-25T05:37:08.403421+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:37:08.403421+0000","last_undegraded":"2024-04-25T05:37:08.403421+0000","last_fullsized":"2024-04-25T05:37:08.403421+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.339192+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":187,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382765+0000","last_change":"2024-04-25T05:35:40.845196+0000","last_active":"2024-04-25T05:37:08.382765+0000","last_peered":"2024-04-25T05:37:08.382765+0000","last_clean":"2024-04-25T05:37:08.382765+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:37:08.382765+0000","last_undegraded":"2024-04-25T05:37:08.382765+0000","last_fullsized":"2024-04-25T05:37:08.382765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.845036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:08.541350+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.395104+0000","last_change":"2024-04-25T05:35:30.673162+0000","last_active":"2024-04-25T05:37:08.395104+0000","last_peered":"2024-04-25T05:37:08.395104+0000","last_clean":"2024-04-25T05:37:08.395104+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:37:08.395104+0000","last_undegraded":"2024-04-25T05:37:08.395104+0000","last_fullsized":"2024-04-25T05:37:08.395104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:30.673041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.673041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.407605+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:37:08.407605+0000","last_peered":"2024-04-25T05:37:08.407605+0000","last_clean":"2024-04-25T05:37:08.407605+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:37:08.407605+0000","last_undegraded":"2024-04-25T05:37:08.407605+0000","last_fullsized":"2024-04-25T05:37:08.407605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":172,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397415+0000","last_change":"2024-04-25T05:37:01.599400+0000","last_active":"2024-04-25T05:37:08.397415+0000","last_peered":"2024-04-25T05:37:08.397415+0000","last_clean":"2024-04-25T05:37:08.397415+0000","last_became_active":"2024-04-25T05:36:54.240109+0000","last_became_peered":"2024-04-25T05:36:54.240109+0000","last_unstale":"2024-04-25T05:37:08.397415+0000","last_undegraded":"2024-04-25T05:37:08.397415+0000","last_fullsized":"2024-04-25T05:37:08.397415+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:02.962125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":194,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405582+0000","last_change":"2024-04-25T05:36:20.155217+0000","last_active":"2024-04-25T05:37:08.405582+0000","last_peered":"2024-04-25T05:37:08.405582+0000","last_clean":"2024-04-25T05:37:08.405582+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:37:08.405582+0000","last_undegraded":"2024-04-25T05:37:08.405582+0000","last_fullsized":"2024-04-25T05:37:08.405582+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:40.449029+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":152,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398426+0000","last_change":"2024-04-25T05:36:54.232885+0000","last_active":"2024-04-25T05:37:08.398426+0000","last_peered":"2024-04-25T05:37:08.398426+0000","last_clean":"2024-04-25T05:37:08.398426+0000","last_became_active":"2024-04-25T05:36:54.232503+0000","last_became_peered":"2024-04-25T05:36:54.232503+0000","last_unstale":"2024-04-25T05:37:08.398426+0000","last_undegraded":"2024-04-25T05:37:08.398426+0000","last_fullsized":"2024-04-25T05:37:08.398426+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.481007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.481007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:30.519849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.383066+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:37:08.383066+0000","last_peered":"2024-04-25T05:37:08.383066+0000","last_clean":"2024-04-25T05:37:08.383066+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:37:08.383066+0000","last_undegraded":"2024-04-25T05:37:08.383066+0000","last_fullsized":"2024-04-25T05:37:08.383066+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.303715+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.404100+0000","last_change":"2024-04-25T05:36:54.239000+0000","last_active":"2024-04-25T05:37:08.404100+0000","last_peered":"2024-04-25T05:37:08.404100+0000","last_clean":"2024-04-25T05:37:08.404100+0000","last_became_active":"2024-04-25T05:36:54.238366+0000","last_became_peered":"2024-04-25T05:36:54.238366+0000","last_unstale":"2024-04-25T05:37:08.404100+0000","last_undegraded":"2024-04-25T05:37:08.404100+0000","last_fullsized":"2024-04-25T05:37:08.404100+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.715360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.715360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":175,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.411135+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:37:08.411135+0000","last_peered":"2024-04-25T05:37:08.411135+0000","last_clean":"2024-04-25T05:37:08.411135+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:37:08.411135+0000","last_undegraded":"2024-04-25T05:37:08.411135+0000","last_fullsized":"2024-04-25T05:37:08.411135+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"136'8733","reported_seq":12891,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:17.868363+0000","last_change":"2024-04-25T05:36:55.339375+0000","last_active":"2024-04-25T05:37:17.868363+0000","last_peered":"2024-04-25T05:37:17.868363+0000","last_clean":"2024-04-25T05:36:53.210632+0000","last_became_active":"2024-04-25T05:36:55.308452+0000","last_became_peered":"2024-04-25T05:36:55.308452+0000","last_unstale":"2024-04-25T05:37:17.868363+0000","last_undegraded":"2024-04-25T05:37:17.868363+0000","last_fullsized":"2024-04-25T05:37:17.868363+0000","mapping_epoch":130,"log_start":"108'4544","ondisk_log_start":"108'4544","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'5609","last_scrub_stamp":"2024-04-25T05:36:18.461345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.461345+0000","objects_scrubbed":5551,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371785728,"num_objects":5673,"num_object_clones":0,"num_object_copies":17019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3839,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8733,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1991,"num_bytes_recovered":130482176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":5673}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":160,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382719+0000","last_change":"2024-04-25T05:36:55.047093+0000","last_active":"2024-04-25T05:37:08.382719+0000","last_peered":"2024-04-25T05:37:08.382719+0000","last_clean":"2024-04-25T05:37:08.382719+0000","last_became_active":"2024-04-25T05:36:54.237865+0000","last_became_peered":"2024-04-25T05:36:54.237865+0000","last_unstale":"2024-04-25T05:37:08.382719+0000","last_undegraded":"2024-04-25T05:37:08.382719+0000","last_fullsized":"2024-04-25T05:37:08.382719+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:21.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398477+0000","last_change":"2024-04-25T05:36:14.472050+0000","last_active":"2024-04-25T05:37:08.398477+0000","last_peered":"2024-04-25T05:37:08.398477+0000","last_clean":"2024-04-25T05:37:08.398477+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:37:08.398477+0000","last_undegraded":"2024-04-25T05:37:08.398477+0000","last_fullsized":"2024-04-25T05:37:08.398477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:36:14.471895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.471895+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:17.098974+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":229,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.381987+0000","last_change":"2024-04-25T05:36:54.267732+0000","last_active":"2024-04-25T05:37:08.381987+0000","last_peered":"2024-04-25T05:37:08.381987+0000","last_clean":"2024-04-25T05:37:08.381987+0000","last_became_active":"2024-04-25T05:36:54.244247+0000","last_became_peered":"2024-04-25T05:36:54.244247+0000","last_unstale":"2024-04-25T05:37:08.381987+0000","last_undegraded":"2024-04-25T05:37:08.381987+0000","last_fullsized":"2024-04-25T05:37:08.381987+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:45.257419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"130'10","reported_seq":217,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768401+0000","last_change":"2024-04-25T05:36:54.243037+0000","last_active":"2024-04-25T05:37:08.768401+0000","last_peered":"2024-04-25T05:37:08.768401+0000","last_clean":"2024-04-25T05:37:08.768401+0000","last_became_active":"2024-04-25T05:36:54.235542+0000","last_became_peered":"2024-04-25T05:36:54.235542+0000","last_unstale":"2024-04-25T05:37:08.768401+0000","last_undegraded":"2024-04-25T05:37:08.768401+0000","last_fullsized":"2024-04-25T05:37:08.768401+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.322822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"136'8885","reported_seq":8991,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.885595+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:37:17.885595+0000","last_peered":"2024-04-25T05:37:17.885595+0000","last_clean":"2024-04-25T05:37:17.885595+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:37:17.885595+0000","last_undegraded":"2024-04-25T05:37:17.885595+0000","last_fullsized":"2024-04-25T05:37:17.885595+0000","mapping_epoch":88,"log_start":"108'4716","ondisk_log_start":"108'4716","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":17241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":468224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406675+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:37:08.406675+0000","last_peered":"2024-04-25T05:37:08.406675+0000","last_clean":"2024-04-25T05:37:08.406675+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:37:08.406675+0000","last_undegraded":"2024-04-25T05:37:08.406675+0000","last_fullsized":"2024-04-25T05:37:08.406675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398339+0000","last_change":"2024-04-25T05:36:48.522033+0000","last_active":"2024-04-25T05:37:08.398339+0000","last_peered":"2024-04-25T05:37:08.398339+0000","last_clean":"2024-04-25T05:37:08.398339+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:37:08.398339+0000","last_undegraded":"2024-04-25T05:37:08.398339+0000","last_fullsized":"2024-04-25T05:37:08.398339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:48.521908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:48.521908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:53.728133+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398009+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:37:08.398009+0000","last_peered":"2024-04-25T05:37:08.398009+0000","last_clean":"2024-04-25T05:37:08.398009+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:37:08.398009+0000","last_undegraded":"2024-04-25T05:37:08.398009+0000","last_fullsized":"2024-04-25T05:37:08.398009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.389045+0000","last_change":"2024-04-25T05:36:47.489355+0000","last_active":"2024-04-25T05:37:08.389045+0000","last_peered":"2024-04-25T05:37:08.389045+0000","last_clean":"2024-04-25T05:37:08.389045+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:37:08.389045+0000","last_undegraded":"2024-04-25T05:37:08.389045+0000","last_fullsized":"2024-04-25T05:37:08.389045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:47.489244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.489244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.358175+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"136'8736","reported_seq":8919,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.942866+0000","last_change":"2024-04-25T05:36:54.004498+0000","last_active":"2024-04-25T05:37:17.942866+0000","last_peered":"2024-04-25T05:37:17.942866+0000","last_clean":"2024-04-25T05:37:17.942866+0000","last_became_active":"2024-04-25T05:35:09.681316+0000","last_became_peered":"2024-04-25T05:35:09.681316+0000","last_unstale":"2024-04-25T05:37:17.942866+0000","last_undegraded":"2024-04-25T05:37:17.942866+0000","last_fullsized":"2024-04-25T05:37:17.942866+0000","mapping_epoch":88,"log_start":"104'3876","ondisk_log_start":"104'3876","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7637","last_scrub_stamp":"2024-04-25T05:36:54.004340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.004340+0000","objects_scrubbed":6885,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.098402+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379191296,"num_objects":5786,"num_object_clones":0,"num_object_copies":17358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8736,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.396952+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:37:08.396952+0000","last_peered":"2024-04-25T05:37:08.396952+0000","last_clean":"2024-04-25T05:37:08.396952+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:37:08.396952+0000","last_undegraded":"2024-04-25T05:37:08.396952+0000","last_fullsized":"2024-04-25T05:37:08.396952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.897023+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397385+0000","last_change":"2024-04-25T05:36:51.574009+0000","last_active":"2024-04-25T05:37:08.397385+0000","last_peered":"2024-04-25T05:37:08.397385+0000","last_clean":"2024-04-25T05:37:08.397385+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:37:08.397385+0000","last_undegraded":"2024-04-25T05:37:08.397385+0000","last_fullsized":"2024-04-25T05:37:08.397385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:51.573898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.573898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.269140+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398825+0000","last_change":"2024-04-25T05:36:54.240931+0000","last_active":"2024-04-25T05:37:08.398825+0000","last_peered":"2024-04-25T05:37:08.398825+0000","last_clean":"2024-04-25T05:37:08.398825+0000","last_became_active":"2024-04-25T05:36:54.232602+0000","last_became_peered":"2024-04-25T05:36:54.232602+0000","last_unstale":"2024-04-25T05:37:08.398825+0000","last_undegraded":"2024-04-25T05:37:08.398825+0000","last_fullsized":"2024-04-25T05:37:08.398825+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.130398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":108,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399191+0000","last_change":"2024-04-25T05:36:54.260249+0000","last_active":"2024-04-25T05:37:08.399191+0000","last_peered":"2024-04-25T05:37:08.399191+0000","last_clean":"2024-04-25T05:37:08.399191+0000","last_became_active":"2024-04-25T05:36:54.235633+0000","last_became_peered":"2024-04-25T05:36:54.235633+0000","last_unstale":"2024-04-25T05:37:08.399191+0000","last_undegraded":"2024-04-25T05:37:08.399191+0000","last_fullsized":"2024-04-25T05:37:08.399191+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"136'8745","reported_seq":8921,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.839646+0000","last_change":"2024-04-25T05:36:31.742804+0000","last_active":"2024-04-25T05:37:17.839646+0000","last_peered":"2024-04-25T05:37:17.839646+0000","last_clean":"2024-04-25T05:37:17.839646+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:37:17.839646+0000","last_undegraded":"2024-04-25T05:37:17.839646+0000","last_fullsized":"2024-04-25T05:37:17.839646+0000","mapping_epoch":88,"log_start":"101'3171","ondisk_log_start":"101'3171","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6352","last_scrub_stamp":"2024-04-25T05:36:31.742451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.742451+0000","objects_scrubbed":6298,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:58.047202+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373620736,"num_objects":5701,"num_object_clones":0,"num_object_copies":17103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406740+0000","last_change":"2024-04-25T05:36:27.172930+0000","last_active":"2024-04-25T05:37:08.406740+0000","last_peered":"2024-04-25T05:37:08.406740+0000","last_clean":"2024-04-25T05:37:08.406740+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:37:08.406740+0000","last_undegraded":"2024-04-25T05:37:08.406740+0000","last_fullsized":"2024-04-25T05:37:08.406740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:27.172809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.172809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.542287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.390261+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:37:08.390261+0000","last_peered":"2024-04-25T05:37:08.390261+0000","last_clean":"2024-04-25T05:37:08.390261+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:37:08.390261+0000","last_undegraded":"2024-04-25T05:37:08.390261+0000","last_fullsized":"2024-04-25T05:37:08.390261+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382436+0000","last_change":"2024-04-25T05:36:54.235336+0000","last_active":"2024-04-25T05:37:08.382436+0000","last_peered":"2024-04-25T05:37:08.382436+0000","last_clean":"2024-04-25T05:37:08.382436+0000","last_became_active":"2024-04-25T05:36:54.234986+0000","last_became_peered":"2024-04-25T05:36:54.234986+0000","last_unstale":"2024-04-25T05:37:08.382436+0000","last_undegraded":"2024-04-25T05:37:08.382436+0000","last_fullsized":"2024-04-25T05:37:08.382436+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.750887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.750887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406509+0000","last_change":"2024-04-25T05:35:40.352567+0000","last_active":"2024-04-25T05:37:08.406509+0000","last_peered":"2024-04-25T05:37:08.406509+0000","last_clean":"2024-04-25T05:37:08.406509+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:37:08.406509+0000","last_undegraded":"2024-04-25T05:37:08.406509+0000","last_fullsized":"2024-04-25T05:37:08.406509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"136'8886","reported_seq":12214,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:17.956077+0000","last_change":"2024-04-25T05:36:55.428129+0000","last_active":"2024-04-25T05:37:17.956077+0000","last_peered":"2024-04-25T05:37:17.956077+0000","last_clean":"2024-04-25T05:36:53.167583+0000","last_became_active":"2024-04-25T05:36:55.354107+0000","last_became_peered":"2024-04-25T05:36:55.354107+0000","last_unstale":"2024-04-25T05:37:17.956077+0000","last_undegraded":"2024-04-25T05:37:17.956077+0000","last_fullsized":"2024-04-25T05:37:17.956077+0000","mapping_epoch":130,"log_start":"105'4039","ondisk_log_start":"105'4039","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387448832,"num_objects":5912,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4431,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1607,"num_bytes_recovered":105250816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":5912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.411961+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:37:08.411961+0000","last_peered":"2024-04-25T05:37:08.411961+0000","last_clean":"2024-04-25T05:37:08.411961+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:37:08.411961+0000","last_undegraded":"2024-04-25T05:37:08.411961+0000","last_fullsized":"2024-04-25T05:37:08.411961+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.505234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382319+0000","last_change":"2024-04-25T05:36:54.257218+0000","last_active":"2024-04-25T05:37:08.382319+0000","last_peered":"2024-04-25T05:37:08.382319+0000","last_clean":"2024-04-25T05:37:08.382319+0000","last_became_active":"2024-04-25T05:36:54.238031+0000","last_became_peered":"2024-04-25T05:36:54.238031+0000","last_unstale":"2024-04-25T05:37:08.382319+0000","last_undegraded":"2024-04-25T05:37:08.382319+0000","last_fullsized":"2024-04-25T05:37:08.382319+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.195800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398607+0000","last_change":"2024-04-25T05:36:54.244540+0000","last_active":"2024-04-25T05:37:08.398607+0000","last_peered":"2024-04-25T05:37:08.398607+0000","last_clean":"2024-04-25T05:37:08.398607+0000","last_became_active":"2024-04-25T05:36:54.244233+0000","last_became_peered":"2024-04-25T05:36:54.244233+0000","last_unstale":"2024-04-25T05:37:08.398607+0000","last_undegraded":"2024-04-25T05:37:08.398607+0000","last_fullsized":"2024-04-25T05:37:08.398607+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:33:57.656996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":265,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.802833+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:37:08.802833+0000","last_peered":"2024-04-25T05:37:08.802833+0000","last_clean":"2024-04-25T05:37:08.802833+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:37:08.802833+0000","last_undegraded":"2024-04-25T05:37:08.802833+0000","last_fullsized":"2024-04-25T05:37:08.802833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403160+0000","last_change":"2024-04-25T05:36:24.214089+0000","last_active":"2024-04-25T05:37:08.403160+0000","last_peered":"2024-04-25T05:37:08.403160+0000","last_clean":"2024-04-25T05:37:08.403160+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:37:08.403160+0000","last_undegraded":"2024-04-25T05:37:08.403160+0000","last_fullsized":"2024-04-25T05:37:08.403160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.446616+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"136'8744","reported_seq":8848,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.944368+0000","last_change":"2024-04-25T05:35:09.679881+0000","last_active":"2024-04-25T05:37:17.944368+0000","last_peered":"2024-04-25T05:37:17.944368+0000","last_clean":"2024-04-25T05:37:17.944368+0000","last_became_active":"2024-04-25T05:35:09.679580+0000","last_became_peered":"2024-04-25T05:35:09.679580+0000","last_unstale":"2024-04-25T05:37:17.944368+0000","last_undegraded":"2024-04-25T05:37:17.944368+0000","last_fullsized":"2024-04-25T05:37:17.944368+0000","mapping_epoch":88,"log_start":"104'3899","ondisk_log_start":"104'3899","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4845,"log_dups_size":0,"ondisk_log_size":4845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378535936,"num_objects":5776,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8744,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":160,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398991+0000","last_change":"2024-04-25T05:36:54.626908+0000","last_active":"2024-04-25T05:37:08.398991+0000","last_peered":"2024-04-25T05:37:08.398991+0000","last_clean":"2024-04-25T05:37:08.398991+0000","last_became_active":"2024-04-25T05:36:54.225394+0000","last_became_peered":"2024-04-25T05:36:54.225394+0000","last_unstale":"2024-04-25T05:37:08.398991+0000","last_undegraded":"2024-04-25T05:37:08.398991+0000","last_fullsized":"2024-04-25T05:37:08.398991+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.511619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.408000+0000","last_change":"2024-04-25T05:36:54.240763+0000","last_active":"2024-04-25T05:37:08.408000+0000","last_peered":"2024-04-25T05:37:08.408000+0000","last_clean":"2024-04-25T05:37:08.408000+0000","last_became_active":"2024-04-25T05:36:54.240014+0000","last_became_peered":"2024-04-25T05:36:54.240014+0000","last_unstale":"2024-04-25T05:37:08.408000+0000","last_undegraded":"2024-04-25T05:37:08.408000+0000","last_fullsized":"2024-04-25T05:37:08.408000+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:40.339655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.339655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":173,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.392771+0000","last_change":"2024-04-25T05:36:16.674171+0000","last_active":"2024-04-25T05:37:08.392771+0000","last_peered":"2024-04-25T05:37:08.392771+0000","last_clean":"2024-04-25T05:37:08.392771+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:37:08.392771+0000","last_undegraded":"2024-04-25T05:37:08.392771+0000","last_fullsized":"2024-04-25T05:37:08.392771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:27.824707+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":199,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.400188+0000","last_change":"2024-04-25T05:35:39.312001+0000","last_active":"2024-04-25T05:37:08.400188+0000","last_peered":"2024-04-25T05:37:08.400188+0000","last_clean":"2024-04-25T05:37:08.400188+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:37:08.400188+0000","last_undegraded":"2024-04-25T05:37:08.400188+0000","last_fullsized":"2024-04-25T05:37:08.400188+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.389490+0000","last_change":"2024-04-25T05:35:42.672247+0000","last_active":"2024-04-25T05:37:08.389490+0000","last_peered":"2024-04-25T05:37:08.389490+0000","last_clean":"2024-04-25T05:37:08.389490+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:37:08.389490+0000","last_undegraded":"2024-04-25T05:37:08.389490+0000","last_fullsized":"2024-04-25T05:37:08.389490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.672114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.672114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"136'8710","reported_seq":8815,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.922495+0000","last_change":"2024-04-25T05:35:09.683951+0000","last_active":"2024-04-25T05:37:17.922495+0000","last_peered":"2024-04-25T05:37:17.922495+0000","last_clean":"2024-04-25T05:37:17.922495+0000","last_became_active":"2024-04-25T05:35:09.682815+0000","last_became_peered":"2024-04-25T05:35:09.682815+0000","last_unstale":"2024-04-25T05:37:17.922495+0000","last_undegraded":"2024-04-25T05:37:17.922495+0000","last_fullsized":"2024-04-25T05:37:17.922495+0000","mapping_epoch":88,"log_start":"104'3867","ondisk_log_start":"104'3867","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377487360,"num_objects":5760,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.383033+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:37:08.383033+0000","last_peered":"2024-04-25T05:37:08.383033+0000","last_clean":"2024-04-25T05:37:08.383033+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:37:08.383033+0000","last_undegraded":"2024-04-25T05:37:08.383033+0000","last_fullsized":"2024-04-25T05:37:08.383033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.990922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":424,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397495+0000","last_change":"2024-04-25T05:36:57.317664+0000","last_active":"2024-04-25T05:37:08.397495+0000","last_peered":"2024-04-25T05:37:08.397495+0000","last_clean":"2024-04-25T05:37:08.397495+0000","last_became_active":"2024-04-25T05:36:57.317332+0000","last_became_peered":"2024-04-25T05:36:57.317332+0000","last_unstale":"2024-04-25T05:37:08.397495+0000","last_undegraded":"2024-04-25T05:37:08.397495+0000","last_fullsized":"2024-04-25T05:37:08.397495+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:36:28.364006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.364006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.795292+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.413172+0000","last_change":"2024-04-25T05:36:54.233589+0000","last_active":"2024-04-25T05:37:08.413172+0000","last_peered":"2024-04-25T05:37:08.413172+0000","last_clean":"2024-04-25T05:37:08.413172+0000","last_became_active":"2024-04-25T05:36:54.232305+0000","last_became_peered":"2024-04-25T05:36:54.232305+0000","last_unstale":"2024-04-25T05:37:08.413172+0000","last_undegraded":"2024-04-25T05:37:08.413172+0000","last_fullsized":"2024-04-25T05:37:08.413172+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.215324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"130'17","reported_seq":90,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768464+0000","last_change":"2024-04-25T05:36:54.268339+0000","last_active":"2024-04-25T05:37:08.768464+0000","last_peered":"2024-04-25T05:37:08.768464+0000","last_clean":"2024-04-25T05:37:08.768464+0000","last_became_active":"2024-04-25T05:36:54.245169+0000","last_became_peered":"2024-04-25T05:36:54.245169+0000","last_unstale":"2024-04-25T05:37:08.768464+0000","last_undegraded":"2024-04-25T05:37:08.768464+0000","last_fullsized":"2024-04-25T05:37:08.768464+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:27.678020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.678020+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398426+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:37:08.398426+0000","last_peered":"2024-04-25T05:37:08.398426+0000","last_clean":"2024-04-25T05:37:08.398426+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:37:08.398426+0000","last_undegraded":"2024-04-25T05:37:08.398426+0000","last_fullsized":"2024-04-25T05:37:08.398426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:31.527148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"136'8794","reported_seq":8980,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.688165+0000","last_change":"2024-04-25T05:36:51.500685+0000","last_active":"2024-04-25T05:37:17.688165+0000","last_peered":"2024-04-25T05:37:17.688165+0000","last_clean":"2024-04-25T05:37:17.688165+0000","last_became_active":"2024-04-25T05:35:09.683250+0000","last_became_peered":"2024-04-25T05:35:09.683250+0000","last_unstale":"2024-04-25T05:37:17.688165+0000","last_undegraded":"2024-04-25T05:37:17.688165+0000","last_fullsized":"2024-04-25T05:37:17.688165+0000","mapping_epoch":88,"log_start":"104'3763","ondisk_log_start":"104'3763","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7574","last_scrub_stamp":"2024-04-25T05:36:51.500427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.500427+0000","objects_scrubbed":7061,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:06.145379+0000","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378929152,"num_objects":5782,"num_object_clones":0,"num_object_copies":17346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8794,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399294+0000","last_change":"2024-04-25T05:35:38.262254+0000","last_active":"2024-04-25T05:37:08.399294+0000","last_peered":"2024-04-25T05:37:08.399294+0000","last_clean":"2024-04-25T05:37:08.399294+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:37:08.399294+0000","last_undegraded":"2024-04-25T05:37:08.399294+0000","last_fullsized":"2024-04-25T05:37:08.399294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.261996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.261996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406565+0000","last_change":"2024-04-25T05:36:26.153204+0000","last_active":"2024-04-25T05:37:08.406565+0000","last_peered":"2024-04-25T05:37:08.406565+0000","last_clean":"2024-04-25T05:37:08.406565+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:37:08.406565+0000","last_undegraded":"2024-04-25T05:37:08.406565+0000","last_fullsized":"2024-04-25T05:37:08.406565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:36.991917+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":362,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:15.985269+0000","last_change":"2024-04-25T05:36:20.496553+0000","last_active":"2024-04-25T05:37:15.985269+0000","last_peered":"2024-04-25T05:37:15.985269+0000","last_clean":"2024-04-25T05:37:15.985269+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:37:15.985269+0000","last_undegraded":"2024-04-25T05:37:15.985269+0000","last_fullsized":"2024-04-25T05:37:15.985269+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:36:20.496280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.496280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:30.974274+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405103+0000","last_change":"2024-04-25T05:36:32.152204+0000","last_active":"2024-04-25T05:37:08.405103+0000","last_peered":"2024-04-25T05:37:08.405103+0000","last_clean":"2024-04-25T05:37:08.405103+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:37:08.405103+0000","last_undegraded":"2024-04-25T05:37:08.405103+0000","last_fullsized":"2024-04-25T05:37:08.405103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:34.408462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.383061+0000","last_change":"2024-04-25T05:37:07.166910+0000","last_active":"2024-04-25T05:37:08.383061+0000","last_peered":"2024-04-25T05:37:08.383061+0000","last_clean":"2024-04-25T05:37:08.383061+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:37:08.383061+0000","last_undegraded":"2024-04-25T05:37:08.383061+0000","last_fullsized":"2024-04-25T05:37:08.383061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:36.034711+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"136'8808","reported_seq":8994,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.646776+0000","last_change":"2024-04-25T05:36:51.881839+0000","last_active":"2024-04-25T05:37:17.646776+0000","last_peered":"2024-04-25T05:37:17.646776+0000","last_clean":"2024-04-25T05:37:17.646776+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:37:17.646776+0000","last_undegraded":"2024-04-25T05:37:17.646776+0000","last_fullsized":"2024-04-25T05:37:17.646776+0000","mapping_epoch":88,"log_start":"102'3433","ondisk_log_start":"102'3433","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7619","last_scrub_stamp":"2024-04-25T05:36:51.881693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.881693+0000","objects_scrubbed":7027,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:13.086436+0000","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379846656,"num_objects":5796,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8808,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":152,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399486+0000","last_change":"2024-04-25T05:36:54.234953+0000","last_active":"2024-04-25T05:37:08.399486+0000","last_peered":"2024-04-25T05:37:08.399486+0000","last_clean":"2024-04-25T05:37:08.399486+0000","last_became_active":"2024-04-25T05:36:54.234602+0000","last_became_peered":"2024-04-25T05:36:54.234602+0000","last_unstale":"2024-04-25T05:37:08.399486+0000","last_undegraded":"2024-04-25T05:37:08.399486+0000","last_fullsized":"2024-04-25T05:37:08.399486+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399596+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:37:08.399596+0000","last_peered":"2024-04-25T05:37:08.399596+0000","last_clean":"2024-04-25T05:37:08.399596+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:37:08.399596+0000","last_undegraded":"2024-04-25T05:37:08.399596+0000","last_fullsized":"2024-04-25T05:37:08.399596+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:28.664326+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":202,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406256+0000","last_change":"2024-04-25T05:35:35.249157+0000","last_active":"2024-04-25T05:37:08.406256+0000","last_peered":"2024-04-25T05:37:08.406256+0000","last_clean":"2024-04-25T05:37:08.406256+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:37:08.406256+0000","last_undegraded":"2024-04-25T05:37:08.406256+0000","last_fullsized":"2024-04-25T05:37:08.406256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:35.248443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.248443+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403327+0000","last_change":"2024-04-25T05:36:54.237191+0000","last_active":"2024-04-25T05:37:08.403327+0000","last_peered":"2024-04-25T05:37:08.403327+0000","last_clean":"2024-04-25T05:37:08.403327+0000","last_became_active":"2024-04-25T05:36:54.234985+0000","last_became_peered":"2024-04-25T05:36:54.234985+0000","last_unstale":"2024-04-25T05:37:08.403327+0000","last_undegraded":"2024-04-25T05:37:08.403327+0000","last_fullsized":"2024-04-25T05:37:08.403327+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:33.818836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"136'8957","reported_seq":14129,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:17.698497+0000","last_change":"2024-04-25T05:36:55.336842+0000","last_active":"2024-04-25T05:37:17.698497+0000","last_peered":"2024-04-25T05:37:17.698497+0000","last_clean":"2024-04-25T05:36:53.189850+0000","last_became_active":"2024-04-25T05:36:55.314441+0000","last_became_peered":"2024-04-25T05:36:55.314441+0000","last_unstale":"2024-04-25T05:37:17.698497+0000","last_undegraded":"2024-04-25T05:37:17.698497+0000","last_fullsized":"2024-04-25T05:37:17.698497+0000","mapping_epoch":130,"log_start":"106'4367","ondisk_log_start":"106'4367","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382402560,"num_objects":5835,"num_object_clones":0,"num_object_copies":17505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3518,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8957,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2530,"num_bytes_recovered":165806080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,6],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"3,4,6","objects":5835}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403678+0000","last_change":"2024-04-25T05:35:44.317463+0000","last_active":"2024-04-25T05:37:08.403678+0000","last_peered":"2024-04-25T05:37:08.403678+0000","last_clean":"2024-04-25T05:37:08.403678+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:37:08.403678+0000","last_undegraded":"2024-04-25T05:37:08.403678+0000","last_fullsized":"2024-04-25T05:37:08.403678+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:09.893502+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382731+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:37:08.382731+0000","last_peered":"2024-04-25T05:37:08.382731+0000","last_clean":"2024-04-25T05:37:08.382731+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:37:08.382731+0000","last_undegraded":"2024-04-25T05:37:08.382731+0000","last_fullsized":"2024-04-25T05:37:08.382731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:35.016206+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":111,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.390012+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:37:08.390012+0000","last_peered":"2024-04-25T05:37:08.390012+0000","last_clean":"2024-04-25T05:37:08.390012+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:37:08.390012+0000","last_undegraded":"2024-04-25T05:37:08.390012+0000","last_fullsized":"2024-04-25T05:37:08.390012+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"130'9","reported_seq":208,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.794719+0000","last_change":"2024-04-25T05:36:54.253353+0000","last_active":"2024-04-25T05:37:08.794719+0000","last_peered":"2024-04-25T05:37:08.794719+0000","last_clean":"2024-04-25T05:37:08.794719+0000","last_became_active":"2024-04-25T05:36:54.238623+0000","last_became_peered":"2024-04-25T05:36:54.238623+0000","last_unstale":"2024-04-25T05:37:08.794719+0000","last_undegraded":"2024-04-25T05:37:08.794719+0000","last_fullsized":"2024-04-25T05:37:08.794719+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:42.311077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.311077+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"136'8828","reported_seq":8935,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.841990+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:37:17.841990+0000","last_peered":"2024-04-25T05:37:17.841990+0000","last_clean":"2024-04-25T05:37:17.841990+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:37:17.841990+0000","last_undegraded":"2024-04-25T05:37:17.841990+0000","last_fullsized":"2024-04-25T05:37:17.841990+0000","mapping_epoch":88,"log_start":"108'4627","ondisk_log_start":"108'4627","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381943808,"num_objects":5828,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":160,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398103+0000","last_change":"2024-04-25T05:36:59.627764+0000","last_active":"2024-04-25T05:37:08.398103+0000","last_peered":"2024-04-25T05:37:08.398103+0000","last_clean":"2024-04-25T05:37:08.398103+0000","last_became_active":"2024-04-25T05:36:54.234860+0000","last_became_peered":"2024-04-25T05:36:54.234860+0000","last_unstale":"2024-04-25T05:37:08.398103+0000","last_undegraded":"2024-04-25T05:37:08.398103+0000","last_fullsized":"2024-04-25T05:37:08.398103+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:26.357457+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":160,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.405203+0000","last_change":"2024-04-25T05:35:36.294561+0000","last_active":"2024-04-25T05:37:08.405203+0000","last_peered":"2024-04-25T05:37:08.405203+0000","last_clean":"2024-04-25T05:37:08.405203+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:37:08.405203+0000","last_undegraded":"2024-04-25T05:37:08.405203+0000","last_fullsized":"2024-04-25T05:37:08.405203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":211,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397897+0000","last_change":"2024-04-25T05:36:54.273512+0000","last_active":"2024-04-25T05:37:08.397897+0000","last_peered":"2024-04-25T05:37:08.397897+0000","last_clean":"2024-04-25T05:37:08.397897+0000","last_became_active":"2024-04-25T05:36:54.248063+0000","last_became_peered":"2024-04-25T05:36:54.248063+0000","last_unstale":"2024-04-25T05:37:08.397897+0000","last_undegraded":"2024-04-25T05:37:08.397897+0000","last_fullsized":"2024-04-25T05:37:08.397897+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:50.634187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":198,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.797416+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:37:08.797416+0000","last_peered":"2024-04-25T05:37:08.797416+0000","last_clean":"2024-04-25T05:37:08.797416+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:37:08.797416+0000","last_undegraded":"2024-04-25T05:37:08.797416+0000","last_fullsized":"2024-04-25T05:37:08.797416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:43.089867+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"136'8788","reported_seq":8895,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.850150+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:37:17.850150+0000","last_peered":"2024-04-25T05:37:17.850150+0000","last_clean":"2024-04-25T05:37:17.850150+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:37:17.850150+0000","last_undegraded":"2024-04-25T05:37:17.850150+0000","last_fullsized":"2024-04-25T05:37:17.850150+0000","mapping_epoch":88,"log_start":"108'4669","ondisk_log_start":"108'4669","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397304+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:37:08.397304+0000","last_peered":"2024-04-25T05:37:08.397304+0000","last_clean":"2024-04-25T05:37:08.397304+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:37:08.397304+0000","last_undegraded":"2024-04-25T05:37:08.397304+0000","last_fullsized":"2024-04-25T05:37:08.397304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.155991+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.407141+0000","last_change":"2024-04-25T05:35:06.377537+0000","last_active":"2024-04-25T05:37:08.407141+0000","last_peered":"2024-04-25T05:37:08.407141+0000","last_clean":"2024-04-25T05:37:08.407141+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:37:08.407141+0000","last_undegraded":"2024-04-25T05:37:08.407141+0000","last_fullsized":"2024-04-25T05:37:08.407141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":210,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403687+0000","last_change":"2024-04-25T05:35:30.243679+0000","last_active":"2024-04-25T05:37:08.403687+0000","last_peered":"2024-04-25T05:37:08.403687+0000","last_clean":"2024-04-25T05:37:08.403687+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:37:08.403687+0000","last_undegraded":"2024-04-25T05:37:08.403687+0000","last_fullsized":"2024-04-25T05:37:08.403687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397609+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:37:08.397609+0000","last_peered":"2024-04-25T05:37:08.397609+0000","last_clean":"2024-04-25T05:37:08.397609+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:37:08.397609+0000","last_undegraded":"2024-04-25T05:37:08.397609+0000","last_fullsized":"2024-04-25T05:37:08.397609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:37.962428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"136'8758","reported_seq":12647,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:17.891901+0000","last_change":"2024-04-25T05:36:55.410743+0000","last_active":"2024-04-25T05:37:17.891901+0000","last_peered":"2024-04-25T05:37:17.891901+0000","last_clean":"2024-04-25T05:36:53.144949+0000","last_became_active":"2024-04-25T05:36:55.337352+0000","last_became_peered":"2024-04-25T05:36:55.337352+0000","last_unstale":"2024-04-25T05:37:17.891901+0000","last_undegraded":"2024-04-25T05:37:17.891901+0000","last_fullsized":"2024-04-25T05:37:17.891901+0000","mapping_epoch":130,"log_start":"108'4620","ondisk_log_start":"108'4620","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377094144,"num_objects":5754,"num_object_clones":0,"num_object_copies":17262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4022,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8758,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1890,"num_bytes_recovered":123863040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":5754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382435+0000","last_change":"2024-04-25T05:36:54.001702+0000","last_active":"2024-04-25T05:37:08.382435+0000","last_peered":"2024-04-25T05:37:08.382435+0000","last_clean":"2024-04-25T05:37:08.382435+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:37:08.382435+0000","last_undegraded":"2024-04-25T05:37:08.382435+0000","last_fullsized":"2024-04-25T05:37:08.382435+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.057784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":189,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768416+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:37:08.768416+0000","last_peered":"2024-04-25T05:37:08.768416+0000","last_clean":"2024-04-25T05:37:08.768416+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:37:08.768416+0000","last_undegraded":"2024-04-25T05:37:08.768416+0000","last_fullsized":"2024-04-25T05:37:08.768416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.403597+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:37:08.403597+0000","last_peered":"2024-04-25T05:37:08.403597+0000","last_clean":"2024-04-25T05:37:08.403597+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:37:08.403597+0000","last_undegraded":"2024-04-25T05:37:08.403597+0000","last_fullsized":"2024-04-25T05:37:08.403597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":179,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.400302+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:37:08.400302+0000","last_peered":"2024-04-25T05:37:08.400302+0000","last_clean":"2024-04-25T05:37:08.400302+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:37:08.400302+0000","last_undegraded":"2024-04-25T05:37:08.400302+0000","last_fullsized":"2024-04-25T05:37:08.400302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"136'8863","reported_seq":12484,"reported_epoch":136,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:17.854322+0000","last_change":"2024-04-25T05:36:55.415940+0000","last_active":"2024-04-25T05:37:17.854322+0000","last_peered":"2024-04-25T05:37:17.854322+0000","last_clean":"2024-04-25T05:36:53.182883+0000","last_became_active":"2024-04-25T05:36:55.354455+0000","last_became_peered":"2024-04-25T05:36:55.354455+0000","last_unstale":"2024-04-25T05:37:17.854322+0000","last_undegraded":"2024-04-25T05:37:17.854322+0000","last_fullsized":"2024-04-25T05:37:17.854322+0000","mapping_epoch":130,"log_start":"108'4688","ondisk_log_start":"108'4688","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":4899,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381943832,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4248,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8863,"num_write_kb":470081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1727,"num_bytes_recovered":113180672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,6],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":5829}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.414263+0000","last_change":"2024-04-25T05:36:54.232549+0000","last_active":"2024-04-25T05:37:08.414263+0000","last_peered":"2024-04-25T05:37:08.414263+0000","last_clean":"2024-04-25T05:37:08.414263+0000","last_became_active":"2024-04-25T05:36:54.231886+0000","last_became_peered":"2024-04-25T05:36:54.231886+0000","last_unstale":"2024-04-25T05:37:08.414263+0000","last_undegraded":"2024-04-25T05:37:08.414263+0000","last_fullsized":"2024-04-25T05:37:08.414263+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.382961+0000","last_change":"2024-04-25T05:37:08.197220+0000","last_active":"2024-04-25T05:37:08.382961+0000","last_peered":"2024-04-25T05:37:08.382961+0000","last_clean":"2024-04-25T05:37:08.382961+0000","last_became_active":"2024-04-25T05:36:54.244909+0000","last_became_peered":"2024-04-25T05:36:54.244909+0000","last_unstale":"2024-04-25T05:37:08.382961+0000","last_undegraded":"2024-04-25T05:37:08.382961+0000","last_fullsized":"2024-04-25T05:37:08.382961+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:24.881588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397695+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:37:08.397695+0000","last_peered":"2024-04-25T05:37:08.397695+0000","last_clean":"2024-04-25T05:37:08.397695+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:37:08.397695+0000","last_undegraded":"2024-04-25T05:37:08.397695+0000","last_fullsized":"2024-04-25T05:37:08.397695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":204,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399330+0000","last_change":"2024-04-25T05:36:54.269613+0000","last_active":"2024-04-25T05:37:08.399330+0000","last_peered":"2024-04-25T05:37:08.399330+0000","last_clean":"2024-04-25T05:37:08.399330+0000","last_became_active":"2024-04-25T05:36:54.244739+0000","last_became_peered":"2024-04-25T05:36:54.244739+0000","last_unstale":"2024-04-25T05:37:08.399330+0000","last_undegraded":"2024-04-25T05:37:08.399330+0000","last_fullsized":"2024-04-25T05:37:08.399330+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:24.863584+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"136'8768","reported_seq":9014,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.789403+0000","last_change":"2024-04-25T05:36:53.309704+0000","last_active":"2024-04-25T05:37:17.789403+0000","last_peered":"2024-04-25T05:37:17.789403+0000","last_clean":"2024-04-25T05:37:17.789403+0000","last_became_active":"2024-04-25T05:35:09.678971+0000","last_became_peered":"2024-04-25T05:35:09.678971+0000","last_unstale":"2024-04-25T05:37:17.789403+0000","last_undegraded":"2024-04-25T05:37:17.789403+0000","last_fullsized":"2024-04-25T05:37:17.789403+0000","mapping_epoch":88,"log_start":"101'3206","ondisk_log_start":"101'3206","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7661","last_scrub_stamp":"2024-04-25T05:36:53.309266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.309266+0000","objects_scrubbed":6949,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:06.908353+0000","scrub_duration":2789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378273792,"num_objects":5772,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.388820+0000","last_change":"2024-04-25T05:35:39.752489+0000","last_active":"2024-04-25T05:37:08.388820+0000","last_peered":"2024-04-25T05:37:08.388820+0000","last_clean":"2024-04-25T05:37:08.388820+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:37:08.388820+0000","last_undegraded":"2024-04-25T05:37:08.388820+0000","last_fullsized":"2024-04-25T05:37:08.388820+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:39.752319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.752319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399839+0000","last_change":"2024-04-25T05:36:49.750652+0000","last_active":"2024-04-25T05:37:08.399839+0000","last_peered":"2024-04-25T05:37:08.399839+0000","last_clean":"2024-04-25T05:37:08.399839+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:37:08.399839+0000","last_undegraded":"2024-04-25T05:37:08.399839+0000","last_fullsized":"2024-04-25T05:37:08.399839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.472869+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.397395+0000","last_change":"2024-04-25T05:36:54.226814+0000","last_active":"2024-04-25T05:37:08.397395+0000","last_peered":"2024-04-25T05:37:08.397395+0000","last_clean":"2024-04-25T05:37:08.397395+0000","last_became_active":"2024-04-25T05:36:54.226138+0000","last_became_peered":"2024-04-25T05:36:54.226138+0000","last_unstale":"2024-04-25T05:37:08.397395+0000","last_undegraded":"2024-04-25T05:37:08.397395+0000","last_fullsized":"2024-04-25T05:37:08.397395+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:32.885743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398986+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:37:08.398986+0000","last_peered":"2024-04-25T05:37:08.398986+0000","last_clean":"2024-04-25T05:37:08.398986+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:37:08.398986+0000","last_undegraded":"2024-04-25T05:37:08.398986+0000","last_fullsized":"2024-04-25T05:37:08.398986+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"136'9039","reported_seq":9145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:17.919572+0000","last_change":"2024-04-25T05:35:09.682779+0000","last_active":"2024-04-25T05:37:17.919572+0000","last_peered":"2024-04-25T05:37:17.919572+0000","last_clean":"2024-04-25T05:37:17.919572+0000","last_became_active":"2024-04-25T05:35:09.681887+0000","last_became_peered":"2024-04-25T05:35:09.681887+0000","last_unstale":"2024-04-25T05:37:17.919572+0000","last_undegraded":"2024-04-25T05:37:17.919572+0000","last_fullsized":"2024-04-25T05:37:17.919572+0000","mapping_epoch":88,"log_start":"105'4256","ondisk_log_start":"105'4256","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392232960,"num_objects":5985,"num_object_clones":0,"num_object_copies":17955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":480768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.399645+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:37:08.399645+0000","last_peered":"2024-04-25T05:37:08.399645+0000","last_clean":"2024-04-25T05:37:08.399645+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:37:08.399645+0000","last_undegraded":"2024-04-25T05:37:08.399645+0000","last_fullsized":"2024-04-25T05:37:08.399645+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:31.276359+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":201,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.768206+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:37:08.768206+0000","last_peered":"2024-04-25T05:37:08.768206+0000","last_clean":"2024-04-25T05:37:08.768206+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:37:08.768206+0000","last_undegraded":"2024-04-25T05:37:08.768206+0000","last_fullsized":"2024-04-25T05:37:08.768206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.406388+0000","last_change":"2024-04-25T05:36:54.712552+0000","last_active":"2024-04-25T05:37:08.406388+0000","last_peered":"2024-04-25T05:37:08.406388+0000","last_clean":"2024-04-25T05:37:08.406388+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:37:08.406388+0000","last_undegraded":"2024-04-25T05:37:08.406388+0000","last_fullsized":"2024-04-25T05:37:08.406388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:08.084311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":186,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-04-25T05:37:08.398946+0000","last_change":"2024-04-25T05:36:54.254870+0000","last_active":"2024-04-25T05:37:08.398946+0000","last_peered":"2024-04-25T05:37:08.398946+0000","last_clean":"2024-04-25T05:37:08.398946+0000","last_became_active":"2024-04-25T05:36:54.234440+0000","last_became_peered":"2024-04-25T05:36:54.234440+0000","last_unstale":"2024-04-25T05:37:08.398946+0000","last_undegraded":"2024-04-25T05:37:08.398946+0000","last_fullsized":"2024-04-25T05:37:08.398946+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:53.169712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6074597400,"num_objects":92692,"num_object_clones":0,"num_object_copies":278076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20058,"num_objects_unfound":0,"num_objects_dirty":92692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141042,"num_write_kb":7479425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9745,"num_bytes_recovered":638582784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18817626112,"data_stored":18817613896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75301,"ondisk_log_size":75301,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":6268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070509,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949676,"kb_used_data":1874084,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91795476,"statfs":{"total":95995035648,"available":93998567424,"internally_reserved":0,"allocated":1919062016,"data_stored":1918652097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574472,"kb_used_data":2475264,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91170680,"statfs":{"total":95995035648,"available":93358776320,"internally_reserved":0,"allocated":2534670336,"data_stored":2534230807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3095992,"kb_used_data":2996784,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":90649160,"statfs":{"total":95995035648,"available":92824739840,"internally_reserved":0,"allocated":3068706816,"data_stored":3068283293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2547600,"kb_used_data":2444232,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91197552,"statfs":{"total":95995035648,"available":93386293248,"internally_reserved":0,"allocated":2502893568,"data_stored":2502510514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3603704,"kb_used_data":3455920,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90141448,"statfs":{"total":95995035648,"available":92304842752,"internally_reserved":0,"allocated":3538862080,"data_stored":3538438349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1940016,"kb_used_data":1868584,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91805136,"statfs":{"total":95995035648,"available":94008459264,"internally_reserved":0,"allocated":1913430016,"data_stored":1913014651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691894,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1554464,"kb_used_data":1483032,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92190688,"statfs":{"total":95995035648,"available":94403264512,"internally_reserved":0,"allocated":1518624768,"data_stored":1518225301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3052015616,"data_stored":3052011544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2523860992,"data_stored":2523856920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2499608576,"data_stored":2499608576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3535667200,"data_stored":3535667200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1893269504,"data_stored":1893269504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1508376576,"data_stored":1508376576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1902776320,"data_stored":1902772248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1902051328,"data_stored":1902051328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:37:20.529 INFO:tasks.thrashosds.thrasher:Setting 7.4 to [7, 5, 1, 3, 6, 2] 2024-04-25T05:37:20.529 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.4', '7', '5', '1', '3', '6', '2'] 2024-04-25T05:37:20.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 7.4 7 5 1 3 6 2 2024-04-25T05:37:20.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:20.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:20.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:20.755 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:20.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:20.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:20.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:20.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:21.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:21.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:21.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:21.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:21.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:21 smithi043 ceph-mon[27010]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:21 smithi043 ceph-mon[27010]: pgmap v146: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:21.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:21 smithi043 ceph-mon[37492]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:21 smithi043 ceph-mon[37492]: pgmap v146: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:21 smithi060 ceph-mon[30898]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:21 smithi060 ceph-mon[30898]: pgmap v146: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:21.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:21.526 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:21.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:21.652 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:21.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:21.778 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:21.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:21.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:21.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:22.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:22.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:22.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:22.187 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:22.266 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:22.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:22.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:22.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:22.471 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:22.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.597 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:22.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:22.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:22.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:22.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:22.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:22.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:23.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:23.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:23.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:23.192 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[27010]: pgmap v147: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[27010]: osdmap e138: 8 total, 8 up, 7 in 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[37492]: pgmap v147: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:23.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:23 smithi043 ceph-mon[37492]: osdmap e138: 8 total, 8 up, 7 in 2024-04-25T05:37:23.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:23.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:23.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:23.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:23 smithi060 ceph-mon[30898]: pgmap v147: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 20058/278766 objects misplaced (7.195%); 27 MiB/s, 429 objects/s recovering 2024-04-25T05:37:23.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:23 smithi060 ceph-mon[30898]: osdmap e138: 8 total, 8 up, 7 in 2024-04-25T05:37:23.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:23.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:23.378 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:23.420 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:23.420 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:37:23.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.420 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:145744 7.4 e138/88 7:263f6b48:::benchmark_data_smithi043_74356_object28875:head v 138'9060, mlcod=138'9058)", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.416936+0000", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00073690099999999998, 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00076670999999999998, 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416936+0000", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416936+0000", 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416936+0000", 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2599999999999998e-07 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416940+0000", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0509999999999998e-06 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416946+0000", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8510000000000004e-06 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416951+0000", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2619999999999998e-06 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.417196+0000", 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024426500000000002 2024-04-25T05:37:23.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.417216+0000", 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0292000000000002e-05 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:145736 7.d 7:b4b2d5b7:::benchmark_data_smithi043_74356_object28867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.409734+0000", 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0079386270000000002, 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080496449999999994, 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.424 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 145736 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409734+0000", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409734+0000", 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409735+0000", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2619999999999999e-06 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409740+0000", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1880000000000001e-06 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409750+0000", 2024-04-25T05:37:23.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5959999999999993e-06 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409757+0000", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.418e-06 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415881+0000", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0061238029999999997 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416010+0000", 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129054 2024-04-25T05:37:23.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,6", 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416150+0000", 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013987099999999999 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:145739 7.1 e138/130 7:8414e169:::benchmark_data_smithi043_74356_object28870:head v 138'9265, mlcod=138'9262)", 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.409985+0000", 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0076872080000000001, 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0079018539999999998, 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.428 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409985+0000", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409985+0000", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409986+0000", 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3299999999999998e-07 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409990+0000", 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8909999999999997e-06 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.409996+0000", 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3589999999999996e-06 2024-04-25T05:37:23.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.410001+0000", 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2460000000000003e-06 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.413406+0000", 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034047040000000002 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.413426+0000", 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9862000000000001e-05 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(7.d 138/130 [PushReplyOp(7:b5e01928:::benchmark_data_smithi043_74356_object115440:head)])", 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.412197+0000", 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0054756930000000002, 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0057654580000000002, 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412197+0000", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412197+0000", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412197+0000", 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3099999999999999e-07 2024-04-25T05:37:23.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412200+0000", 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9720000000000001e-06 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412206+0000", 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8130000000000001e-06 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412213+0000", 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1280000000000002e-06 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:145743 7.d 7.983697bd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.412724+0000", 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0049483289999999996, 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053029979999999997, 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:23.436 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 145743 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412724+0000", 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412724+0000", 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412725+0000", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.184e-06 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412731+0000", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0889999999999999e-06 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412741+0000", 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0791000000000001e-05 2024-04-25T05:37:23.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.412750+0000", 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6780000000000006e-06 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:145745 7.d 7.7ff0adad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.415564+0000", 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0021083880000000001, 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002544265, 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:23.439 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 145745 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415564+0000", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415564+0000", 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415565+0000", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2300000000000001e-06 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415571+0000", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1220000000000003e-06 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415580+0000", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4520000000000003e-06 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415587+0000", 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2699999999999999e-06 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:145747 7.b 7:d9a22d97:::benchmark_data_smithi043_74356_object28878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.415600+0000", 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020725000000000001, 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025775820000000001, 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:37:23.442 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 145747 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415600+0000", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415600+0000", 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415601+0000", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7100000000000002e-07 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415603+0000", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5179999999999999e-06 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415609+0000", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8719999999999999e-06 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415615+0000", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7610000000000004e-06 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415923+0000", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000308709 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416041+0000", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117803 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5,6", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416176+0000", 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013448999999999999 2024-04-25T05:37:23.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:145748 7.d 7.3f71452d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.415624+0000", 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020490109999999999, 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026321740000000001, 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:23.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 145748 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415624+0000", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415624+0000", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415624+0000", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2300000000000002e-07 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415626+0000", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1169999999999998e-06 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415631+0000", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4e-06 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415637+0000", 2024-04-25T05:37:23.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.959e-06 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:145746 7.1 e138/130 7:81b75b94:::benchmark_data_smithi043_74356_object28877:head v 138'9266, mlcod=138'9262)", 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.415656+0000", 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.002016636, 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026593749999999998, 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:37:23.449 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415656+0000", 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415656+0000", 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415656+0000", 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5400000000000002e-07 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415661+0000", 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5210000000000004e-06 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415667+0000", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.925e-06 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.415672+0000", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4210000000000003e-06 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.417035+0000", 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0013626000000000001 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.417055+0000", 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0248e-05 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:145749 7.1 e138/130)", 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:23.416887+0000", 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00078577400000000004, 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001486043, 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:23.453 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416887+0000", 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416887+0000", 2024-04-25T05:37:23.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416887+0000", 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2e-07 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416892+0000", 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.062e-06 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:23.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416901+0000", 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2700000000000004e-06 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:23.416907+0000", 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4030000000000004e-06 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 10 2024-04-25T05:37:23.456 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:23.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.507 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:23.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:23.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:37:23.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:23.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:23.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:23.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:23.913 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:23.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:24.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:24.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:24.085 INFO:teuthology.orchestra.run.smithi043.stderr:set 7.4 pg_upmap_items mapping to [7->5,1->3,6->2] 2024-04-25T05:37:24.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:24.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:37:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:24.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4014795995' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4014795995' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:24.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:24.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:24.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:24 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4014795995' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]: dispatch 2024-04-25T05:37:24.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:37:24.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:24.593 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:24.609 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:24.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:24.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:24.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:24.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:24.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:24.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:24.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:25.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:25.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[27010]: pgmap v149: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 17007/266004 objects misplaced (6.394%); 35 MiB/s, 555 objects/s recovering 2024-04-25T05:37:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]': finished 2024-04-25T05:37:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[27010]: osdmap e139: 8 total, 8 up, 7 in 2024-04-25T05:37:25.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[27010]: osdmap e140: 8 total, 8 up, 7 in 2024-04-25T05:37:25.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[37492]: pgmap v149: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 17007/266004 objects misplaced (6.394%); 35 MiB/s, 555 objects/s recovering 2024-04-25T05:37:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]': finished 2024-04-25T05:37:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[37492]: osdmap e139: 8 total, 8 up, 7 in 2024-04-25T05:37:25.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:25 smithi043 ceph-mon[37492]: osdmap e140: 8 total, 8 up, 7 in 2024-04-25T05:37:25.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:25.248 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:25.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:25.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:25.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:25 smithi060 ceph-mon[30898]: pgmap v149: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s; 17007/266004 objects misplaced (6.394%); 35 MiB/s, 555 objects/s recovering 2024-04-25T05:37:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:25 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.4", "id": [7, 5, 1, 3, 6, 2]}]': finished 2024-04-25T05:37:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:25 smithi060 ceph-mon[30898]: osdmap e139: 8 total, 8 up, 7 in 2024-04-25T05:37:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:25 smithi060 ceph-mon[30898]: osdmap e140: 8 total, 8 up, 7 in 2024-04-25T05:37:25.505 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:25.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:25.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:25.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:25.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:25.828 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:25.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:25.890 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:25.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:25.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:25.951 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:25.995 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:26.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:26.038 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:26.049 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:26.087 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:26.129 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:26.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:26.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:26.170 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:26.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:26.200 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:26.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:26.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:26.297 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:26.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:26.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:26.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:26.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:26.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:26.802 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:26.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:26.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:26.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:26 smithi060 ceph-mon[30898]: pgmap v152: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 17007/266004 objects misplaced (6.394%); 22 MiB/s, 359 objects/s recovering 2024-04-25T05:37:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:26 smithi060 ceph-mon[30898]: osdmap e141: 8 total, 8 up, 7 in 2024-04-25T05:37:26.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:26 smithi060 ceph-mon[30898]: 7.4 starting backfill to osd.3 from (0'0,0'0] MAX to 138'9088 2024-04-25T05:37:26.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3990590365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:27.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:27.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:27.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[37492]: pgmap v152: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 17007/266004 objects misplaced (6.394%); 22 MiB/s, 359 objects/s recovering 2024-04-25T05:37:27.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[37492]: osdmap e141: 8 total, 8 up, 7 in 2024-04-25T05:37:27.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[37492]: 7.4 starting backfill to osd.3 from (0'0,0'0] MAX to 138'9088 2024-04-25T05:37:27.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3990590365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:27.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[27010]: pgmap v152: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 17007/266004 objects misplaced (6.394%); 22 MiB/s, 359 objects/s recovering 2024-04-25T05:37:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[27010]: osdmap e141: 8 total, 8 up, 7 in 2024-04-25T05:37:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[27010]: 7.4 starting backfill to osd.3 from (0'0,0'0] MAX to 138'9088 2024-04-25T05:37:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3990590365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:27.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:27.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:27.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:27.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:27.347 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:27.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:27.485 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:27.586 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:27.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:27.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:27.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:27.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:27.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:27.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:27.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:28.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:28.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:28.162 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:28.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:28.288 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:28.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:28.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:28.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:28.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:28.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:28.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:28.736 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:37:28.736 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:37:28.753 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:37:28.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:28.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:28 smithi043 ceph-mon[27010]: pgmap v154: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 839 op/s; 15872/264546 objects misplaced (6.000%); 36 MiB/s, 571 objects/s recovering 2024-04-25T05:37:28.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:37:28.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:28 smithi043 ceph-mon[37492]: pgmap v154: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 839 op/s; 15872/264546 objects misplaced (6.000%); 36 MiB/s, 571 objects/s recovering 2024-04-25T05:37:28.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:37:28.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:28.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:28.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:29.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:29.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:29.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:29.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:29.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:28 smithi060 ceph-mon[30898]: pgmap v154: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 5.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 839 op/s; 15872/264546 objects misplaced (6.000%); 36 MiB/s, 571 objects/s recovering 2024-04-25T05:37:29.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:37:29.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:29.362 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:29.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:29.486 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:29.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:29.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:29.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:29.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:29.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:37:29.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[27010]: osdmap e142: 8 total, 8 up, 7 in 2024-04-25T05:37:29.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:29.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:29.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:37:29.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:29 smithi043 ceph-mon[37492]: osdmap e142: 8 total, 8 up, 7 in 2024-04-25T05:37:29.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:29.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:29.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:29.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:30.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:30.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:30.161 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:30.242 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:37:30.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:37:30.242 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T05:37:30.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:37:30.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:29 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:30.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1873121901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:37:30.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:29 smithi060 ceph-mon[30898]: osdmap e142: 8 total, 8 up, 7 in 2024-04-25T05:37:30.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:30.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:30.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:30.432 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:30.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:30.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:30.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:30.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:30.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:30.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:30.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:30.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:31.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:31.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:31.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:30 smithi043 ceph-mon[37492]: pgmap v156: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 493 op/s; 17032/257667 objects misplaced (6.610%); 54 MiB/s, 857 objects/s recovering 2024-04-25T05:37:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:30 smithi043 ceph-mon[27010]: pgmap v156: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 493 op/s; 17032/257667 objects misplaced (6.610%); 54 MiB/s, 857 objects/s recovering 2024-04-25T05:37:31.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:31.208 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:30 smithi060 ceph-mon[30898]: pgmap v156: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 493 op/s; 17032/257667 objects misplaced (6.610%); 54 MiB/s, 857 objects/s recovering 2024-04-25T05:37:31.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:31.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:31.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:31.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:31.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:31.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:31.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:31.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.839 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:31.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:31.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:31.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:32.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:37:32.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:32.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:37:32.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:37:32.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:32.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:32.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:37:32.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:37:32.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:37:32.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:32.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:31 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:32.091 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:32.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:32.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:31 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:37:32.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:31 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:37:32.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:31 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:32.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:31 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:32.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.416 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:32.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:32.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:32.539 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:32.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:32.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:32.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:32.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[27010]: pgmap v157: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 398 op/s; 17032/257667 objects misplaced (6.610%); 43 MiB/s, 692 objects/s recovering 2024-04-25T05:37:32.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[27010]: osdmap e143: 8 total, 8 up, 7 in 2024-04-25T05:37:32.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:32.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[37492]: pgmap v157: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 398 op/s; 17032/257667 objects misplaced (6.610%); 43 MiB/s, 692 objects/s recovering 2024-04-25T05:37:32.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[37492]: osdmap e143: 8 total, 8 up, 7 in 2024-04-25T05:37:32.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:32.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:32.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:33.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:33.075 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:33.100 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:33.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:33.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:33.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:32 smithi060 ceph-mon[30898]: pgmap v157: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 398 op/s; 17032/257667 objects misplaced (6.610%); 43 MiB/s, 692 objects/s recovering 2024-04-25T05:37:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:32 smithi060 ceph-mon[30898]: osdmap e143: 8 total, 8 up, 7 in 2024-04-25T05:37:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:33.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:33.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:33.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:33.481 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:33.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:33.604 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:33.614 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:37:33.614 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 143, 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:37:31.764759+0000", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:36:53.196620+0000", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5804032, 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nodeep-scrub", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:37:33.615 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 7, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:37:33.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 1, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 1, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 1, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 1, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 1, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "21", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.619 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.620 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.621 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 7.8899998664855957, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 7.8899998664855957, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.37999999523162842, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 3, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 3, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.625 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.626 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:37:33.627 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.628 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:33.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.630 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.631 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.632 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.633 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.634 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.635 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.636 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.637 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.639 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4900000095367432, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4900000095367432, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.642 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.643 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.644 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.4900000095367432, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.4900000095367432, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:33.645 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:37:33.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.648 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.649 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.650 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:37:33.651 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 7, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_0", 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:35:08.463631+0000", 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:37:33.652 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:37:33.653 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "91", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:37:33.654 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:37:33.655 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:37:33.656 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4900000095367432, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4900000095367432, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:37:33.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 140, 2024-04-25T05:37:33.658 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.662 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.663 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:37:33.665 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.666 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 96, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.667 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:37:33.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:37:33.669 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.670 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:37:33.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:37:33.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.674 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 130, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.676 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:37:33.677 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.678 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 129, 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.680 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.682 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 88, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:37:33.683 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.684 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:37:33.685 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 132, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:37:33.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:37:33.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:37:04.225827+0000", 2024-04-25T05:37:33.691 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:35.267959+0000", 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:37:33.692 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:59.003964+0000", 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.693 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:37:21.528641+0000", 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.694 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:33.086906+0000", 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:37:30.933566+0000", 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:43.196834+0000", 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:37:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:36:29.659289+0000", 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [], 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.2", 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.12", 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.698 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.4", 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 3 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:37:33.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.1", 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.4", 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.7", 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.b", 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.c", 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "7.d", 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: 6 2024-04-25T05:37:33.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:37:33.703 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:33.704 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:37:33.704 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:37:33.704 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:37:33.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:33.706 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:33.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:33.730 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:33.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:33.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:33.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:33.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:34.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:34.081 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:37:34.081 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:37:34.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:34.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/796743779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:34.081 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:34.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/796743779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:34.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:34.125 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:34.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:34.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:34.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/796743779' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:37:34.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:34.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:37:34.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:34.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:34.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:34.544 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:34.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:34.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:34.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:34.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:34.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:34.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:34.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:35.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:37:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:37:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[27010]: pgmap v159: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 849 op/s; 13866/245709 objects misplaced (5.643%); 57 MiB/s, 908 objects/s recovering 2024-04-25T05:37:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:37:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:37:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:34 smithi043 ceph-mon[37492]: pgmap v159: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 849 op/s; 13866/245709 objects misplaced (5.643%); 57 MiB/s, 908 objects/s recovering 2024-04-25T05:37:35.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:35.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:35.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:34 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:37:35.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:34 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:37:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:34 smithi060 ceph-mon[30898]: pgmap v159: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 849 op/s; 13866/245709 objects misplaced (5.643%); 57 MiB/s, 908 objects/s recovering 2024-04-25T05:37:35.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:35.361 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:35.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:35.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:35.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:35.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:35.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:35.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:35.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:35.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:35.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:35.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:36.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:36.135 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:36.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:35 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:36.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:35 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:36.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:35 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:36.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:35 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:36.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:36.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:35 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:36.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:35 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:36.261 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:36.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:36.388 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:36.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:36.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:36.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:36.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:36.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:36.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:36.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:36.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:37.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:37.041 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[27010]: pgmap v160: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s; 13866/245709 objects misplaced (5.643%); 48 MiB/s, 771 objects/s recovering 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[27010]: osdmap e144: 8 total, 8 up, 7 in 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:37:37.096 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:37:37.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[37492]: pgmap v160: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s; 13866/245709 objects misplaced (5.643%); 48 MiB/s, 771 objects/s recovering 2024-04-25T05:37:37.097 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:36 smithi043 ceph-mon[37492]: osdmap e144: 8 total, 8 up, 7 in 2024-04-25T05:37:37.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:37.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:36 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:37:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:36 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:37:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:36 smithi060 ceph-mon[30898]: pgmap v160: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s; 13866/245709 objects misplaced (5.643%); 48 MiB/s, 771 objects/s recovering 2024-04-25T05:37:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:36 smithi060 ceph-mon[30898]: osdmap e144: 8 total, 8 up, 7 in 2024-04-25T05:37:37.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:37.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:37.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:37.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:37.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:37.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:37.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:37.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:37.817 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:37.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:37.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:37.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:38.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:38.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:37:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:37:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:38.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[37492]: osdmap e145: 8 total, 8 up, 7 in 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:37:38.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:37 smithi043 ceph-mon[27010]: osdmap e145: 8 total, 8 up, 7 in 2024-04-25T05:37:38.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:38.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:37:38.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:37 smithi060 ceph-mon[30898]: osdmap e145: 8 total, 8 up, 7 in 2024-04-25T05:37:38.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:38.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:38.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:38.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:38.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:38.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:38.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:38.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:38.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:38.878 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:38.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:38.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:37:38.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:39.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:39.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:39.129 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[27010]: pgmap v163: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 792 op/s; 13266/243426 objects misplaced (5.450%); 31 MiB/s, 494 objects/s recovering 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:37:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:38 smithi043 ceph-mon[37492]: pgmap v163: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 792 op/s; 13266/243426 objects misplaced (5.450%); 31 MiB/s, 494 objects/s recovering 2024-04-25T05:37:39.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:39.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:38 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:37:39.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:38 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:37:39.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:38 smithi060 ceph-mon[30898]: pgmap v163: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 792 op/s; 13266/243426 objects misplaced (5.450%); 31 MiB/s, 494 objects/s recovering 2024-04-25T05:37:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:39.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:39.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:39.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:39.506 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:39.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:39.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:39.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:39.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:39.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:39.994 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:40.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:40.036 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:40.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:40.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:40.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:40.286 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:40.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:40.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:40.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:40.535 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:40.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:40.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:40.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:40.860 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:37:40.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:40.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:41.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:41.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[27010]: pgmap v164: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 10775/233667 objects misplaced (4.611%); 38 MiB/s, 614 objects/s recovering 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/660428846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[37492]: pgmap v164: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 10775/233667 objects misplaced (4.611%); 38 MiB/s, 614 objects/s recovering 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/660428846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:41.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:41.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:40 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:37:41.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:40 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:37:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:40 smithi060 ceph-mon[30898]: pgmap v164: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s; 10775/233667 objects misplaced (4.611%); 38 MiB/s, 614 objects/s recovering 2024-04-25T05:37:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/660428846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:37:41.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:41.326 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:41.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:41.453 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:41.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:41.576 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:41.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:41.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:41.848 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:41.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:41.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:41.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:42.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:42.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:37:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:37:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[27010]: osdmap e146: 8 total, 8 up, 7 in 2024-04-25T05:37:42.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:37:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:37:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:41 smithi043 ceph-mon[37492]: osdmap e146: 8 total, 8 up, 7 in 2024-04-25T05:37:42.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:42.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:41 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:37:42.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:41 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:37:42.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:41 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:37:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:41 smithi060 ceph-mon[30898]: osdmap e146: 8 total, 8 up, 7 in 2024-04-25T05:37:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:42.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:42.283 INFO:tasks.thrashosds.thrasher:j is {'epoch': 143, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:37:31.764759+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:36:53.196620+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-25T05:35:08.463631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 132, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:37:04.225827+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:35.267959+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:59.003964+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:37:21.528641+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:33.086906+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:37:30.933566+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:36:43.196834+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:36:29.659289+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '7.4', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.1', 'osds': [4, 3, 6]}, {'pgid': '7.4', 'osds': [0, 4, 1]}, {'pgid': '7.7', 'osds': [0, 3, 6]}, {'pgid': '7.b', 'osds': [1, 5, 6]}, {'pgid': '7.c', 'osds': [3, 7, 6]}, {'pgid': '7.d', 'osds': [1, 0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:37:42.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:37:42.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:42.373 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:42.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:42.495 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:42.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:42.618 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:42.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:42.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:42.867 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:42.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:42.995 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:42.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:43.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: pgmap v166: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 10775/233667 objects misplaced (4.611%); 26 MiB/s, 412 objects/s recovering 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[37492]: osdmap e147: 8 total, 8 up, 7 in 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: pgmap v166: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 10775/233667 objects misplaced (4.611%); 26 MiB/s, 412 objects/s recovering 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:43.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:42 smithi043 ceph-mon[27010]: osdmap e147: 8 total, 8 up, 7 in 2024-04-25T05:37:43.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:43.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: pgmap v166: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 10775/233667 objects misplaced (4.611%); 26 MiB/s, 412 objects/s recovering 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:42 smithi060 ceph-mon[30898]: osdmap e147: 8 total, 8 up, 7 in 2024-04-25T05:37:43.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:43.342 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:43.343 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:43.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:43.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:43.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:43.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:43.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:43.692 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:43.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:43.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:43.958 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:43.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:44.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:44.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: osdmap e148: 8 total, 8 up, 7 in 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: 3.3 deep-scrub starts 2024-04-25T05:37:44.150 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:43 smithi060 ceph-mon[30898]: 3.3 deep-scrub ok 2024-04-25T05:37:44.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: osdmap e148: 8 total, 8 up, 7 in 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: 3.3 deep-scrub starts 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[27010]: 3.3 deep-scrub ok 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: osdmap e148: 8 total, 8 up, 7 in 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: 3.3 deep-scrub starts 2024-04-25T05:37:44.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:43 smithi043 ceph-mon[37492]: 3.3 deep-scrub ok 2024-04-25T05:37:44.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:44.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:44.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:44.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:44.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:44.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:44.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:44.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:44.754 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:44.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:37:44.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:44.905 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:44.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:45.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:45.029 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:45.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:45.138 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:44 smithi043 ceph-mon[27010]: pgmap v169: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 7190/223470 objects misplaced (3.217%); 53 MiB/s, 847 objects/s recovering 2024-04-25T05:37:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/59636151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:45.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:44 smithi043 ceph-mon[37492]: pgmap v169: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 7190/223470 objects misplaced (3.217%); 53 MiB/s, 847 objects/s recovering 2024-04-25T05:37:45.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/59636151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:45.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:45.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:44 smithi060 ceph-mon[30898]: pgmap v169: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 7190/223470 objects misplaced (3.217%); 53 MiB/s, 847 objects/s recovering 2024-04-25T05:37:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/59636151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:45.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:45.279 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:45.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:45.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:45.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:45.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:45.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:45.666 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:45.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:45.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:45.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:45.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:45.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:46.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:46.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:46.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:46.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:46.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:46.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:46.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:46.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:46.624 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:46.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:46.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:46.770 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:46.770 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:37:46.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:162563 7.5 e148/88 7:a02f0588:::benchmark_data_smithi043_74356_object45694:head v 148'10109, mlcod=148'10106)", 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:46.766614+0000", 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00072900700000000005, 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075884699999999999, 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766614+0000", 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766614+0000", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766614+0000", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0999999999999998e-07 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766620+0000", 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.976e-06 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766630+0000", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4530000000000008e-06 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766637+0000", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9299999999999997e-06 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766694+0000", 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7355000000000002e-05 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766713+0000", 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9219999999999999e-05 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:162559 7.a e148/88)", 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:46.766694+0000", 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.000648136, 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00076559000000000002, 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:46.774 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766694+0000", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766694+0000", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766695+0000", 2024-04-25T05:37:46.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7399999999999998e-07 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766700+0000", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2959999999999998e-06 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766708+0000", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9999999999999996e-06 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.766715+0000", 2024-04-25T05:37:46.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6189999999999996e-06 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24734.0:162548 7.e e148/88)", 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:46.767313+0000", 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 2.9365000000000001e-05, 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019952199999999999, 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:46.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767313+0000", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767313+0000", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767314+0000", 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4999999999999998e-07 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767317+0000", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2040000000000002e-06 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767324+0000", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1389999999999998e-06 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.767330+0000", 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2960000000000004e-06 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:162548 7.e 7:772e6f9a:::benchmark_data_smithi043_74356_object45679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:46.748737+0000", 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.018605067999999999, 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018841714999999998, 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 162548 2024-04-25T05:37:46.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748737+0000", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748737+0000", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748738+0000", 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0500000000000004e-07 2024-04-25T05:37:46.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748741+0000", 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7130000000000001e-06 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748747+0000", 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1809999999999998e-06 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:37:46.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.748753+0000", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7520000000000002e-06 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.759876+0000", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011123218000000001 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.760024+0000", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014778799999999999 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.760164+0000", 2024-04-25T05:37:46.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000140767 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:46.764122+0000", 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039577450000000004 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-04-25T05:37:46.784 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:46.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:46.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:46.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:47.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:47.054 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:37:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:37:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[27010]: pgmap v170: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 567 op/s; 7190/223470 objects misplaced (3.217%); 33 MiB/s, 528 objects/s recovering 2024-04-25T05:37:47.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:37:47.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:37:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:46 smithi043 ceph-mon[37492]: pgmap v170: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 567 op/s; 7190/223470 objects misplaced (3.217%); 33 MiB/s, 528 objects/s recovering 2024-04-25T05:37:47.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:47.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:47.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:46 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:37:47.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:46 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:37:47.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:46 smithi060 ceph-mon[30898]: pgmap v170: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 567 op/s; 7190/223470 objects misplaced (3.217%); 33 MiB/s, 528 objects/s recovering 2024-04-25T05:37:47.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:47.316 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:47.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:47.440 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:47.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:47.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:47.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:47.712 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:47.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:47.823 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:37:47.834 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:37:47.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:47.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:47.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:37:47.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:47.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:48.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:48.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:37:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:48 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:37:48.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:48.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:48.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:37:48.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:48.366 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:48.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:48.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:48.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:37:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:48 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:37:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:48 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:37:48.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:48.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:48.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:48.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:48.798 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:48.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:48.937 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:48.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:49.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:49.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:49.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:49.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[27010]: pgmap v171: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 600 op/s; 7190/221307 objects misplaced (3.249%); 29 MiB/s, 461 objects/s recovering 2024-04-25T05:37:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[27010]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:37:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:37:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[37492]: pgmap v171: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 600 op/s; 7190/221307 objects misplaced (3.249%); 29 MiB/s, 461 objects/s recovering 2024-04-25T05:37:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[37492]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:37:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:37:49.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:49.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:49.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:49 smithi060 ceph-mon[30898]: pgmap v171: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 600 op/s; 7190/221307 objects misplaced (3.249%); 29 MiB/s, 461 objects/s recovering 2024-04-25T05:37:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:49 smithi060 ceph-mon[30898]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:37:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:49 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:37:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:49 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:37:49.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:49.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:49.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:49.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:49.812 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:49.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:49.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:49.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:50.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:50.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:50.079 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:50.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:50.226 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-04-25T05:37:47.721237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4819982899,"num_objects":73769,"num_object_clones":0,"num_object_copies":221307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7190,"num_objects_unfound":0,"num_objects_dirty":73769,"num_whiteouts":0,"num_read":1443,"num_read_kb":1257,"num_write":161002,"num_write_kb":7481886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24903,"num_bytes_recovered":1627591462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76089,"ondisk_log_size":76089,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14728276,"kb_used_data":13885320,"kb_used_omap":10,"kb_used_meta":842869,"kb_avail":641487788,"statfs":{"total":671965249536,"available":656883494912,"internally_reserved":0,"allocated":14218567680,"data_stored":14215663874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":863097962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-270008320,"num_objects":-4120,"num_object_clones":0,"num_object_copies":-12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3585,"num_objects_unfound":0,"num_objects_dirty":-4120,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3172,"num_bytes_recovered":207880192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.872363"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":161,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.873998+0000","last_change":"2024-04-25T05:34:04.720464+0000","last_active":"2024-04-25T05:37:42.873998+0000","last_peered":"2024-04-25T05:37:42.873998+0000","last_clean":"2024-04-25T05:37:42.873998+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:37:42.873998+0000","last_undegraded":"2024-04-25T05:37:42.873998+0000","last_fullsized":"2024-04-25T05:37:42.873998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885724+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:37:42.885724+0000","last_peered":"2024-04-25T05:37:42.885724+0000","last_clean":"2024-04-25T05:37:42.885724+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:37:42.885724+0000","last_undegraded":"2024-04-25T05:37:42.885724+0000","last_fullsized":"2024-04-25T05:37:42.885724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877592+0000","last_change":"2024-04-25T05:36:18.176520+0000","last_active":"2024-04-25T05:37:42.877592+0000","last_peered":"2024-04-25T05:37:42.877592+0000","last_clean":"2024-04-25T05:37:42.877592+0000","last_became_active":"2024-04-25T05:33:58.677227+0000","last_became_peered":"2024-04-25T05:33:58.677227+0000","last_unstale":"2024-04-25T05:37:42.877592+0000","last_undegraded":"2024-04-25T05:37:42.877592+0000","last_fullsized":"2024-04-25T05:37:42.877592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.176394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.176394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.873794+0000","last_change":"2024-04-25T05:36:54.265352+0000","last_active":"2024-04-25T05:37:42.873794+0000","last_peered":"2024-04-25T05:37:42.873794+0000","last_clean":"2024-04-25T05:37:42.873794+0000","last_became_active":"2024-04-25T05:36:54.243668+0000","last_became_peered":"2024-04-25T05:36:54.243668+0000","last_unstale":"2024-04-25T05:37:42.873794+0000","last_undegraded":"2024-04-25T05:37:42.873794+0000","last_fullsized":"2024-04-25T05:37:42.873794+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":168,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885626+0000","last_change":"2024-04-25T05:36:54.236185+0000","last_active":"2024-04-25T05:37:42.885626+0000","last_peered":"2024-04-25T05:37:42.885626+0000","last_clean":"2024-04-25T05:37:42.885626+0000","last_became_active":"2024-04-25T05:36:54.235695+0000","last_became_peered":"2024-04-25T05:36:54.235695+0000","last_unstale":"2024-04-25T05:37:42.885626+0000","last_undegraded":"2024-04-25T05:37:42.885626+0000","last_fullsized":"2024-04-25T05:37:42.885626+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:47.617414+0000","last_change":"2024-04-25T05:37:47.617448+0000","last_active":"2024-04-25T05:37:47.617414+0000","last_peered":"2024-04-25T05:37:47.617414+0000","last_clean":"2024-04-25T05:37:47.617414+0000","last_became_active":"2024-04-25T05:36:54.229992+0000","last_became_peered":"2024-04-25T05:36:54.229992+0000","last_unstale":"2024-04-25T05:37:47.617414+0000","last_undegraded":"2024-04-25T05:37:47.617414+0000","last_fullsized":"2024-04-25T05:37:47.617414+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:47.617351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:47.617351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:16.683166+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887022+0000","last_change":"2024-04-25T05:36:54.230655+0000","last_active":"2024-04-25T05:37:42.887022+0000","last_peered":"2024-04-25T05:37:42.887022+0000","last_clean":"2024-04-25T05:37:42.887022+0000","last_became_active":"2024-04-25T05:36:54.230169+0000","last_became_peered":"2024-04-25T05:36:54.230169+0000","last_unstale":"2024-04-25T05:37:42.887022+0000","last_undegraded":"2024-04-25T05:37:42.887022+0000","last_fullsized":"2024-04-25T05:37:42.887022+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":243,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886621+0000","last_change":"2024-04-25T05:34:00.690377+0000","last_active":"2024-04-25T05:37:42.886621+0000","last_peered":"2024-04-25T05:37:42.886621+0000","last_clean":"2024-04-25T05:37:42.886621+0000","last_became_active":"2024-04-25T05:34:00.690053+0000","last_became_peered":"2024-04-25T05:34:00.690053+0000","last_unstale":"2024-04-25T05:37:42.886621+0000","last_undegraded":"2024-04-25T05:37:42.886621+0000","last_fullsized":"2024-04-25T05:37:42.886621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877046+0000","last_change":"2024-04-25T05:35:32.284750+0000","last_active":"2024-04-25T05:37:42.877046+0000","last_peered":"2024-04-25T05:37:42.877046+0000","last_clean":"2024-04-25T05:37:42.877046+0000","last_became_active":"2024-04-25T05:34:04.724016+0000","last_became_peered":"2024-04-25T05:34:04.724016+0000","last_unstale":"2024-04-25T05:37:42.877046+0000","last_undegraded":"2024-04-25T05:37:42.877046+0000","last_fullsized":"2024-04-25T05:37:42.877046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.284637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.284637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916713+0000","last_change":"2024-04-25T05:36:29.981966+0000","last_active":"2024-04-25T05:37:42.916713+0000","last_peered":"2024-04-25T05:37:42.916713+0000","last_clean":"2024-04-25T05:37:42.916713+0000","last_became_active":"2024-04-25T05:34:02.709166+0000","last_became_peered":"2024-04-25T05:34:02.709166+0000","last_unstale":"2024-04-25T05:37:42.916713+0000","last_undegraded":"2024-04-25T05:37:42.916713+0000","last_fullsized":"2024-04-25T05:37:42.916713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.093687+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916121+0000","last_change":"2024-04-25T05:36:54.244217+0000","last_active":"2024-04-25T05:37:42.916121+0000","last_peered":"2024-04-25T05:37:42.916121+0000","last_clean":"2024-04-25T05:37:42.916121+0000","last_became_active":"2024-04-25T05:36:54.243924+0000","last_became_peered":"2024-04-25T05:36:54.243924+0000","last_unstale":"2024-04-25T05:37:42.916121+0000","last_undegraded":"2024-04-25T05:37:42.916121+0000","last_fullsized":"2024-04-25T05:37:42.916121+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.105851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874232+0000","last_change":"2024-04-25T05:36:49.748038+0000","last_active":"2024-04-25T05:37:42.874232+0000","last_peered":"2024-04-25T05:37:42.874232+0000","last_clean":"2024-04-25T05:37:42.874232+0000","last_became_active":"2024-04-25T05:34:00.689416+0000","last_became_peered":"2024-04-25T05:34:00.689416+0000","last_unstale":"2024-04-25T05:37:42.874232+0000","last_undegraded":"2024-04-25T05:37:42.874232+0000","last_fullsized":"2024-04-25T05:37:42.874232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:11.511083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.915314+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:37:42.915314+0000","last_peered":"2024-04-25T05:37:42.915314+0000","last_clean":"2024-04-25T05:37:42.915314+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:37:42.915314+0000","last_undegraded":"2024-04-25T05:37:42.915314+0000","last_fullsized":"2024-04-25T05:37:42.915314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:55.892919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907434+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:37:42.907434+0000","last_peered":"2024-04-25T05:37:42.907434+0000","last_clean":"2024-04-25T05:37:42.907434+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:37:42.907434+0000","last_undegraded":"2024-04-25T05:37:42.907434+0000","last_fullsized":"2024-04-25T05:37:42.907434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:59.764698+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916050+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:37:42.916050+0000","last_peered":"2024-04-25T05:37:42.916050+0000","last_clean":"2024-04-25T05:37:42.916050+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:37:42.916050+0000","last_undegraded":"2024-04-25T05:37:42.916050+0000","last_fullsized":"2024-04-25T05:37:42.916050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.095746+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874484+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:37:42.874484+0000","last_peered":"2024-04-25T05:37:42.874484+0000","last_clean":"2024-04-25T05:37:42.874484+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:37:42.874484+0000","last_undegraded":"2024-04-25T05:37:42.874484+0000","last_fullsized":"2024-04-25T05:37:42.874484+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":175,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874906+0000","last_change":"2024-04-25T05:36:54.245109+0000","last_active":"2024-04-25T05:37:42.874906+0000","last_peered":"2024-04-25T05:37:42.874906+0000","last_clean":"2024-04-25T05:37:42.874906+0000","last_became_active":"2024-04-25T05:36:54.244292+0000","last_became_peered":"2024-04-25T05:36:54.244292+0000","last_unstale":"2024-04-25T05:37:42.874906+0000","last_undegraded":"2024-04-25T05:37:42.874906+0000","last_fullsized":"2024-04-25T05:37:42.874906+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:19.691117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884570+0000","last_change":"2024-04-25T05:36:52.722150+0000","last_active":"2024-04-25T05:37:42.884570+0000","last_peered":"2024-04-25T05:37:42.884570+0000","last_clean":"2024-04-25T05:37:42.884570+0000","last_became_active":"2024-04-25T05:34:02.708245+0000","last_became_peered":"2024-04-25T05:34:02.708245+0000","last_unstale":"2024-04-25T05:37:42.884570+0000","last_undegraded":"2024-04-25T05:37:42.884570+0000","last_fullsized":"2024-04-25T05:37:42.884570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.475127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881126+0000","last_change":"2024-04-25T05:36:55.198379+0000","last_active":"2024-04-25T05:37:42.881126+0000","last_peered":"2024-04-25T05:37:42.881126+0000","last_clean":"2024-04-25T05:37:42.881126+0000","last_became_active":"2024-04-25T05:33:58.683576+0000","last_became_peered":"2024-04-25T05:33:58.683576+0000","last_unstale":"2024-04-25T05:37:42.881126+0000","last_undegraded":"2024-04-25T05:37:42.881126+0000","last_fullsized":"2024-04-25T05:37:42.881126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:11.778316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":202,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881909+0000","last_change":"2024-04-25T05:35:28.590848+0000","last_active":"2024-04-25T05:37:42.881909+0000","last_peered":"2024-04-25T05:37:42.881909+0000","last_clean":"2024-04-25T05:37:42.881909+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:37:42.881909+0000","last_undegraded":"2024-04-25T05:37:42.881909+0000","last_fullsized":"2024-04-25T05:37:42.881909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:28.590759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.590759+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":168,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887463+0000","last_change":"2024-04-25T05:36:54.237606+0000","last_active":"2024-04-25T05:37:42.887463+0000","last_peered":"2024-04-25T05:37:42.887463+0000","last_clean":"2024-04-25T05:37:42.887463+0000","last_became_active":"2024-04-25T05:36:54.237282+0000","last_became_peered":"2024-04-25T05:36:54.237282+0000","last_unstale":"2024-04-25T05:37:42.887463+0000","last_undegraded":"2024-04-25T05:37:42.887463+0000","last_fullsized":"2024-04-25T05:37:42.887463+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":165,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874824+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:37:42.874824+0000","last_peered":"2024-04-25T05:37:42.874824+0000","last_clean":"2024-04-25T05:37:42.874824+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:37:42.874824+0000","last_undegraded":"2024-04-25T05:37:42.874824+0000","last_fullsized":"2024-04-25T05:37:42.874824+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874234+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:37:42.874234+0000","last_peered":"2024-04-25T05:37:42.874234+0000","last_clean":"2024-04-25T05:37:42.874234+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:37:42.874234+0000","last_undegraded":"2024-04-25T05:37:42.874234+0000","last_fullsized":"2024-04-25T05:37:42.874234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.515357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":205,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887504+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:37:42.887504+0000","last_peered":"2024-04-25T05:37:42.887504+0000","last_clean":"2024-04-25T05:37:42.887504+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:37:42.887504+0000","last_undegraded":"2024-04-25T05:37:42.887504+0000","last_fullsized":"2024-04-25T05:37:42.887504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907902+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:37:42.907902+0000","last_peered":"2024-04-25T05:37:42.907902+0000","last_clean":"2024-04-25T05:37:42.907902+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:37:42.907902+0000","last_undegraded":"2024-04-25T05:37:42.907902+0000","last_fullsized":"2024-04-25T05:37:42.907902+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:00.648018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"130'9","reported_seq":91,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.875188+0000","last_change":"2024-04-25T05:36:54.240381+0000","last_active":"2024-04-25T05:37:42.875188+0000","last_peered":"2024-04-25T05:37:42.875188+0000","last_clean":"2024-04-25T05:37:42.875188+0000","last_became_active":"2024-04-25T05:36:54.232031+0000","last_became_peered":"2024-04-25T05:36:54.232031+0000","last_unstale":"2024-04-25T05:37:42.875188+0000","last_undegraded":"2024-04-25T05:37:42.875188+0000","last_fullsized":"2024-04-25T05:37:42.875188+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885630+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:37:42.885630+0000","last_peered":"2024-04-25T05:37:42.885630+0000","last_clean":"2024-04-25T05:37:42.885630+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:37:42.885630+0000","last_undegraded":"2024-04-25T05:37:42.885630+0000","last_fullsized":"2024-04-25T05:37:42.885630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:10.328531+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":205,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880500+0000","last_change":"2024-04-25T05:36:15.129016+0000","last_active":"2024-04-25T05:37:42.880500+0000","last_peered":"2024-04-25T05:37:42.880500+0000","last_clean":"2024-04-25T05:37:42.880500+0000","last_became_active":"2024-04-25T05:34:01.001925+0000","last_became_peered":"2024-04-25T05:34:01.001925+0000","last_unstale":"2024-04-25T05:37:42.880500+0000","last_undegraded":"2024-04-25T05:37:42.880500+0000","last_fullsized":"2024-04-25T05:37:42.880500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:36:15.128887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.128887+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884296+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:37:42.884296+0000","last_peered":"2024-04-25T05:37:42.884296+0000","last_clean":"2024-04-25T05:37:42.884296+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:37:42.884296+0000","last_undegraded":"2024-04-25T05:37:42.884296+0000","last_fullsized":"2024-04-25T05:37:42.884296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880999+0000","last_change":"2024-04-25T05:36:14.171466+0000","last_active":"2024-04-25T05:37:42.880999+0000","last_peered":"2024-04-25T05:37:42.880999+0000","last_clean":"2024-04-25T05:37:42.880999+0000","last_became_active":"2024-04-25T05:34:02.710301+0000","last_became_peered":"2024-04-25T05:34:02.710301+0000","last_unstale":"2024-04-25T05:37:42.880999+0000","last_undegraded":"2024-04-25T05:37:42.880999+0000","last_fullsized":"2024-04-25T05:37:42.880999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.171349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.171349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885713+0000","last_change":"2024-04-25T05:35:36.324777+0000","last_active":"2024-04-25T05:37:42.885713+0000","last_peered":"2024-04-25T05:37:42.885713+0000","last_clean":"2024-04-25T05:37:42.885713+0000","last_became_active":"2024-04-25T05:33:58.676166+0000","last_became_peered":"2024-04-25T05:33:58.676166+0000","last_unstale":"2024-04-25T05:37:42.885713+0000","last_undegraded":"2024-04-25T05:37:42.885713+0000","last_fullsized":"2024-04-25T05:37:42.885713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":120,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916784+0000","last_change":"2024-04-25T05:37:35.189759+0000","last_active":"2024-04-25T05:37:42.916784+0000","last_peered":"2024-04-25T05:37:42.916784+0000","last_clean":"2024-04-25T05:37:42.916784+0000","last_became_active":"2024-04-25T05:36:54.237414+0000","last_became_peered":"2024-04-25T05:36:54.237414+0000","last_unstale":"2024-04-25T05:37:42.916784+0000","last_undegraded":"2024-04-25T05:37:42.916784+0000","last_fullsized":"2024-04-25T05:37:42.916784+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:37:35.189581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:35.189581+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:56.385705+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907262+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:37:42.907262+0000","last_peered":"2024-04-25T05:37:42.907262+0000","last_clean":"2024-04-25T05:37:42.907262+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:37:42.907262+0000","last_undegraded":"2024-04-25T05:37:42.907262+0000","last_fullsized":"2024-04-25T05:37:42.907262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:44.881845+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":49,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.873864+0000","last_change":"2024-04-25T05:36:54.248389+0000","last_active":"2024-04-25T05:37:42.873864+0000","last_peered":"2024-04-25T05:37:42.873864+0000","last_clean":"2024-04-25T05:37:42.873864+0000","last_became_active":"2024-04-25T05:36:54.247892+0000","last_became_peered":"2024-04-25T05:36:54.247892+0000","last_unstale":"2024-04-25T05:37:42.873864+0000","last_undegraded":"2024-04-25T05:37:42.873864+0000","last_fullsized":"2024-04-25T05:37:42.873864+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877814+0000","last_change":"2024-04-25T05:36:21.217686+0000","last_active":"2024-04-25T05:37:42.877814+0000","last_peered":"2024-04-25T05:37:42.877814+0000","last_clean":"2024-04-25T05:37:42.877814+0000","last_became_active":"2024-04-25T05:33:58.677994+0000","last_became_peered":"2024-04-25T05:33:58.677994+0000","last_unstale":"2024-04-25T05:37:42.877814+0000","last_undegraded":"2024-04-25T05:37:42.877814+0000","last_fullsized":"2024-04-25T05:37:42.877814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.483356+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885820+0000","last_change":"2024-04-25T05:36:54.262135+0000","last_active":"2024-04-25T05:37:42.885820+0000","last_peered":"2024-04-25T05:37:42.885820+0000","last_clean":"2024-04-25T05:37:42.885820+0000","last_became_active":"2024-04-25T05:36:54.244002+0000","last_became_peered":"2024-04-25T05:36:54.244002+0000","last_unstale":"2024-04-25T05:37:42.885820+0000","last_undegraded":"2024-04-25T05:37:42.885820+0000","last_fullsized":"2024-04-25T05:37:42.885820+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":168,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886927+0000","last_change":"2024-04-25T05:36:54.236715+0000","last_active":"2024-04-25T05:37:42.886927+0000","last_peered":"2024-04-25T05:37:42.886927+0000","last_clean":"2024-04-25T05:37:42.886927+0000","last_became_active":"2024-04-25T05:36:54.235240+0000","last_became_peered":"2024-04-25T05:36:54.235240+0000","last_unstale":"2024-04-25T05:37:42.886927+0000","last_undegraded":"2024-04-25T05:37:42.886927+0000","last_fullsized":"2024-04-25T05:37:42.886927+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907507+0000","last_change":"2024-04-25T05:37:00.575011+0000","last_active":"2024-04-25T05:37:42.907507+0000","last_peered":"2024-04-25T05:37:42.907507+0000","last_clean":"2024-04-25T05:37:42.907507+0000","last_became_active":"2024-04-25T05:36:54.239270+0000","last_became_peered":"2024-04-25T05:36:54.239270+0000","last_unstale":"2024-04-25T05:37:42.907507+0000","last_undegraded":"2024-04-25T05:37:42.907507+0000","last_fullsized":"2024-04-25T05:37:42.907507+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:24.324484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916427+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:37:42.916427+0000","last_peered":"2024-04-25T05:37:42.916427+0000","last_clean":"2024-04-25T05:37:42.916427+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:37:42.916427+0000","last_undegraded":"2024-04-25T05:37:42.916427+0000","last_fullsized":"2024-04-25T05:37:42.916427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":229,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886521+0000","last_change":"2024-04-25T05:36:54.252887+0000","last_active":"2024-04-25T05:37:42.886521+0000","last_peered":"2024-04-25T05:37:42.886521+0000","last_clean":"2024-04-25T05:37:42.886521+0000","last_became_active":"2024-04-25T05:36:54.232522+0000","last_became_peered":"2024-04-25T05:36:54.232522+0000","last_unstale":"2024-04-25T05:37:42.886521+0000","last_undegraded":"2024-04-25T05:37:42.886521+0000","last_fullsized":"2024-04-25T05:37:42.886521+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.876712+0000","last_change":"2024-04-25T05:35:34.276800+0000","last_active":"2024-04-25T05:37:42.876712+0000","last_peered":"2024-04-25T05:37:42.876712+0000","last_clean":"2024-04-25T05:37:42.876712+0000","last_became_active":"2024-04-25T05:34:04.726270+0000","last_became_peered":"2024-04-25T05:34:04.726270+0000","last_unstale":"2024-04-25T05:37:42.876712+0000","last_undegraded":"2024-04-25T05:37:42.876712+0000","last_fullsized":"2024-04-25T05:37:42.876712+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.276673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.276673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885548+0000","last_change":"2024-04-25T05:34:02.710335+0000","last_active":"2024-04-25T05:37:42.885548+0000","last_peered":"2024-04-25T05:37:42.885548+0000","last_clean":"2024-04-25T05:37:42.885548+0000","last_became_active":"2024-04-25T05:34:02.709956+0000","last_became_peered":"2024-04-25T05:34:02.709956+0000","last_unstale":"2024-04-25T05:37:42.885548+0000","last_undegraded":"2024-04-25T05:37:42.885548+0000","last_fullsized":"2024-04-25T05:37:42.885548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907877+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:37:42.907877+0000","last_peered":"2024-04-25T05:37:42.907877+0000","last_clean":"2024-04-25T05:37:42.907877+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:37:42.907877+0000","last_undegraded":"2024-04-25T05:37:42.907877+0000","last_fullsized":"2024-04-25T05:37:42.907877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:13.840716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":155,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.915428+0000","last_change":"2024-04-25T05:37:42.212545+0000","last_active":"2024-04-25T05:37:42.915428+0000","last_peered":"2024-04-25T05:37:42.915428+0000","last_clean":"2024-04-25T05:37:42.915428+0000","last_became_active":"2024-04-25T05:36:54.238351+0000","last_became_peered":"2024-04-25T05:36:54.238351+0000","last_unstale":"2024-04-25T05:37:42.915428+0000","last_undegraded":"2024-04-25T05:37:42.915428+0000","last_fullsized":"2024-04-25T05:37:42.915428+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:47.891340+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886222+0000","last_change":"2024-04-25T05:34:04.725641+0000","last_active":"2024-04-25T05:37:42.886222+0000","last_peered":"2024-04-25T05:37:42.886222+0000","last_clean":"2024-04-25T05:37:42.886222+0000","last_became_active":"2024-04-25T05:34:04.725318+0000","last_became_peered":"2024-04-25T05:34:04.725318+0000","last_unstale":"2024-04-25T05:37:42.886222+0000","last_undegraded":"2024-04-25T05:37:42.886222+0000","last_fullsized":"2024-04-25T05:37:42.886222+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884061+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:37:42.884061+0000","last_peered":"2024-04-25T05:37:42.884061+0000","last_clean":"2024-04-25T05:37:42.884061+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:37:42.884061+0000","last_undegraded":"2024-04-25T05:37:42.884061+0000","last_fullsized":"2024-04-25T05:37:42.884061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874644+0000","last_change":"2024-04-25T05:36:54.240953+0000","last_active":"2024-04-25T05:37:42.874644+0000","last_peered":"2024-04-25T05:37:42.874644+0000","last_clean":"2024-04-25T05:37:42.874644+0000","last_became_active":"2024-04-25T05:36:54.235617+0000","last_became_peered":"2024-04-25T05:36:54.235617+0000","last_unstale":"2024-04-25T05:37:42.874644+0000","last_undegraded":"2024-04-25T05:37:42.874644+0000","last_fullsized":"2024-04-25T05:37:42.874644+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874634+0000","last_change":"2024-04-25T05:36:54.258480+0000","last_active":"2024-04-25T05:37:42.874634+0000","last_peered":"2024-04-25T05:37:42.874634+0000","last_clean":"2024-04-25T05:37:42.874634+0000","last_became_active":"2024-04-25T05:36:54.236292+0000","last_became_peered":"2024-04-25T05:36:54.236292+0000","last_unstale":"2024-04-25T05:37:42.874634+0000","last_undegraded":"2024-04-25T05:37:42.874634+0000","last_fullsized":"2024-04-25T05:37:42.874634+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880154+0000","last_change":"2024-04-25T05:36:30.109005+0000","last_active":"2024-04-25T05:37:42.880154+0000","last_peered":"2024-04-25T05:37:42.880154+0000","last_clean":"2024-04-25T05:37:42.880154+0000","last_became_active":"2024-04-25T05:34:04.724310+0000","last_became_peered":"2024-04-25T05:34:04.724310+0000","last_unstale":"2024-04-25T05:37:42.880154+0000","last_undegraded":"2024-04-25T05:37:42.880154+0000","last_fullsized":"2024-04-25T05:37:42.880154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":228,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916273+0000","last_change":"2024-04-25T05:35:39.823582+0000","last_active":"2024-04-25T05:37:42.916273+0000","last_peered":"2024-04-25T05:37:42.916273+0000","last_clean":"2024-04-25T05:37:42.916273+0000","last_became_active":"2024-04-25T05:34:02.708168+0000","last_became_peered":"2024-04-25T05:34:02.708168+0000","last_unstale":"2024-04-25T05:37:42.916273+0000","last_undegraded":"2024-04-25T05:37:42.916273+0000","last_fullsized":"2024-04-25T05:37:42.916273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:39.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.823449+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907444+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:37:42.907444+0000","last_peered":"2024-04-25T05:37:42.907444+0000","last_clean":"2024-04-25T05:37:42.907444+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:37:42.907444+0000","last_undegraded":"2024-04-25T05:37:42.907444+0000","last_fullsized":"2024-04-25T05:37:42.907444+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:51.933293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":217,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887229+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:37:42.887229+0000","last_peered":"2024-04-25T05:37:42.887229+0000","last_clean":"2024-04-25T05:37:42.887229+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:37:42.887229+0000","last_undegraded":"2024-04-25T05:37:42.887229+0000","last_fullsized":"2024-04-25T05:37:42.887229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.719798+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":169,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.875201+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:37:42.875201+0000","last_peered":"2024-04-25T05:37:42.875201+0000","last_clean":"2024-04-25T05:37:42.875201+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:37:42.875201+0000","last_undegraded":"2024-04-25T05:37:42.875201+0000","last_fullsized":"2024-04-25T05:37:42.875201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.179078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":219,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887623+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:37:42.887623+0000","last_peered":"2024-04-25T05:37:42.887623+0000","last_clean":"2024-04-25T05:37:42.887623+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:37:42.887623+0000","last_undegraded":"2024-04-25T05:37:42.887623+0000","last_fullsized":"2024-04-25T05:37:42.887623+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.878268+0000","last_change":"2024-04-25T05:36:31.118006+0000","last_active":"2024-04-25T05:37:42.878268+0000","last_peered":"2024-04-25T05:37:42.878268+0000","last_clean":"2024-04-25T05:37:42.878268+0000","last_became_active":"2024-04-25T05:33:58.685454+0000","last_became_peered":"2024-04-25T05:33:58.685454+0000","last_unstale":"2024-04-25T05:37:42.878268+0000","last_undegraded":"2024-04-25T05:37:42.878268+0000","last_fullsized":"2024-04-25T05:37:42.878268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.339192+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916599+0000","last_change":"2024-04-25T05:35:40.845196+0000","last_active":"2024-04-25T05:37:42.916599+0000","last_peered":"2024-04-25T05:37:42.916599+0000","last_clean":"2024-04-25T05:37:42.916599+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:37:42.916599+0000","last_undegraded":"2024-04-25T05:37:42.916599+0000","last_fullsized":"2024-04-25T05:37:42.916599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.845036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":169,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.882252+0000","last_change":"2024-04-25T05:35:30.673162+0000","last_active":"2024-04-25T05:37:42.882252+0000","last_peered":"2024-04-25T05:37:42.882252+0000","last_clean":"2024-04-25T05:37:42.882252+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:37:42.882252+0000","last_undegraded":"2024-04-25T05:37:42.882252+0000","last_fullsized":"2024-04-25T05:37:42.882252+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:30.673041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.673041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":166,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887293+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:37:42.887293+0000","last_peered":"2024-04-25T05:37:42.887293+0000","last_clean":"2024-04-25T05:37:42.887293+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:37:42.887293+0000","last_undegraded":"2024-04-25T05:37:42.887293+0000","last_fullsized":"2024-04-25T05:37:42.887293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907328+0000","last_change":"2024-04-25T05:37:01.599400+0000","last_active":"2024-04-25T05:37:42.907328+0000","last_peered":"2024-04-25T05:37:42.907328+0000","last_clean":"2024-04-25T05:37:42.907328+0000","last_became_active":"2024-04-25T05:36:54.240109+0000","last_became_peered":"2024-04-25T05:36:54.240109+0000","last_unstale":"2024-04-25T05:37:42.907328+0000","last_undegraded":"2024-04-25T05:37:42.907328+0000","last_fullsized":"2024-04-25T05:37:42.907328+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:02.962125+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":218,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880836+0000","last_change":"2024-04-25T05:36:20.155217+0000","last_active":"2024-04-25T05:37:42.880836+0000","last_peered":"2024-04-25T05:37:42.880836+0000","last_clean":"2024-04-25T05:37:42.880836+0000","last_became_active":"2024-04-25T05:34:01.001763+0000","last_became_peered":"2024-04-25T05:34:01.001763+0000","last_unstale":"2024-04-25T05:37:42.880836+0000","last_undegraded":"2024-04-25T05:37:42.880836+0000","last_fullsized":"2024-04-25T05:37:42.880836+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907803+0000","last_change":"2024-04-25T05:37:41.064307+0000","last_active":"2024-04-25T05:37:42.907803+0000","last_peered":"2024-04-25T05:37:42.907803+0000","last_clean":"2024-04-25T05:37:42.907803+0000","last_became_active":"2024-04-25T05:36:54.232503+0000","last_became_peered":"2024-04-25T05:36:54.232503+0000","last_unstale":"2024-04-25T05:37:42.907803+0000","last_undegraded":"2024-04-25T05:37:42.907803+0000","last_fullsized":"2024-04-25T05:37:42.907803+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:59.335848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916486+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:37:42.916486+0000","last_peered":"2024-04-25T05:37:42.916486+0000","last_clean":"2024-04-25T05:37:42.916486+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:37:42.916486+0000","last_undegraded":"2024-04-25T05:37:42.916486+0000","last_fullsized":"2024-04-25T05:37:42.916486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.303715+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":49,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877268+0000","last_change":"2024-04-25T05:36:54.239000+0000","last_active":"2024-04-25T05:37:42.877268+0000","last_peered":"2024-04-25T05:37:42.877268+0000","last_clean":"2024-04-25T05:37:42.877268+0000","last_became_active":"2024-04-25T05:36:54.238366+0000","last_became_peered":"2024-04-25T05:36:54.238366+0000","last_unstale":"2024-04-25T05:37:42.877268+0000","last_undegraded":"2024-04-25T05:37:42.877268+0000","last_fullsized":"2024-04-25T05:37:42.877268+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.715360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.715360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887568+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:37:42.887568+0000","last_peered":"2024-04-25T05:37:42.887568+0000","last_clean":"2024-04-25T05:37:42.887568+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:37:42.887568+0000","last_undegraded":"2024-04-25T05:37:42.887568+0000","last_fullsized":"2024-04-25T05:37:42.887568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"148'9914","reported_seq":19752,"reported_epoch":148,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:42.919587+0000","last_change":"2024-04-25T05:36:55.339375+0000","last_active":"2024-04-25T05:37:42.919587+0000","last_peered":"2024-04-25T05:37:42.919587+0000","last_clean":"2024-04-25T05:36:53.210632+0000","last_became_active":"2024-04-25T05:36:55.308452+0000","last_became_peered":"2024-04-25T05:36:55.308452+0000","last_unstale":"2024-04-25T05:37:42.919587+0000","last_undegraded":"2024-04-25T05:37:42.919587+0000","last_fullsized":"2024-04-25T05:37:42.919587+0000","mapping_epoch":130,"log_start":"115'5744","ondisk_log_start":"115'5744","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'5609","last_scrub_stamp":"2024-04-25T05:36:18.461345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.461345+0000","objects_scrubbed":5551,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294387712,"num_objects":4492,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":529,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9914,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4819,"num_bytes_recovered":315817984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":4492}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.915802+0000","last_change":"2024-04-25T05:36:55.047093+0000","last_active":"2024-04-25T05:37:42.915802+0000","last_peered":"2024-04-25T05:37:42.915802+0000","last_clean":"2024-04-25T05:37:42.915802+0000","last_became_active":"2024-04-25T05:36:54.237865+0000","last_became_peered":"2024-04-25T05:36:54.237865+0000","last_unstale":"2024-04-25T05:37:42.915802+0000","last_undegraded":"2024-04-25T05:37:42.915802+0000","last_fullsized":"2024-04-25T05:37:42.915802+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:21.827547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":202,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.908593+0000","last_change":"2024-04-25T05:37:36.045417+0000","last_active":"2024-04-25T05:37:42.908593+0000","last_peered":"2024-04-25T05:37:42.908593+0000","last_clean":"2024-04-25T05:37:42.908593+0000","last_became_active":"2024-04-25T05:33:58.688760+0000","last_became_peered":"2024-04-25T05:33:58.688760+0000","last_unstale":"2024-04-25T05:37:42.908593+0000","last_undegraded":"2024-04-25T05:37:42.908593+0000","last_fullsized":"2024-04-25T05:37:42.908593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:02.905711+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":253,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.915319+0000","last_change":"2024-04-25T05:36:54.267732+0000","last_active":"2024-04-25T05:37:42.915319+0000","last_peered":"2024-04-25T05:37:42.915319+0000","last_clean":"2024-04-25T05:37:42.915319+0000","last_became_active":"2024-04-25T05:36:54.244247+0000","last_became_peered":"2024-04-25T05:36:54.244247+0000","last_unstale":"2024-04-25T05:37:42.915319+0000","last_undegraded":"2024-04-25T05:37:42.915319+0000","last_fullsized":"2024-04-25T05:37:42.915319+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:45.257419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"130'10","reported_seq":247,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907519+0000","last_change":"2024-04-25T05:36:54.243037+0000","last_active":"2024-04-25T05:37:42.907519+0000","last_peered":"2024-04-25T05:37:42.907519+0000","last_clean":"2024-04-25T05:37:42.907519+0000","last_became_active":"2024-04-25T05:36:54.235542+0000","last_became_peered":"2024-04-25T05:36:54.235542+0000","last_unstale":"2024-04-25T05:37:42.907519+0000","last_undegraded":"2024-04-25T05:37:42.907519+0000","last_fullsized":"2024-04-25T05:37:42.907519+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:44.322822+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"147'9993","reported_seq":10124,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884328+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:37:42.884328+0000","last_peered":"2024-04-25T05:37:42.884328+0000","last_clean":"2024-04-25T05:37:42.884328+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:37:42.884328+0000","last_undegraded":"2024-04-25T05:37:42.884328+0000","last_fullsized":"2024-04-25T05:37:42.884328+0000","mapping_epoch":88,"log_start":"116'5861","ondisk_log_start":"116'5861","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304021504,"num_objects":4639,"num_object_clones":0,"num_object_copies":13917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":468224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":162,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885713+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:37:42.885713+0000","last_peered":"2024-04-25T05:37:42.885713+0000","last_clean":"2024-04-25T05:37:42.885713+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:37:42.885713+0000","last_undegraded":"2024-04-25T05:37:42.885713+0000","last_fullsized":"2024-04-25T05:37:42.885713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907956+0000","last_change":"2024-04-25T05:36:48.522033+0000","last_active":"2024-04-25T05:37:42.907956+0000","last_peered":"2024-04-25T05:37:42.907956+0000","last_clean":"2024-04-25T05:37:42.907956+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:37:42.907956+0000","last_undegraded":"2024-04-25T05:37:42.907956+0000","last_fullsized":"2024-04-25T05:37:42.907956+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:48.521908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:48.521908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:53.728133+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":224,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874451+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:37:42.874451+0000","last_peered":"2024-04-25T05:37:42.874451+0000","last_clean":"2024-04-25T05:37:42.874451+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:37:42.874451+0000","last_undegraded":"2024-04-25T05:37:42.874451+0000","last_fullsized":"2024-04-25T05:37:42.874451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881379+0000","last_change":"2024-04-25T05:36:47.489355+0000","last_active":"2024-04-25T05:37:42.881379+0000","last_peered":"2024-04-25T05:37:42.881379+0000","last_clean":"2024-04-25T05:37:42.881379+0000","last_became_active":"2024-04-25T05:34:02.710017+0000","last_became_peered":"2024-04-25T05:34:02.710017+0000","last_unstale":"2024-04-25T05:37:42.881379+0000","last_undegraded":"2024-04-25T05:37:42.881379+0000","last_fullsized":"2024-04-25T05:37:42.881379+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:47.489244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.489244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.358175+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"148'9963","reported_seq":10171,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:43.014571+0000","last_change":"2024-04-25T05:36:54.004498+0000","last_active":"2024-04-25T05:37:43.014571+0000","last_peered":"2024-04-25T05:37:43.014571+0000","last_clean":"2024-04-25T05:37:43.014571+0000","last_became_active":"2024-04-25T05:35:09.681316+0000","last_became_peered":"2024-04-25T05:35:09.681316+0000","last_unstale":"2024-04-25T05:37:43.014571+0000","last_undegraded":"2024-04-25T05:37:43.014571+0000","last_fullsized":"2024-04-25T05:37:43.014571+0000","mapping_epoch":88,"log_start":"111'5176","ondisk_log_start":"111'5176","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7637","last_scrub_stamp":"2024-04-25T05:36:54.004340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.004340+0000","objects_scrubbed":6885,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.098402+0000","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298778624,"num_objects":4559,"num_object_clones":0,"num_object_copies":13677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9963,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907200+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:37:42.907200+0000","last_peered":"2024-04-25T05:37:42.907200+0000","last_clean":"2024-04-25T05:37:42.907200+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:37:42.907200+0000","last_undegraded":"2024-04-25T05:37:42.907200+0000","last_fullsized":"2024-04-25T05:37:42.907200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:51.897023+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881548+0000","last_change":"2024-04-25T05:36:51.574009+0000","last_active":"2024-04-25T05:37:42.881548+0000","last_peered":"2024-04-25T05:37:42.881548+0000","last_clean":"2024-04-25T05:37:42.881548+0000","last_became_active":"2024-04-25T05:34:02.710011+0000","last_became_peered":"2024-04-25T05:34:02.710011+0000","last_unstale":"2024-04-25T05:37:42.881548+0000","last_undegraded":"2024-04-25T05:37:42.881548+0000","last_fullsized":"2024-04-25T05:37:42.881548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:51.573898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.573898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.269140+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.908002+0000","last_change":"2024-04-25T05:36:54.240931+0000","last_active":"2024-04-25T05:37:42.908002+0000","last_peered":"2024-04-25T05:37:42.908002+0000","last_clean":"2024-04-25T05:37:42.908002+0000","last_became_active":"2024-04-25T05:36:54.232602+0000","last_became_peered":"2024-04-25T05:36:54.232602+0000","last_unstale":"2024-04-25T05:37:42.908002+0000","last_undegraded":"2024-04-25T05:37:42.908002+0000","last_fullsized":"2024-04-25T05:37:42.908002+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.130398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":132,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883901+0000","last_change":"2024-04-25T05:36:54.260249+0000","last_active":"2024-04-25T05:37:42.883901+0000","last_peered":"2024-04-25T05:37:42.883901+0000","last_clean":"2024-04-25T05:37:42.883901+0000","last_became_active":"2024-04-25T05:36:54.235633+0000","last_became_peered":"2024-04-25T05:36:54.235633+0000","last_unstale":"2024-04-25T05:37:42.883901+0000","last_undegraded":"2024-04-25T05:37:42.883901+0000","last_fullsized":"2024-04-25T05:37:42.883901+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"147'9898","reported_seq":10099,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881078+0000","last_change":"2024-04-25T05:36:31.742804+0000","last_active":"2024-04-25T05:37:42.881078+0000","last_peered":"2024-04-25T05:37:42.881078+0000","last_clean":"2024-04-25T05:37:42.881078+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:37:42.881078+0000","last_undegraded":"2024-04-25T05:37:42.881078+0000","last_fullsized":"2024-04-25T05:37:42.881078+0000","mapping_epoch":88,"log_start":"106'4271","ondisk_log_start":"106'4271","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6352","last_scrub_stamp":"2024-04-25T05:36:31.742451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.742451+0000","objects_scrubbed":6298,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:58.047202+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9898,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877765+0000","last_change":"2024-04-25T05:36:27.172930+0000","last_active":"2024-04-25T05:37:42.877765+0000","last_peered":"2024-04-25T05:37:42.877765+0000","last_clean":"2024-04-25T05:37:42.877765+0000","last_became_active":"2024-04-25T05:34:04.725528+0000","last_became_peered":"2024-04-25T05:34:04.725528+0000","last_unstale":"2024-04-25T05:37:42.877765+0000","last_undegraded":"2024-04-25T05:37:42.877765+0000","last_fullsized":"2024-04-25T05:37:42.877765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:27.172809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.172809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.542287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.882262+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:37:42.882262+0000","last_peered":"2024-04-25T05:37:42.882262+0000","last_clean":"2024-04-25T05:37:42.882262+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:37:42.882262+0000","last_undegraded":"2024-04-25T05:37:42.882262+0000","last_fullsized":"2024-04-25T05:37:42.882262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":196,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916128+0000","last_change":"2024-04-25T05:37:30.237718+0000","last_active":"2024-04-25T05:37:42.916128+0000","last_peered":"2024-04-25T05:37:42.916128+0000","last_clean":"2024-04-25T05:37:42.916128+0000","last_became_active":"2024-04-25T05:36:54.234986+0000","last_became_peered":"2024-04-25T05:36:54.234986+0000","last_unstale":"2024-04-25T05:37:42.916128+0000","last_undegraded":"2024-04-25T05:37:42.916128+0000","last_fullsized":"2024-04-25T05:37:42.916128+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:46.447267+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881266+0000","last_change":"2024-04-25T05:35:40.352567+0000","last_active":"2024-04-25T05:37:42.881266+0000","last_peered":"2024-04-25T05:37:42.881266+0000","last_clean":"2024-04-25T05:37:42.881266+0000","last_became_active":"2024-04-25T05:34:00.696069+0000","last_became_peered":"2024-04-25T05:34:00.696069+0000","last_unstale":"2024-04-25T05:37:42.881266+0000","last_undegraded":"2024-04-25T05:37:42.881266+0000","last_fullsized":"2024-04-25T05:37:42.881266+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"148'10032","reported_seq":18284,"reported_epoch":148,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:43.012067+0000","last_change":"2024-04-25T05:36:55.428129+0000","last_active":"2024-04-25T05:37:43.012067+0000","last_peered":"2024-04-25T05:37:43.012067+0000","last_clean":"2024-04-25T05:36:53.167583+0000","last_became_active":"2024-04-25T05:36:55.354107+0000","last_became_peered":"2024-04-25T05:36:55.354107+0000","last_unstale":"2024-04-25T05:37:43.012067+0000","last_undegraded":"2024-04-25T05:37:43.012067+0000","last_fullsized":"2024-04-25T05:37:43.012067+0000","mapping_epoch":130,"log_start":"111'5239","ondisk_log_start":"111'5239","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312344576,"num_objects":4766,"num_object_clones":0,"num_object_copies":14298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1385,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10032,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4057,"num_bytes_recovered":265814016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":4766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887362+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:37:42.887362+0000","last_peered":"2024-04-25T05:37:42.887362+0000","last_clean":"2024-04-25T05:37:42.887362+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:37:42.887362+0000","last_undegraded":"2024-04-25T05:37:42.887362+0000","last_fullsized":"2024-04-25T05:37:42.887362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:52.505234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":202,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.915647+0000","last_change":"2024-04-25T05:36:54.257218+0000","last_active":"2024-04-25T05:37:42.915647+0000","last_peered":"2024-04-25T05:37:42.915647+0000","last_clean":"2024-04-25T05:37:42.915647+0000","last_became_active":"2024-04-25T05:36:54.238031+0000","last_became_peered":"2024-04-25T05:36:54.238031+0000","last_unstale":"2024-04-25T05:37:42.915647+0000","last_undegraded":"2024-04-25T05:37:42.915647+0000","last_fullsized":"2024-04-25T05:37:42.915647+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:49.195800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:43.613462+0000","last_change":"2024-04-25T05:37:43.613485+0000","last_active":"2024-04-25T05:37:43.613462+0000","last_peered":"2024-04-25T05:37:43.613462+0000","last_clean":"2024-04-25T05:37:43.613462+0000","last_became_active":"2024-04-25T05:36:54.244233+0000","last_became_peered":"2024-04-25T05:36:54.244233+0000","last_unstale":"2024-04-25T05:37:43.613462+0000","last_undegraded":"2024-04-25T05:37:43.613462+0000","last_fullsized":"2024-04-25T05:37:43.613462+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:37:43.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:06.774207+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":294,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883647+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:37:42.883647+0000","last_peered":"2024-04-25T05:37:42.883647+0000","last_clean":"2024-04-25T05:37:42.883647+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:37:42.883647+0000","last_undegraded":"2024-04-25T05:37:42.883647+0000","last_fullsized":"2024-04-25T05:37:42.883647+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":174,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.879704+0000","last_change":"2024-04-25T05:36:24.214089+0000","last_active":"2024-04-25T05:37:42.879704+0000","last_peered":"2024-04-25T05:37:42.879704+0000","last_clean":"2024-04-25T05:37:42.879704+0000","last_became_active":"2024-04-25T05:34:02.707970+0000","last_became_peered":"2024-04-25T05:34:02.707970+0000","last_unstale":"2024-04-25T05:37:42.879704+0000","last_undegraded":"2024-04-25T05:37:42.879704+0000","last_fullsized":"2024-04-25T05:37:42.879704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"148'9900","reported_seq":12592,"reported_epoch":148,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:43.026188+0000","last_change":"2024-04-25T05:37:25.950575+0000","last_active":"2024-04-25T05:37:43.026188+0000","last_peered":"2024-04-25T05:37:43.026188+0000","last_clean":"2024-04-25T05:37:24.079121+0000","last_became_active":"2024-04-25T05:37:25.849435+0000","last_became_peered":"2024-04-25T05:37:25.849435+0000","last_unstale":"2024-04-25T05:37:43.026188+0000","last_undegraded":"2024-04-25T05:37:43.026188+0000","last_fullsized":"2024-04-25T05:37:43.026188+0000","mapping_epoch":140,"log_start":"111'5099","ondisk_log_start":"111'5099","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302776320,"num_objects":4620,"num_object_clones":0,"num_object_copies":13860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3408,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1276,"num_bytes_recovered":83558400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":4620}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.908124+0000","last_change":"2024-04-25T05:36:54.626908+0000","last_active":"2024-04-25T05:37:42.908124+0000","last_peered":"2024-04-25T05:37:42.908124+0000","last_clean":"2024-04-25T05:37:42.908124+0000","last_became_active":"2024-04-25T05:36:54.225394+0000","last_became_peered":"2024-04-25T05:36:54.225394+0000","last_unstale":"2024-04-25T05:37:42.908124+0000","last_undegraded":"2024-04-25T05:37:42.908124+0000","last_fullsized":"2024-04-25T05:37:42.908124+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.511619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883474+0000","last_change":"2024-04-25T05:36:54.240763+0000","last_active":"2024-04-25T05:37:42.883474+0000","last_peered":"2024-04-25T05:37:42.883474+0000","last_clean":"2024-04-25T05:37:42.883474+0000","last_became_active":"2024-04-25T05:36:54.240014+0000","last_became_peered":"2024-04-25T05:36:54.240014+0000","last_unstale":"2024-04-25T05:37:42.883474+0000","last_undegraded":"2024-04-25T05:37:42.883474+0000","last_fullsized":"2024-04-25T05:37:42.883474+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:40.339655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.339655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881674+0000","last_change":"2024-04-25T05:36:16.674171+0000","last_active":"2024-04-25T05:37:42.881674+0000","last_peered":"2024-04-25T05:37:42.881674+0000","last_clean":"2024-04-25T05:37:42.881674+0000","last_became_active":"2024-04-25T05:33:57.668594+0000","last_became_peered":"2024-04-25T05:33:57.668594+0000","last_unstale":"2024-04-25T05:37:42.881674+0000","last_undegraded":"2024-04-25T05:37:42.881674+0000","last_fullsized":"2024-04-25T05:37:42.881674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.882960+0000","last_change":"2024-04-25T05:35:39.312001+0000","last_active":"2024-04-25T05:37:42.882960+0000","last_peered":"2024-04-25T05:37:42.882960+0000","last_clean":"2024-04-25T05:37:42.882960+0000","last_became_active":"2024-04-25T05:34:00.693844+0000","last_became_peered":"2024-04-25T05:34:00.693844+0000","last_unstale":"2024-04-25T05:37:42.882960+0000","last_undegraded":"2024-04-25T05:37:42.882960+0000","last_fullsized":"2024-04-25T05:37:42.882960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881740+0000","last_change":"2024-04-25T05:35:42.672247+0000","last_active":"2024-04-25T05:37:42.881740+0000","last_peered":"2024-04-25T05:37:42.881740+0000","last_clean":"2024-04-25T05:37:42.881740+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:37:42.881740+0000","last_undegraded":"2024-04-25T05:37:42.881740+0000","last_fullsized":"2024-04-25T05:37:42.881740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.672114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.672114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"148'9820","reported_seq":9951,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.998084+0000","last_change":"2024-04-25T05:35:09.683951+0000","last_active":"2024-04-25T05:37:42.998084+0000","last_peered":"2024-04-25T05:37:42.998084+0000","last_clean":"2024-04-25T05:37:42.998084+0000","last_became_active":"2024-04-25T05:35:09.682815+0000","last_became_peered":"2024-04-25T05:35:09.682815+0000","last_unstale":"2024-04-25T05:37:42.998084+0000","last_undegraded":"2024-04-25T05:37:42.998084+0000","last_fullsized":"2024-04-25T05:37:42.998084+0000","mapping_epoch":88,"log_start":"110'4967","ondisk_log_start":"110'4967","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304742400,"num_objects":4650,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916586+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:37:42.916586+0000","last_peered":"2024-04-25T05:37:42.916586+0000","last_clean":"2024-04-25T05:37:42.916586+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:37:42.916586+0000","last_undegraded":"2024-04-25T05:37:42.916586+0000","last_fullsized":"2024-04-25T05:37:42.916586+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":448,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907313+0000","last_change":"2024-04-25T05:36:57.317664+0000","last_active":"2024-04-25T05:37:42.907313+0000","last_peered":"2024-04-25T05:37:42.907313+0000","last_clean":"2024-04-25T05:37:42.907313+0000","last_became_active":"2024-04-25T05:36:57.317332+0000","last_became_peered":"2024-04-25T05:36:57.317332+0000","last_unstale":"2024-04-25T05:37:42.907313+0000","last_undegraded":"2024-04-25T05:37:42.907313+0000","last_fullsized":"2024-04-25T05:37:42.907313+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:36:28.364006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.364006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.795292+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884188+0000","last_change":"2024-04-25T05:36:54.233589+0000","last_active":"2024-04-25T05:37:42.884188+0000","last_peered":"2024-04-25T05:37:42.884188+0000","last_clean":"2024-04-25T05:37:42.884188+0000","last_became_active":"2024-04-25T05:36:54.232305+0000","last_became_peered":"2024-04-25T05:36:54.232305+0000","last_unstale":"2024-04-25T05:37:42.884188+0000","last_undegraded":"2024-04-25T05:37:42.884188+0000","last_fullsized":"2024-04-25T05:37:42.884188+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:54.215324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"130'17","reported_seq":120,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874855+0000","last_change":"2024-04-25T05:36:54.268339+0000","last_active":"2024-04-25T05:37:42.874855+0000","last_peered":"2024-04-25T05:37:42.874855+0000","last_clean":"2024-04-25T05:37:42.874855+0000","last_became_active":"2024-04-25T05:36:54.245169+0000","last_became_peered":"2024-04-25T05:36:54.245169+0000","last_unstale":"2024-04-25T05:37:42.874855+0000","last_undegraded":"2024-04-25T05:37:42.874855+0000","last_fullsized":"2024-04-25T05:37:42.874855+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:27.678020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.678020+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.908034+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:37:42.908034+0000","last_peered":"2024-04-25T05:37:42.908034+0000","last_clean":"2024-04-25T05:37:42.908034+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:37:42.908034+0000","last_undegraded":"2024-04-25T05:37:42.908034+0000","last_fullsized":"2024-04-25T05:37:42.908034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:31.527148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"148'10163","reported_seq":10372,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:47.689144+0000","last_change":"2024-04-25T05:36:51.500685+0000","last_active":"2024-04-25T05:37:47.689144+0000","last_peered":"2024-04-25T05:37:47.689144+0000","last_clean":"2024-04-25T05:37:47.689144+0000","last_became_active":"2024-04-25T05:35:09.683250+0000","last_became_peered":"2024-04-25T05:35:09.683250+0000","last_unstale":"2024-04-25T05:37:47.689144+0000","last_undegraded":"2024-04-25T05:37:47.689144+0000","last_fullsized":"2024-04-25T05:37:47.689144+0000","mapping_epoch":88,"log_start":"111'5063","ondisk_log_start":"111'5063","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7574","last_scrub_stamp":"2024-04-25T05:36:51.500427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.500427+0000","objects_scrubbed":7061,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:06.145379+0000","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289210368,"num_objects":4413,"num_object_clones":0,"num_object_copies":13239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883367+0000","last_change":"2024-04-25T05:35:38.262254+0000","last_active":"2024-04-25T05:37:42.883367+0000","last_peered":"2024-04-25T05:37:42.883367+0000","last_clean":"2024-04-25T05:37:42.883367+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:37:42.883367+0000","last_undegraded":"2024-04-25T05:37:42.883367+0000","last_fullsized":"2024-04-25T05:37:42.883367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.261996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.261996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.877543+0000","last_change":"2024-04-25T05:36:26.153204+0000","last_active":"2024-04-25T05:37:42.877543+0000","last_peered":"2024-04-25T05:37:42.877543+0000","last_clean":"2024-04-25T05:37:42.877543+0000","last_became_active":"2024-04-25T05:33:58.682140+0000","last_became_peered":"2024-04-25T05:33:58.682140+0000","last_unstale":"2024-04-25T05:37:42.877543+0000","last_undegraded":"2024-04-25T05:37:42.877543+0000","last_fullsized":"2024-04-25T05:37:42.877543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":424,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:46.016305+0000","last_change":"2024-04-25T05:37:44.995167+0000","last_active":"2024-04-25T05:37:46.016305+0000","last_peered":"2024-04-25T05:37:46.016305+0000","last_clean":"2024-04-25T05:37:46.016305+0000","last_became_active":"2024-04-25T05:33:57.670343+0000","last_became_peered":"2024-04-25T05:33:57.670343+0000","last_unstale":"2024-04-25T05:37:46.016305+0000","last_undegraded":"2024-04-25T05:37:46.016305+0000","last_fullsized":"2024-04-25T05:37:46.016305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:03.155432+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":223,"num_read_kb":218,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880251+0000","last_change":"2024-04-25T05:36:32.152204+0000","last_active":"2024-04-25T05:37:42.880251+0000","last_peered":"2024-04-25T05:37:42.880251+0000","last_clean":"2024-04-25T05:37:42.880251+0000","last_became_active":"2024-04-25T05:34:00.693805+0000","last_became_peered":"2024-04-25T05:34:00.693805+0000","last_unstale":"2024-04-25T05:37:42.880251+0000","last_undegraded":"2024-04-25T05:37:42.880251+0000","last_fullsized":"2024-04-25T05:37:42.880251+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916828+0000","last_change":"2024-04-25T05:37:07.166910+0000","last_active":"2024-04-25T05:37:42.916828+0000","last_peered":"2024-04-25T05:37:42.916828+0000","last_clean":"2024-04-25T05:37:42.916828+0000","last_became_active":"2024-04-25T05:34:02.709037+0000","last_became_peered":"2024-04-25T05:34:02.709037+0000","last_unstale":"2024-04-25T05:37:42.916828+0000","last_undegraded":"2024-04-25T05:37:42.916828+0000","last_fullsized":"2024-04-25T05:37:42.916828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:36.034711+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"148'10231","reported_seq":10441,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:47.676714+0000","last_change":"2024-04-25T05:36:51.881839+0000","last_active":"2024-04-25T05:37:47.676714+0000","last_peered":"2024-04-25T05:37:47.676714+0000","last_clean":"2024-04-25T05:37:47.676714+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:37:47.676714+0000","last_undegraded":"2024-04-25T05:37:47.676714+0000","last_fullsized":"2024-04-25T05:37:47.676714+0000","mapping_epoch":88,"log_start":"109'4833","ondisk_log_start":"109'4833","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7619","last_scrub_stamp":"2024-04-25T05:36:51.881693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.881693+0000","objects_scrubbed":7027,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:13.086436+0000","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286588928,"num_objects":4373,"num_object_clones":0,"num_object_copies":13119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":176,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885811+0000","last_change":"2024-04-25T05:36:54.234953+0000","last_active":"2024-04-25T05:37:42.885811+0000","last_peered":"2024-04-25T05:37:42.885811+0000","last_clean":"2024-04-25T05:37:42.885811+0000","last_became_active":"2024-04-25T05:36:54.234602+0000","last_became_peered":"2024-04-25T05:36:54.234602+0000","last_unstale":"2024-04-25T05:37:42.885811+0000","last_undegraded":"2024-04-25T05:37:42.885811+0000","last_fullsized":"2024-04-25T05:37:42.885811+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883561+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:37:42.883561+0000","last_peered":"2024-04-25T05:37:42.883561+0000","last_clean":"2024-04-25T05:37:42.883561+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:37:42.883561+0000","last_undegraded":"2024-04-25T05:37:42.883561+0000","last_fullsized":"2024-04-25T05:37:42.883561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":226,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.878724+0000","last_change":"2024-04-25T05:35:35.249157+0000","last_active":"2024-04-25T05:37:42.878724+0000","last_peered":"2024-04-25T05:37:42.878724+0000","last_clean":"2024-04-25T05:37:42.878724+0000","last_became_active":"2024-04-25T05:34:00.999559+0000","last_became_peered":"2024-04-25T05:34:00.999559+0000","last_unstale":"2024-04-25T05:37:42.878724+0000","last_undegraded":"2024-04-25T05:37:42.878724+0000","last_fullsized":"2024-04-25T05:37:42.878724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:35.248443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.248443+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":49,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.878179+0000","last_change":"2024-04-25T05:36:54.237191+0000","last_active":"2024-04-25T05:37:42.878179+0000","last_peered":"2024-04-25T05:37:42.878179+0000","last_clean":"2024-04-25T05:37:42.878179+0000","last_became_active":"2024-04-25T05:36:54.234985+0000","last_became_peered":"2024-04-25T05:36:54.234985+0000","last_unstale":"2024-04-25T05:37:42.878179+0000","last_undegraded":"2024-04-25T05:37:42.878179+0000","last_fullsized":"2024-04-25T05:37:42.878179+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"148'10318","reported_seq":21888,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:47.673133+0000","last_change":"2024-04-25T05:37:42.904598+0000","last_active":"2024-04-25T05:37:47.673133+0000","last_peered":"2024-04-25T05:37:47.673133+0000","last_clean":"2024-04-25T05:37:47.673133+0000","last_became_active":"2024-04-25T05:37:42.889210+0000","last_became_peered":"2024-04-25T05:37:42.889210+0000","last_unstale":"2024-04-25T05:37:47.673133+0000","last_undegraded":"2024-04-25T05:37:47.673133+0000","last_fullsized":"2024-04-25T05:37:47.673133+0000","mapping_epoch":147,"log_start":"113'5767","ondisk_log_start":"113'5767","created":88,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293208064,"num_objects":4474,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10318,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":374472704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":170,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.876823+0000","last_change":"2024-04-25T05:35:44.317463+0000","last_active":"2024-04-25T05:37:42.876823+0000","last_peered":"2024-04-25T05:37:42.876823+0000","last_clean":"2024-04-25T05:37:42.876823+0000","last_became_active":"2024-04-25T05:34:04.728318+0000","last_became_peered":"2024-04-25T05:34:04.728318+0000","last_unstale":"2024-04-25T05:37:42.876823+0000","last_undegraded":"2024-04-25T05:37:42.876823+0000","last_fullsized":"2024-04-25T05:37:42.876823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916462+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:37:42.916462+0000","last_peered":"2024-04-25T05:37:42.916462+0000","last_clean":"2024-04-25T05:37:42.916462+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:37:42.916462+0000","last_undegraded":"2024-04-25T05:37:42.916462+0000","last_fullsized":"2024-04-25T05:37:42.916462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":136,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881978+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:37:42.881978+0000","last_peered":"2024-04-25T05:37:42.881978+0000","last_clean":"2024-04-25T05:37:42.881978+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:37:42.881978+0000","last_undegraded":"2024-04-25T05:37:42.881978+0000","last_fullsized":"2024-04-25T05:37:42.881978+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"130'9","reported_seq":237,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881365+0000","last_change":"2024-04-25T05:36:54.253353+0000","last_active":"2024-04-25T05:37:42.881365+0000","last_peered":"2024-04-25T05:37:42.881365+0000","last_clean":"2024-04-25T05:37:42.881365+0000","last_became_active":"2024-04-25T05:36:54.238623+0000","last_became_peered":"2024-04-25T05:36:54.238623+0000","last_unstale":"2024-04-25T05:37:42.881365+0000","last_undegraded":"2024-04-25T05:37:42.881365+0000","last_fullsized":"2024-04-25T05:37:42.881365+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:42.311077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.311077+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"147'9952","reported_seq":10083,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884028+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:37:42.884028+0000","last_peered":"2024-04-25T05:37:42.884028+0000","last_clean":"2024-04-25T05:37:42.884028+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:37:42.884028+0000","last_undegraded":"2024-04-25T05:37:42.884028+0000","last_fullsized":"2024-04-25T05:37:42.884028+0000","mapping_epoch":88,"log_start":"116'5827","ondisk_log_start":"116'5827","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308281344,"num_objects":4704,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.907668+0000","last_change":"2024-04-25T05:36:59.627764+0000","last_active":"2024-04-25T05:37:42.907668+0000","last_peered":"2024-04-25T05:37:42.907668+0000","last_clean":"2024-04-25T05:37:42.907668+0000","last_became_active":"2024-04-25T05:36:54.234860+0000","last_became_peered":"2024-04-25T05:36:54.234860+0000","last_unstale":"2024-04-25T05:37:42.907668+0000","last_undegraded":"2024-04-25T05:37:42.907668+0000","last_fullsized":"2024-04-25T05:37:42.907668+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:26.357457+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.880347+0000","last_change":"2024-04-25T05:35:36.294561+0000","last_active":"2024-04-25T05:37:42.880347+0000","last_peered":"2024-04-25T05:37:42.880347+0000","last_clean":"2024-04-25T05:37:42.880347+0000","last_became_active":"2024-04-25T05:33:58.682089+0000","last_became_peered":"2024-04-25T05:33:58.682089+0000","last_unstale":"2024-04-25T05:37:42.880347+0000","last_undegraded":"2024-04-25T05:37:42.880347+0000","last_fullsized":"2024-04-25T05:37:42.880347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":235,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874406+0000","last_change":"2024-04-25T05:36:54.273512+0000","last_active":"2024-04-25T05:37:42.874406+0000","last_peered":"2024-04-25T05:37:42.874406+0000","last_clean":"2024-04-25T05:37:42.874406+0000","last_became_active":"2024-04-25T05:36:54.248063+0000","last_became_peered":"2024-04-25T05:36:54.248063+0000","last_unstale":"2024-04-25T05:37:42.874406+0000","last_undegraded":"2024-04-25T05:37:42.874406+0000","last_fullsized":"2024-04-25T05:37:42.874406+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:50.634187+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":227,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886184+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:37:42.886184+0000","last_peered":"2024-04-25T05:37:42.886184+0000","last_clean":"2024-04-25T05:37:42.886184+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:37:42.886184+0000","last_undegraded":"2024-04-25T05:37:42.886184+0000","last_fullsized":"2024-04-25T05:37:42.886184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:43.089867+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"148'9901","reported_seq":10032,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886867+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:37:42.886867+0000","last_peered":"2024-04-25T05:37:42.886867+0000","last_clean":"2024-04-25T05:37:42.886867+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:37:42.886867+0000","last_undegraded":"2024-04-25T05:37:42.886867+0000","last_fullsized":"2024-04-25T05:37:42.886867+0000","mapping_epoch":88,"log_start":"115'5769","ondisk_log_start":"115'5769","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304283648,"num_objects":4643,"num_object_clones":0,"num_object_copies":13929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9901,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":169,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874094+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:37:42.874094+0000","last_peered":"2024-04-25T05:37:42.874094+0000","last_clean":"2024-04-25T05:37:42.874094+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:37:42.874094+0000","last_undegraded":"2024-04-25T05:37:42.874094+0000","last_fullsized":"2024-04-25T05:37:42.874094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887072+0000","last_change":"2024-04-25T05:35:06.377537+0000","last_active":"2024-04-25T05:37:42.887072+0000","last_peered":"2024-04-25T05:37:42.887072+0000","last_clean":"2024-04-25T05:37:42.887072+0000","last_became_active":"2024-04-25T05:33:58.675749+0000","last_became_peered":"2024-04-25T05:33:58.675749+0000","last_unstale":"2024-04-25T05:37:42.887072+0000","last_undegraded":"2024-04-25T05:37:42.887072+0000","last_fullsized":"2024-04-25T05:37:42.887072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":234,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.886083+0000","last_change":"2024-04-25T05:35:30.243679+0000","last_active":"2024-04-25T05:37:42.886083+0000","last_peered":"2024-04-25T05:37:42.886083+0000","last_clean":"2024-04-25T05:37:42.886083+0000","last_became_active":"2024-04-25T05:34:00.999301+0000","last_became_peered":"2024-04-25T05:34:00.999301+0000","last_unstale":"2024-04-25T05:37:42.886083+0000","last_undegraded":"2024-04-25T05:37:42.886083+0000","last_fullsized":"2024-04-25T05:37:42.886083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874026+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:37:42.874026+0000","last_peered":"2024-04-25T05:37:42.874026+0000","last_clean":"2024-04-25T05:37:42.874026+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:37:42.874026+0000","last_undegraded":"2024-04-25T05:37:42.874026+0000","last_fullsized":"2024-04-25T05:37:42.874026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"147'9893","reported_seq":19090,"reported_epoch":148,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:42.938108+0000","last_change":"2024-04-25T05:36:55.410743+0000","last_active":"2024-04-25T05:37:42.938108+0000","last_peered":"2024-04-25T05:37:42.938108+0000","last_clean":"2024-04-25T05:36:53.144949+0000","last_became_active":"2024-04-25T05:36:55.337352+0000","last_became_peered":"2024-04-25T05:36:55.337352+0000","last_unstale":"2024-04-25T05:37:42.938108+0000","last_undegraded":"2024-04-25T05:37:42.938108+0000","last_fullsized":"2024-04-25T05:37:42.938108+0000","mapping_epoch":130,"log_start":"116'5762","ondisk_log_start":"116'5762","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302710784,"num_objects":4619,"num_object_clones":0,"num_object_copies":13857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":865,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":296943616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":4619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916356+0000","last_change":"2024-04-25T05:36:54.001702+0000","last_active":"2024-04-25T05:37:42.916356+0000","last_peered":"2024-04-25T05:37:42.916356+0000","last_clean":"2024-04-25T05:37:42.916356+0000","last_became_active":"2024-04-25T05:34:04.723835+0000","last_became_peered":"2024-04-25T05:34:04.723835+0000","last_unstale":"2024-04-25T05:37:42.916356+0000","last_undegraded":"2024-04-25T05:37:42.916356+0000","last_fullsized":"2024-04-25T05:37:42.916356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.057784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":219,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874984+0000","last_change":"2024-04-25T05:34:02.708836+0000","last_active":"2024-04-25T05:37:42.874984+0000","last_peered":"2024-04-25T05:37:42.874984+0000","last_clean":"2024-04-25T05:37:42.874984+0000","last_became_active":"2024-04-25T05:34:02.708535+0000","last_became_peered":"2024-04-25T05:34:02.708535+0000","last_unstale":"2024-04-25T05:37:42.874984+0000","last_undegraded":"2024-04-25T05:37:42.874984+0000","last_fullsized":"2024-04-25T05:37:42.874984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885990+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:37:42.885990+0000","last_peered":"2024-04-25T05:37:42.885990+0000","last_clean":"2024-04-25T05:37:42.885990+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:37:42.885990+0000","last_undegraded":"2024-04-25T05:37:42.885990+0000","last_fullsized":"2024-04-25T05:37:42.885990+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":203,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.884734+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:37:42.884734+0000","last_peered":"2024-04-25T05:37:42.884734+0000","last_clean":"2024-04-25T05:37:42.884734+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:37:42.884734+0000","last_undegraded":"2024-04-25T05:37:42.884734+0000","last_fullsized":"2024-04-25T05:37:42.884734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"147'10030","reported_seq":19031,"reported_epoch":148,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:37:42.919280+0000","last_change":"2024-04-25T05:36:55.415940+0000","last_active":"2024-04-25T05:37:42.919280+0000","last_peered":"2024-04-25T05:37:42.919280+0000","last_clean":"2024-04-25T05:36:53.182883+0000","last_became_active":"2024-04-25T05:36:55.354455+0000","last_became_peered":"2024-04-25T05:36:55.354455+0000","last_unstale":"2024-04-25T05:37:42.919280+0000","last_undegraded":"2024-04-25T05:37:42.919280+0000","last_fullsized":"2024-04-25T05:37:42.919280+0000","mapping_epoch":130,"log_start":"115'5788","ondisk_log_start":"115'5788","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":4899,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463320,"num_objects":4662,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1003,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10030,"num_write_kb":470081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":288620568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,6],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":4662}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":168,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.887133+0000","last_change":"2024-04-25T05:36:54.232549+0000","last_active":"2024-04-25T05:37:42.887133+0000","last_peered":"2024-04-25T05:37:42.887133+0000","last_clean":"2024-04-25T05:37:42.887133+0000","last_became_active":"2024-04-25T05:36:54.231886+0000","last_became_peered":"2024-04-25T05:36:54.231886+0000","last_unstale":"2024-04-25T05:37:42.887133+0000","last_undegraded":"2024-04-25T05:37:42.887133+0000","last_fullsized":"2024-04-25T05:37:42.887133+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":188,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.916779+0000","last_change":"2024-04-25T05:37:08.197220+0000","last_active":"2024-04-25T05:37:42.916779+0000","last_peered":"2024-04-25T05:37:42.916779+0000","last_clean":"2024-04-25T05:37:42.916779+0000","last_became_active":"2024-04-25T05:36:54.244909+0000","last_became_peered":"2024-04-25T05:36:54.244909+0000","last_unstale":"2024-04-25T05:37:42.916779+0000","last_undegraded":"2024-04-25T05:37:42.916779+0000","last_fullsized":"2024-04-25T05:37:42.916779+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:24.881588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874124+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:37:42.874124+0000","last_peered":"2024-04-25T05:37:42.874124+0000","last_clean":"2024-04-25T05:37:42.874124+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:37:42.874124+0000","last_undegraded":"2024-04-25T05:37:42.874124+0000","last_fullsized":"2024-04-25T05:37:42.874124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":228,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874780+0000","last_change":"2024-04-25T05:36:54.269613+0000","last_active":"2024-04-25T05:37:42.874780+0000","last_peered":"2024-04-25T05:37:42.874780+0000","last_clean":"2024-04-25T05:37:42.874780+0000","last_became_active":"2024-04-25T05:36:54.244739+0000","last_became_peered":"2024-04-25T05:36:54.244739+0000","last_unstale":"2024-04-25T05:37:42.874780+0000","last_undegraded":"2024-04-25T05:37:42.874780+0000","last_fullsized":"2024-04-25T05:37:42.874780+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"147'9950","reported_seq":10220,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.881957+0000","last_change":"2024-04-25T05:36:53.309704+0000","last_active":"2024-04-25T05:37:42.881957+0000","last_peered":"2024-04-25T05:37:42.881957+0000","last_clean":"2024-04-25T05:37:42.881957+0000","last_became_active":"2024-04-25T05:35:09.678971+0000","last_became_peered":"2024-04-25T05:35:09.678971+0000","last_unstale":"2024-04-25T05:37:42.881957+0000","last_undegraded":"2024-04-25T05:37:42.881957+0000","last_fullsized":"2024-04-25T05:37:42.881957+0000","mapping_epoch":88,"log_start":"106'4306","ondisk_log_start":"106'4306","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7661","last_scrub_stamp":"2024-04-25T05:36:53.309266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.309266+0000","objects_scrubbed":6949,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:06.908353+0000","scrub_duration":2789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300810240,"num_objects":4590,"num_object_clones":0,"num_object_copies":13770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":169,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.882627+0000","last_change":"2024-04-25T05:35:39.752489+0000","last_active":"2024-04-25T05:37:42.882627+0000","last_peered":"2024-04-25T05:37:42.882627+0000","last_clean":"2024-04-25T05:37:42.882627+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:37:42.882627+0000","last_undegraded":"2024-04-25T05:37:42.882627+0000","last_fullsized":"2024-04-25T05:37:42.882627+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:39.752319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.752319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.883795+0000","last_change":"2024-04-25T05:36:49.750652+0000","last_active":"2024-04-25T05:37:42.883795+0000","last_peered":"2024-04-25T05:37:42.883795+0000","last_clean":"2024-04-25T05:37:42.883795+0000","last_became_active":"2024-04-25T05:34:02.708341+0000","last_became_peered":"2024-04-25T05:34:02.708341+0000","last_unstale":"2024-04-25T05:37:42.883795+0000","last_undegraded":"2024-04-25T05:37:42.883795+0000","last_fullsized":"2024-04-25T05:37:42.883795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:56.472869+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":49,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874123+0000","last_change":"2024-04-25T05:36:54.226814+0000","last_active":"2024-04-25T05:37:42.874123+0000","last_peered":"2024-04-25T05:37:42.874123+0000","last_clean":"2024-04-25T05:37:42.874123+0000","last_became_active":"2024-04-25T05:36:54.226138+0000","last_became_peered":"2024-04-25T05:36:54.226138+0000","last_unstale":"2024-04-25T05:37:42.874123+0000","last_undegraded":"2024-04-25T05:37:42.874123+0000","last_fullsized":"2024-04-25T05:37:42.874123+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":205,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874869+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:37:42.874869+0000","last_peered":"2024-04-25T05:37:42.874869+0000","last_clean":"2024-04-25T05:37:42.874869+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:37:42.874869+0000","last_undegraded":"2024-04-25T05:37:42.874869+0000","last_fullsized":"2024-04-25T05:37:42.874869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"148'10237","reported_seq":10368,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.983155+0000","last_change":"2024-04-25T05:35:09.682779+0000","last_active":"2024-04-25T05:37:42.983155+0000","last_peered":"2024-04-25T05:37:42.983155+0000","last_clean":"2024-04-25T05:37:42.983155+0000","last_became_active":"2024-04-25T05:35:09.681887+0000","last_became_peered":"2024-04-25T05:35:09.681887+0000","last_unstale":"2024-04-25T05:37:42.983155+0000","last_undegraded":"2024-04-25T05:37:42.983155+0000","last_fullsized":"2024-04-25T05:37:42.983155+0000","mapping_epoch":88,"log_start":"112'5456","ondisk_log_start":"112'5456","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313720832,"num_objects":4787,"num_object_clones":0,"num_object_copies":14361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":480768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":169,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.875024+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:37:42.875024+0000","last_peered":"2024-04-25T05:37:42.875024+0000","last_clean":"2024-04-25T05:37:42.875024+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:37:42.875024+0000","last_undegraded":"2024-04-25T05:37:42.875024+0000","last_fullsized":"2024-04-25T05:37:42.875024+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.882444+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:37:42.882444+0000","last_peered":"2024-04-25T05:37:42.882444+0000","last_clean":"2024-04-25T05:37:42.882444+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:37:42.882444+0000","last_undegraded":"2024-04-25T05:37:42.882444+0000","last_fullsized":"2024-04-25T05:37:42.882444+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.885464+0000","last_change":"2024-04-25T05:36:54.712552+0000","last_active":"2024-04-25T05:37:42.885464+0000","last_peered":"2024-04-25T05:37:42.885464+0000","last_clean":"2024-04-25T05:37:42.885464+0000","last_became_active":"2024-04-25T05:33:58.674615+0000","last_became_peered":"2024-04-25T05:33:58.674615+0000","last_unstale":"2024-04-25T05:37:42.885464+0000","last_undegraded":"2024-04-25T05:37:42.885464+0000","last_fullsized":"2024-04-25T05:37:42.885464+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:08.084311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":210,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-04-25T05:37:42.874788+0000","last_change":"2024-04-25T05:36:54.254870+0000","last_active":"2024-04-25T05:37:42.874788+0000","last_peered":"2024-04-25T05:37:42.874788+0000","last_clean":"2024-04-25T05:37:42.874788+0000","last_became_active":"2024-04-25T05:36:54.234440+0000","last_became_peered":"2024-04-25T05:36:54.234440+0000","last_unstale":"2024-04-25T05:37:42.874788+0000","last_undegraded":"2024-04-25T05:37:42.874788+0000","last_fullsized":"2024-04-25T05:37:42.874788+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:53.169712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4819386392,"num_objects":73539,"num_object_clones":0,"num_object_copies":220617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7190,"num_objects_unfound":0,"num_objects_dirty":73539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160195,"num_write_kb":7479425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24802,"num_bytes_recovered":1625227288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15730819072,"data_stored":15730802784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75267,"ondisk_log_size":75267,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":223,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":6268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070515,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497028,"kb_used_data":1425596,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92248124,"statfs":{"total":95995035648,"available":94462078976,"internally_reserved":0,"allocated":1459810304,"data_stored":1459396643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2462232,"kb_used_data":2278348,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91282920,"statfs":{"total":95995035648,"available":93473710080,"internally_reserved":0,"allocated":2333028352,"data_stored":2332590343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2523144,"kb_used_data":2389244,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":91222008,"statfs":{"total":95995035648,"available":93411336192,"internally_reserved":0,"allocated":2446585856,"data_stored":2446163853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2409184,"kb_used_data":2269716,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91335968,"statfs":{"total":95995035648,"available":93528031232,"internally_reserved":0,"allocated":2324189184,"data_stored":2323807650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3163664,"kb_used_data":3001984,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":90581488,"statfs":{"total":95995035648,"available":92755443712,"internally_reserved":0,"allocated":3074031616,"data_stored":3073605333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986240,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1493256,"kb_used_data":1403776,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92251896,"statfs":{"total":95995035648,"available":94465941504,"internally_reserved":0,"allocated":1437466624,"data_stored":1437047517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691900,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179768,"kb_used_data":1116656,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92565384,"statfs":{"total":95995035648,"available":94786953216,"internally_reserved":0,"allocated":1143455744,"data_stored":1143052535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2442596352,"data_stored":2442592280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2342129664,"data_stored":2342125592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2325151744,"data_stored":2325151744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3079999488,"data_stored":3079995416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1434583040,"data_stored":1434583040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1136459776,"data_stored":1136459776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1524764672,"data_stored":1524760600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1445134336,"data_stored":1445134336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:37:50.233 INFO:tasks.thrashosds.thrasher:Setting 4.6 to [5, 1, 7] 2024-04-25T05:37:50.233 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.6', '5', '1', '7'] 2024-04-25T05:37:50.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 4.6 5 1 7 2024-04-25T05:37:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:50.260 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[27010]: osdmap e149: 8 total, 8 up, 7 in 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[37492]: osdmap e149: 8 total, 8 up, 7 in 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:37:50.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:50 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:37:50.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:50.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:50 smithi060 ceph-mon[30898]: osdmap e149: 8 total, 8 up, 7 in 2024-04-25T05:37:50.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:50 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:37:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:50 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:37:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:50 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:37:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:50 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:37:50.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:50.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:50.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:50.671 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:50.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:50.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:50.915 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:50.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:50.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:50.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:51.019 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:51.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:51.122 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:51.167 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[27010]: pgmap v173: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4959/211401 objects misplaced (2.346%); 40 MiB/s, 646 objects/s recovering 2024-04-25T05:37:51.167 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[27010]: osdmap e150: 8 total, 8 up, 7 in 2024-04-25T05:37:51.168 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:37:51.168 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[37492]: pgmap v173: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4959/211401 objects misplaced (2.346%); 40 MiB/s, 646 objects/s recovering 2024-04-25T05:37:51.168 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[37492]: osdmap e150: 8 total, 8 up, 7 in 2024-04-25T05:37:51.168 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:51 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:37:51.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:51.289 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:51.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:51.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:51.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:51 smithi060 ceph-mon[30898]: pgmap v173: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4959/211401 objects misplaced (2.346%); 40 MiB/s, 646 objects/s recovering 2024-04-25T05:37:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:51 smithi060 ceph-mon[30898]: osdmap e150: 8 total, 8 up, 7 in 2024-04-25T05:37:51.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:51 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:37:51.510 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:51.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:51.641 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:51.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:51.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:51.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:51.878 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:37:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:37:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:37:51.878 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:37:51.878 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:37:51.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:51.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:52.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:52.024 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:52.108 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:37:52.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:52.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:52.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:52.294 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:52.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: 4.1c deep-scrub starts 2024-04-25T05:37:52.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: 4.1c deep-scrub ok 2024-04-25T05:37:52.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: osdmap e151: 8 total, 8 up, 7 in 2024-04-25T05:37:52.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1109046055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2579023105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: 4.1c deep-scrub starts 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: 4.1c deep-scrub ok 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: osdmap e151: 8 total, 8 up, 7 in 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1109046055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2579023105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:52.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:52.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: 4.1c deep-scrub starts 2024-04-25T05:37:52.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: 4.1c deep-scrub ok 2024-04-25T05:37:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: osdmap e151: 8 total, 8 up, 7 in 2024-04-25T05:37:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1109046055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:37:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2579023105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:52.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:52.541 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:52.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:52.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.787 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:52.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:52.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:52.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:53.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:53.034 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: pgmap v176: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 671 op/s; 4959/211401 objects misplaced (2.346%); 20 MiB/s, 317 objects/s recovering 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: osdmap e152: 8 total, 8 up, 7 in 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:37:53.113 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: pgmap v176: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 671 op/s; 4959/211401 objects misplaced (2.346%); 20 MiB/s, 317 objects/s recovering 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: osdmap e152: 8 total, 8 up, 7 in 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:37:53.114 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:53.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:53.160 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:53.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:37:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: pgmap v176: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 671 op/s; 4959/211401 objects misplaced (2.346%); 20 MiB/s, 317 objects/s recovering 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: osdmap e152: 8 total, 8 up, 7 in 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:37:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:53.252 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:53.253 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:53.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:53.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:53.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:53.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:53.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:53.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:53.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:53.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:53.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:53.812 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:53.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:53.935 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:53.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:54.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:54.058 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:54.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:54.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:54.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:54.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:54 smithi043 ceph-mon[27010]: osdmap e153: 8 total, 8 up, 7 in 2024-04-25T05:37:54.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:54.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:54 smithi043 ceph-mon[37492]: osdmap e153: 8 total, 8 up, 7 in 2024-04-25T05:37:54.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:54.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:54.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:37:54.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:54 smithi060 ceph-mon[30898]: osdmap e153: 8 total, 8 up, 7 in 2024-04-25T05:37:54.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:54.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:54.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:54.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:54.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:37:54.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:54.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:37:54.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:54.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:54.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:55.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:55 smithi043 ceph-mon[27010]: pgmap v179: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 561 op/s; 1283/204675 objects misplaced (0.627%); 56 MiB/s, 890 objects/s recovering 2024-04-25T05:37:55.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]: dispatch 2024-04-25T05:37:55.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:55.153 INFO:teuthology.orchestra.run.smithi043.stderr:set 4.6 pg_upmap mapping to [5,1,7] 2024-04-25T05:37:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.169 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:55.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:55.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:55.294 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:55 smithi043 ceph-mon[37492]: pgmap v179: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 561 op/s; 1283/204675 objects misplaced (0.627%); 56 MiB/s, 890 objects/s recovering 2024-04-25T05:37:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]: dispatch 2024-04-25T05:37:55.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:55.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:55.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:55 smithi060 ceph-mon[30898]: pgmap v179: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 561 op/s; 1283/204675 objects misplaced (0.627%); 56 MiB/s, 890 objects/s recovering 2024-04-25T05:37:55.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]: dispatch 2024-04-25T05:37:55.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:55.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:55.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:55.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:55.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:55.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:55.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:56.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.090 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:56.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:56.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:56.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:37:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:56.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:56.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:56.335 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:56.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]': finished 2024-04-25T05:37:56.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:56 smithi043 ceph-mon[27010]: osdmap e154: 8 total, 8 up, 7 in 2024-04-25T05:37:56.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]': finished 2024-04-25T05:37:56.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:56 smithi043 ceph-mon[37492]: osdmap e154: 8 total, 8 up, 7 in 2024-04-25T05:37:56.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:56.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2094667156' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.6", "id": [5, 1, 7]}]': finished 2024-04-25T05:37:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:56 smithi060 ceph-mon[30898]: osdmap e154: 8 total, 8 up, 7 in 2024-04-25T05:37:56.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:56.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:56.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:56.730 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:56.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:56.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:56.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:56.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:57.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:57.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:57.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:57.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:57 smithi043 ceph-mon[37492]: pgmap v181: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 485 op/s; 1283/204675 objects misplaced (0.627%); 48 MiB/s, 769 objects/s recovering 2024-04-25T05:37:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:57 smithi043 ceph-mon[37492]: osdmap e155: 8 total, 8 up, 7 in 2024-04-25T05:37:57.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:57 smithi043 ceph-mon[27010]: pgmap v181: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 485 op/s; 1283/204675 objects misplaced (0.627%); 48 MiB/s, 769 objects/s recovering 2024-04-25T05:37:57.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:57 smithi043 ceph-mon[27010]: osdmap e155: 8 total, 8 up, 7 in 2024-04-25T05:37:57.407 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:57 smithi060 ceph-mon[30898]: pgmap v181: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 485 op/s; 1283/204675 objects misplaced (0.627%); 48 MiB/s, 769 objects/s recovering 2024-04-25T05:37:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:57 smithi060 ceph-mon[30898]: osdmap e155: 8 total, 8 up, 7 in 2024-04-25T05:37:57.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:37:57.522 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:57.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:37:57.607 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:57.621 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:37:57.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:57.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:57.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:57.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:57.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:57.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:37:58.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:37:58.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:58.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:58.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:58.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:58.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:58.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:58.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.537 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:58.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:37:58.558 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:37:58.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:58.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:58.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:37:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:37:58.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:37:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:37:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:37:58.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:37:58.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:58.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:37:58.982 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:37:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:37:59.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:59.067 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:37:59.080 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:59.106 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:37:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:37:59.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:37:59.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:59.204 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:59.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:37:59.349 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:37:59.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:59 smithi060 ceph-mon[30898]: pgmap v183: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 535 op/s; 1283/202389 objects misplaced (0.634%); 40 MiB/s, 633 objects/s recovering 2024-04-25T05:37:59.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:37:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3016295323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:59.523 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:59 smithi043 ceph-mon[27010]: pgmap v183: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 535 op/s; 1283/202389 objects misplaced (0.634%); 40 MiB/s, 633 objects/s recovering 2024-04-25T05:37:59.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:37:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3016295323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:59.524 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:59 smithi043 ceph-mon[37492]: pgmap v183: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 535 op/s; 1283/202389 objects misplaced (0.634%); 40 MiB/s, 633 objects/s recovering 2024-04-25T05:37:59.524 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:37:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3016295323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:37:59.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:37:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:37:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:37:59.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:37:59.754 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:37:59.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:37:59.879 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:37:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:37:59.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:37:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:37:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:37:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:37:59.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:00.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:00.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:00.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:00.254 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:00.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:00.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:38:00.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:00.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:00.501 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:00.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:00.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:00.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:00.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:00.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:00.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:01.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:01.037 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:01.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:01.158 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:01.159 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:38:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:01.182 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[37492]: pgmap v184: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 773 B/s rd, 0 B/s wr, 623 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:01.182 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2351860791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.182 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[27010]: pgmap v184: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 773 B/s rd, 0 B/s wr, 623 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:01.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2351860791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:01 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:01.289 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:01.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:01.415 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:01.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:01 smithi060 ceph-mon[30898]: pgmap v184: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 773 B/s rd, 0 B/s wr, 623 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:01.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2351860791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:01 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:01.517 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:01.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:01.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:01.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:01.807 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:01.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:01.931 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:01.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:02.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:02.056 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:02.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:02.180 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:02.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:02.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:02.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:02.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:38:02.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:38:02.425 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 7] 2024-04-25T05:38:02.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd out 0 2024-04-25T05:38:02.431 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:02.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:02.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:02.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[27010]: osdmap e156: 8 total, 8 up, 7 in 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[37492]: osdmap e156: 8 total, 8 up, 7 in 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:02.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-mon[30898]: osdmap e156: 8 total, 8 up, 7 in 2024-04-25T05:38:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:02.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:02.553 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:02.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:02.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:02.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:02.830 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:02.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:02.955 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:02.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:02.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:38:03.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:03.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:03.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:03.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:03.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:03.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:03.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:03 smithi043 ceph-mon[27010]: pgmap v186: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 776 B/s rd, 0 B/s wr, 626 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:03.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:03 smithi043 ceph-mon[37492]: pgmap v186: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 776 B/s rd, 0 B/s wr, 626 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:03.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:03.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:03.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:03 smithi060 ceph-mon[30898]: pgmap v186: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 776 B/s rd, 0 B/s wr, 626 op/s; 809/192300 objects misplaced (0.421%); 3.9 MiB/s, 62 objects/s recovering 2024-04-25T05:38:03.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:03.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:03.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:03.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:03.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:03.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:03.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:03.990 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:03.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:04.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:04.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:04.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:04.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:04.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:04.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:04.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:04.508 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:04.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:04.633 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:04.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:04.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:04.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:04.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:04.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:04.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:05.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:05.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:05.143 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:05.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:05.268 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:05.355 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:38:05.355 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:38:05.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.355 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:176698 7.d e156/151 7:b418b5e5:::benchmark_data_smithi043_74356_object59829:head v 156'11077, mlcod=156'11074)", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:05.353192+0000", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0013039620000000001, 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001336722, 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353192+0000", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353192+0000", 2024-04-25T05:38:05.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353193+0000", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4099999999999998e-07 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353200+0000", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6200000000000001e-06 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353211+0000", 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1046000000000001e-05 2024-04-25T05:38:05.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353217+0000", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8889999999999996e-06 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353262+0000", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4129999999999999e-05 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353281+0000", 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8953e-05 2024-04-25T05:38:05.358 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:176693 7.2 7:42e0f683:::benchmark_data_smithi043_74356_object59824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:05.340446+0000", 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.014050164, 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014164823999999999, 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 176693 2024-04-25T05:38:05.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340446+0000", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340446+0000", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340447+0000", 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.54e-07 2024-04-25T05:38:05.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340451+0000", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0790000000000002e-06 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340460+0000", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8510000000000005e-06 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340467+0000", 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0949999999999998e-06 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.342613+0000", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021459220000000002 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.342741+0000", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000127875 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.342870+0000", 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012886899999999999 2024-04-25T05:38:05.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350550+0000", 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0076807059999999998 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350658+0000", 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010785099999999999 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.363 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:176695 7.2 7:4af1937c:::benchmark_data_smithi043_74356_object59826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:05.340477+0000", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.014018932, 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014237614000000001, 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 176695 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340477+0000", 2024-04-25T05:38:05.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340477+0000", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340478+0000", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2500000000000001e-07 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340480+0000", 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2359999999999999e-06 2024-04-25T05:38:05.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340485+0000", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0180000000000001e-06 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.340490+0000", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.87e-06 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.347875+0000", 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0073851589999999996 2024-04-25T05:38:05.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.348046+0000", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000170991 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.348181+0000", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013470800000000001 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350665+0000", 2024-04-25T05:38:05.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024844889999999999 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:176697 7.c 7:312d9ad3:::benchmark_data_smithi043_74356_object59828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:05.350072+0000", 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0044246859999999997, 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047393649999999997, 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:05.368 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 176697 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350072+0000", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350072+0000", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:05.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350073+0000", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.711e-06 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350078+0000", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7230000000000004e-06 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350088+0000", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0404999999999999e-05 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350096+0000", 2024-04-25T05:38:05.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9079999999999995e-06 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350490+0000", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00039352499999999998 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350606+0000", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116122 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.350717+0000", 2024-04-25T05:38:05.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110608 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353199+0000", 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024820720000000001 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:05.353286+0000", 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7260000000000001e-05 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-04-25T05:38:05.372 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:38:05.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:05.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:05 smithi043 ceph-mon[27010]: pgmap v187: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 404/177711 objects misplaced (0.227%); 6.0 MiB/s, 97 objects/s recovering 2024-04-25T05:38:05.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:05 smithi043 ceph-mon[37492]: pgmap v187: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 404/177711 objects misplaced (0.227%); 6.0 MiB/s, 97 objects/s recovering 2024-04-25T05:38:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:05.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:05.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:05 smithi060 ceph-mon[30898]: pgmap v187: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 404/177711 objects misplaced (0.227%); 6.0 MiB/s, 97 objects/s recovering 2024-04-25T05:38:05.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:05.577 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:05.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:05.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:05.754 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:05.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:05.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:05.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:05.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:05.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:06.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:06.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:06.225 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:06.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:06.259 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:06.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:38:06.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:06.383 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:06.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:06.507 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:06.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:06.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:06.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:06.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:06.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:06.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:38:06.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:07.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:07.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:07.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:07.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:07.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:07.422 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:07 smithi060 ceph-mon[30898]: pgmap v188: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 938 op/s; 404/177711 objects misplaced (0.227%); 5.0 MiB/s, 81 objects/s recovering 2024-04-25T05:38:07.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.545 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:07.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:07.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:07 smithi043 ceph-mon[27010]: pgmap v188: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 938 op/s; 404/177711 objects misplaced (0.227%); 5.0 MiB/s, 81 objects/s recovering 2024-04-25T05:38:07.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:07 smithi043 ceph-mon[37492]: pgmap v188: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 938 op/s; 404/177711 objects misplaced (0.227%); 5.0 MiB/s, 81 objects/s recovering 2024-04-25T05:38:07.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:07.668 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:07.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:07.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:07.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:08.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:08.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:08.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:08.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:08.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:08.284 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:08.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:08.550 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:08 smithi060 ceph-mon[30898]: osdmap e157: 8 total, 8 up, 7 in 2024-04-25T05:38:08.551 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2865989356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:08.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:08 smithi043 ceph-mon[27010]: osdmap e157: 8 total, 8 up, 7 in 2024-04-25T05:38:08.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2865989356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:08.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:08 smithi043 ceph-mon[37492]: osdmap e157: 8 total, 8 up, 7 in 2024-04-25T05:38:08.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2865989356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:08.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:08.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:08.716 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:08.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:08.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:08.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:08.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:08.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:08.977 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:08.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:09.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:09.101 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:09.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:09.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:09.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:09.351 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:09.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:09 smithi060 ceph-mon[30898]: pgmap v190: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 404/175461 objects misplaced (0.230%); 2.8 MiB/s, 45 objects/s recovering 2024-04-25T05:38:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:09 smithi060 ceph-mon[30898]: osdmap e158: 8 total, 8 up, 7 in 2024-04-25T05:38:09.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.618 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:09.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:09.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:09 smithi043 ceph-mon[27010]: pgmap v190: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 404/175461 objects misplaced (0.230%); 2.8 MiB/s, 45 objects/s recovering 2024-04-25T05:38:09.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:09 smithi043 ceph-mon[27010]: osdmap e158: 8 total, 8 up, 7 in 2024-04-25T05:38:09.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:09 smithi043 ceph-mon[37492]: pgmap v190: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 404/175461 objects misplaced (0.230%); 2.8 MiB/s, 45 objects/s recovering 2024-04-25T05:38:09.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:09 smithi043 ceph-mon[37492]: osdmap e158: 8 total, 8 up, 7 in 2024-04-25T05:38:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:09.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:09.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:09.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:09.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:09.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:38:09.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:10.009 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:10.110 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:10.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:10.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:10.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:10.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:10.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:10.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:10.544 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:10.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:10.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:10.729 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:10.730 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:10.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:10.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:10.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:10.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:10.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:11.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:11.195 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:11.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:11.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:11.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:11.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:11.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:11 smithi043 ceph-mon[27010]: pgmap v192: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-04-25T05:38:11.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:11 smithi043 ceph-mon[37492]: pgmap v192: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-04-25T05:38:11.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:11 smithi060 ceph-mon[30898]: pgmap v192: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-04-25T05:38:11.532 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:11.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:11.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:11.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:38:11.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:11 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:38:11.494+0000 7f173ec68700 -1 mon.a@0(leader).osd e158 definitely_dead 0 2024-04-25T05:38:11.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:11.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:38:11.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:11.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:11.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:11.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:12.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:12.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:12.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:12.189 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:12.258 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.0. 2024-04-25T05:38:12.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:12.314 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1906322156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[37492]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1906322156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:12 smithi043 ceph-mon[27010]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:38:12.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:12.437 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:12.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1906322156' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:38:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:12 smithi060 ceph-mon[30898]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:38:12.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:12.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:12.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:12.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:12.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:12.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:12.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:12.934 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:12.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:13.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:13.058 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:13.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:13.183 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:13.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:12 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:13.240 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:38:13.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:13.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:13.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:13.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:13 smithi060 ceph-mon[30898]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 481 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-04-25T05:38:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:38:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:13 smithi060 ceph-mon[30898]: osdmap e159: 8 total, 8 up, 6 in 2024-04-25T05:38:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2141115644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:13.565 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:13.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[27010]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 481 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[27010]: osdmap e159: 8 total, 8 up, 6 in 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2141115644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[37492]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 481 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[37492]: osdmap e159: 8 total, 8 up, 6 in 2024-04-25T05:38:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2141115644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:13.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:13.690 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:13.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:13.816 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:13.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:13.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:13.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:14.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:14.067 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:14.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:14.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:14.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:14.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:14.461 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:14.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:14.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:14 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:14.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:14 smithi060 ceph-mon[30898]: osdmap e160: 8 total, 8 up, 6 in 2024-04-25T05:38:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:14 smithi060 ceph-mon[30898]: osdmap e161: 8 total, 8 up, 6 in 2024-04-25T05:38:14.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:14.583 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[27010]: osdmap e160: 8 total, 8 up, 6 in 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[27010]: osdmap e161: 8 total, 8 up, 6 in 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[37492]: osdmap e160: 8 total, 8 up, 6 in 2024-04-25T05:38:14.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:14 smithi043 ceph-mon[37492]: osdmap e161: 8 total, 8 up, 6 in 2024-04-25T05:38:14.597 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:14.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:14.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:14.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:14.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:14.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:14.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:15.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:15.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:15.142 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:15.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:15.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: pgmap v196: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.18k op/s; 4.1 MiB/s, 64 objects/s recovering 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11385 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11342 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 158'11323 2024-04-25T05:38:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.d starting backfill to osd.7 from (0'0,0'0] MAX to 158'11440 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11523 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11262 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 158'11726 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 158'11378 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: Health check failed: Reduced data availability: 15 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[37492]: osdmap e162: 8 total, 8 up, 6 in 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: pgmap v196: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.18k op/s; 4.1 MiB/s, 64 objects/s recovering 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:38:15.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11385 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11342 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 158'11323 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.d starting backfill to osd.7 from (0'0,0'0] MAX to 158'11440 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11523 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11262 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 158'11726 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 158'11378 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: Health check failed: Reduced data availability: 15 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:15.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:15 smithi043 ceph-mon[27010]: osdmap e162: 8 total, 8 up, 6 in 2024-04-25T05:38:15.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: pgmap v196: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.18k op/s; 4.1 MiB/s, 64 objects/s recovering 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11385 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.4 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11342 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 158'11323 2024-04-25T05:38:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.d starting backfill to osd.7 from (0'0,0'0] MAX to 158'11440 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.7 starting backfill to osd.7 from (0'0,0'0] MAX to 158'11523 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 158'11262 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 158'11726 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 158'11378 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: Health check failed: Reduced data availability: 15 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:15.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:15 smithi060 ceph-mon[30898]: osdmap e162: 8 total, 8 up, 6 in 2024-04-25T05:38:15.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:15.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:15.667 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:15.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:15.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:15.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:15.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:15.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:16.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:16.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:16.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:16.210 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:16.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:16.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:16.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.473 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:16.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:16.545 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:16.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:16.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:16.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:16.645 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:16.694 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:16.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:16.746 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:16.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.756 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:129494 7.c 7:3a6c375b:::benchmark_data_smithi043_74356_object12625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:02.673264+0000", 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 74.079268432000006, 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.153346014, 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 129494 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.757 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673264+0000", 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673264+0000", 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.758 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673266+0000", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3289999999999999e-06 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673270+0000", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3019999999999996e-06 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673279+0000", 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2879999999999998e-06 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673286+0000", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8700000000000003e-06 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673432+0000", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014605799999999999 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673628+0000", 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000196219 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.737777+0000", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.064149103999999998 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.740427+0000", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026490569999999998 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.741304+0000", 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00087774399999999996 2024-04-25T05:38:16.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.742315+0000", 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010110519999999999 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.826435+0000", 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.084119930999999995 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.826489+0000", 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3409999999999999e-05 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.826610+0000", 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012165 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:129496 7.2 7:4c6dc04f:::benchmark_data_smithi043_74356_object12627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:16.763 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:02.673304+0000", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 74.079228455000006, 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.155698794, 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 129496 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673304+0000", 2024-04-25T05:38:16.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673304+0000", 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673305+0000", 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2800000000000002e-07 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673307+0000", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.52e-06 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673313+0000", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3249999999999998e-06 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673318+0000", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0799999999999996e-06 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738314+0000", 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.064996229000000003 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738428+0000", 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011429000000000001 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738554+0000", 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000126072 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.743003+0000", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0044487670000000002 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.826306+0000", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.083302887000000006 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.828890+0000", 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025838219999999999 2024-04-25T05:38:16.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.828959+0000", 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9435000000000002e-05 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.829003+0000", 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3939000000000001e-05 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:129498 7.a 7:5731957c:::benchmark_data_smithi043_74356_object12629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:02.673329+0000", 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 74.079203343000003, 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15567492699999999, 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 129498 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673329+0000", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.771 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673329+0000", 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673330+0000", 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.27e-07 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.772 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673342+0000", 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2283000000000001e-05 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673348+0000", 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.541e-06 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.773 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.673353+0000", 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0900000000000004e-06 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.774 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738341+0000", 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.064988219999999999 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738463+0000", 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121925 2024-04-25T05:38:16.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.738574+0000", 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011166399999999999 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.776 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.740909+0000", 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023348549999999998 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.742985+0000", 2024-04-25T05:38:16.777 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020761590000000002 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.828925+0000", 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.085940004 2024-04-25T05:38:16.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.828969+0000", 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3637000000000003e-05 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.829004+0000", 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5222000000000003e-05 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:131537 7.a 7:5adb581a:::benchmark_data_smithi043_74356_object14668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.624174+0000", 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 71.128359132, 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.162281497, 2024-04-25T05:38:16.780 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 131537 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624174+0000", 2024-04-25T05:38:16.781 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624174+0000", 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624175+0000", 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6899999999999996e-07 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.782 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624178+0000", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2310000000000001e-06 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624185+0000", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9750000000000001e-06 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624190+0000", 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.746e-06 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.783 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624227+0000", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7460999999999999e-05 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624301+0000", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3779e-05 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.624365+0000", 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4677000000000004e-05 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.784 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786160+0000", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.161794086 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786204+0000", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4549000000000003e-05 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786367+0000", 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016254900000000001 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.785 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786407+0000", 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0729000000000003e-05 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786455+0000", 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7746e-05 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(7.c 135/130 [PushReplyOp(7:3215e2f8:::benchmark_data_smithi043_74356_object114618:head)])", 2024-04-25T05:38:16.786 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.625113+0000", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 71.127419333000006, 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.158934254, 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625113+0000", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625113+0000", 2024-04-25T05:38:16.787 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625114+0000", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.79e-07 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625116+0000", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.424e-06 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625121+0000", 2024-04-25T05:38:16.788 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4780000000000002e-06 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625126+0000", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6149999999999996e-06 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.625158+0000", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2072000000000001e-05 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.784048+0000", 2024-04-25T05:38:16.789 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15888938599999999 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.24734.0:131537 7.a e135/88 ondisk, result = 0)", 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.626222+0000", 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 71.126310477999994, 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15994902499999999, 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.626222+0000", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.626222+0000", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.626223+0000", 2024-04-25T05:38:16.791 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6599999999999996e-07 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.626226+0000", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4640000000000002e-06 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.784216+0000", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15799017400000001 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.784226+0000", 2024-04-25T05:38:16.792 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3000000000000007e-06 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786145+0000", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019194730000000001 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786153+0000", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7379999999999994e-06 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.786171+0000", 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8309999999999999e-05 2024-04-25T05:38:16.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165408 7.c 7:30f51da7:::benchmark_data_smithi043_74356_object48539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 25.621550073999998, 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.013940488, 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.794 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165408 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3799999999999998e-07 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130986+0000", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9830000000000001e-06 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130992+0000", 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8289999999999996e-06 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.796 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130997+0000", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4129999999999997e-06 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.131160+0000", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016310700000000001 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.131169+0000", 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3159999999999997e-06 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.797 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132057+0000", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.000888287 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132198+0000", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014108 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132336+0000", 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013787700000000001 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.798 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135299+0000", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002962648 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141378+0000", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060790439999999996 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144822+0000", 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034443059999999998 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.799 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144875+0000", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2799000000000001e-05 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144923+0000", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8361e-05 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165436 7.c 7:3742aa0a:::benchmark_data_smithi043_74356_object48567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:38:16.800 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 25.586487193, 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97903490800000004, 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165436 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166047+0000", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3e-06 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166051+0000", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.048e-06 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166059+0000", 2024-04-25T05:38:16.803 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9319999999999996e-06 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166065+0000", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3849999999999999e-06 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.167368+0000", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001302316 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.167377+0000", 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5529999999999998e-06 2024-04-25T05:38:16.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135334+0000", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96795652399999998 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135460+0000", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012674600000000001 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135592+0000", 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013188100000000001 2024-04-25T05:38:16.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141420+0000", 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058273129999999998 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141456+0000", 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6055e-05 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.806 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144990+0000", 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035342210000000002 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145044+0000", 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3715999999999999e-05 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145080+0000", 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6918000000000001e-05 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165470 7.c 7:382b4df9:::benchmark_data_smithi043_74356_object48601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 25.541207935999999, 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93388353000000002, 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165470 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211326+0000", 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5850000000000001e-06 2024-04-25T05:38:16.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211332+0000", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2499999999999997e-06 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211343+0000", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1252e-05 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211351+0000", 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9850000000000006e-06 2024-04-25T05:38:16.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.213644+0000", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022933939999999998 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.213653+0000", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1449999999999997e-06 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.138752+0000", 2024-04-25T05:38:16.811 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.925098902 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.138925+0000", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017265999999999999 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.139075+0000", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000150354 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141492+0000", 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024170149999999998 2024-04-25T05:38:16.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145087+0000", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003594683 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145128+0000", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0615000000000003e-05 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145172+0000", 2024-04-25T05:38:16.813 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4524999999999998e-05 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145208+0000", 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6164999999999997e-05 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165480 7.c 7:3433329a:::benchmark_data_smithi043_74356_object48611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:38:16.814 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 25.527281282000001, 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92329429500000004, 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.815 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[27010]: pgmap v199: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[27010]: osdmap e163: 8 total, 8 up, 6 in 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/141283520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[37492]: pgmap v199: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[37492]: osdmap e163: 8 total, 8 up, 6 in 2024-04-25T05:38:16.816 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/141283520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:16.816 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165480 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.817 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225253+0000", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.781e-06 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225260+0000", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3899999999999998e-06 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225275+0000", 2024-04-25T05:38:16.818 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5044e-05 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225283+0000", 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.439e-06 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225337+0000", 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3766e-05 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.819 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225346+0000", 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2099999999999999e-06 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141525+0000", 2024-04-25T05:38:16.820 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91617920900000005 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141659+0000", 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013405700000000001 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.821 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141786+0000", 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012625700000000001 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144809+0000", 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003022921 2024-04-25T05:38:16.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.147708+0000", 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002899956 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.823 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148449+0000", 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00074005000000000002 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148496+0000", 2024-04-25T05:38:16.824 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7524000000000001e-05 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148546+0000", 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9691000000000002e-05 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.825 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:174846 7.c 7:3d1643f4:::benchmark_data_smithi043_74356_object57977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.045912+0000", 2024-04-25T05:38:16.826 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.706621064, 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13686062500000001, 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.827 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 174846 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045912+0000", 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045912+0000", 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045913+0000", 2024-04-25T05:38:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4109999999999999e-06 2024-04-25T05:38:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045916+0000", 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3749999999999999e-06 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045924+0000", 2024-04-25T05:38:16.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9640000000000003e-06 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045930+0000", 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9030000000000001e-06 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.047492+0000", 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0015621330000000001 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.047617+0000", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012454400000000001 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.047734+0000", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116692 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.055389+0000", 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0076553289999999998 2024-04-25T05:38:16.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.055444+0000", 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4965e-05 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.065995+0000", 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.01055148 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066048+0000", 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2107000000000001e-05 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.182772+0000", 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.116724722 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:174849 7.c 7:33062414:::benchmark_data_smithi043_74356_object57980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.045944+0000", 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.706588312999999, 2024-04-25T05:38:16.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13795849800000001, 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 174849 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045944+0000", 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045944+0000", 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045945+0000", 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7099999999999997e-07 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045947+0000", 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.221e-06 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045952+0000", 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2769999999999996e-06 2024-04-25T05:38:16.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045957+0000", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0889999999999999e-06 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.051086+0000", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005128718 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.051200+0000", 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113897 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.051310+0000", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011027900000000001 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.055511+0000", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042009780000000002 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.055749+0000", 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023741499999999999 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.183805+0000", 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12805630300000001 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.183863+0000", 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7924999999999999e-05 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.183903+0000", 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0024999999999997e-05 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:174851 7.3 7:c2bfb931:::benchmark_data_smithi043_74356_object57982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.045963+0000", 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.70656965, 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13944822200000001, 2024-04-25T05:38:16.842 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 174851 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045963+0000", 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045963+0000", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045963+0000", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3799999999999998e-07 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045965+0000", 2024-04-25T05:38:16.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8780000000000001e-06 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045970+0000", 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4619999999999998e-06 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.045974+0000", 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5090000000000004e-06 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.046053+0000", 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8708000000000006e-05 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.046174+0000", 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120794 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.046286+0000", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111922 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.053330+0000", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070439159999999999 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.053342+0000", 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.206e-05 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.185310+0000", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.131968744 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.185370+0000", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9854e-05 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.185411+0000", 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1037e-05 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.848 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175163 7.3 7:c9a341e6:::benchmark_data_smithi043_74356_object58294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.473054+0000", 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.27947837, 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13807248899999999, 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175163 2024-04-25T05:38:16.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473054+0000", 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473054+0000", 2024-04-25T05:38:16.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473055+0000", 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0500000000000002e-07 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473060+0000", 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3649999999999997e-06 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473067+0000", 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.525e-06 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473074+0000", 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350000000000002e-06 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473128+0000", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3495000000000002e-05 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473231+0000", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010333 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.473340+0000", 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000108556 2024-04-25T05:38:16.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.475497+0000", 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021575840000000002 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476321+0000", 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00082353299999999999 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.611028+0000", 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13470710599999999 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.611079+0000", 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1062e-05 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.611127+0000", 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7892999999999997e-05 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.855 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175164 7.c 7:32b6ec19:::benchmark_data_smithi043_74356_object58295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.474583+0000", 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.277949882, 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.140241796, 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175164 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.858 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474583+0000", 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474583+0000", 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474584+0000", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1900000000000001e-07 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474588+0000", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3030000000000001e-06 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474597+0000", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8089999999999999e-06 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474603+0000", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6290000000000004e-06 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.475654+0000", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001050816 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.475753+0000", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8877999999999998e-05 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.475863+0000", 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109388 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608333+0000", 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13247014500000001 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608376+0000", 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3782999999999997e-05 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614731+0000", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0063549230000000002 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614778+0000", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6623999999999999e-05 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614825+0000", 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6678999999999998e-05 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.863 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175165 7.c 7:338d7f6e:::benchmark_data_smithi043_74356_object58296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.474615+0000", 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.277918172, 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14036636799999999, 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.864 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175165 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474615+0000", 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474615+0000", 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.865 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474615+0000", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8599999999999998e-07 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474618+0000", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7410000000000001e-06 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474624+0000", 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2639999999999997e-06 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474630+0000", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5949999999999998e-06 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480083+0000", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054529160000000004 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480192+0000", 2024-04-25T05:38:16.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109345 2024-04-25T05:38:16.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480290+0000", 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7900999999999997e-05 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608413+0000", 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12812357499999999 2024-04-25T05:38:16.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.612320+0000", 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039063049999999997 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614899+0000", 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025790380000000001 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614941+0000", 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2256000000000003e-05 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614981+0000", 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9946e-05 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:175160 7.6 e156/88 7:6ff542c0:::benchmark_data_smithi043_74356_object58291:head v 156'10813, mlcod=156'10809)", 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.476325+0000", 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.276207422000001, 2024-04-25T05:38:16.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13666378000000001, 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476325+0000", 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476325+0000", 2024-04-25T05:38:16.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476326+0000", 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.61e-07 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476332+0000", 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4690000000000002e-06 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476342+0000", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.55e-06 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476349+0000", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8630000000000002e-06 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.477278+0000", 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00092865999999999999 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.477300+0000", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1994999999999999e-05 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.612948+0000", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.135648567 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.612989+0000", 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1014999999999999e-05 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.875 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPush(7.4 156/140 [PushOp(7:2e607895:::benchmark_data_smithi043_74356_object79424:head, version: 110'4963, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2e607895:::benchmark_data_smithi043_74356_object79424:head@110'4963, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.476366+0000", 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.276166985, 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13846308800000001, 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476366+0000", 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476366+0000", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476366+0000", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0899999999999997e-07 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476400+0000", 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4242000000000002e-05 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476417+0000", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6212e-05 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476431+0000", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4059e-05 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479947+0000", 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035167900000000001 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479963+0000", 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5773e-05 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614829+0000", 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.134865703 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175167 7.c 7:35e27462:::benchmark_data_smithi043_74356_object58298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.879 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.479904+0000", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.272628281999999, 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.137808651, 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175167 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479904+0000", 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479904+0000", 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479906+0000", 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3120000000000001e-06 2024-04-25T05:38:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479910+0000", 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4519999999999999e-06 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479920+0000", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5189999999999998e-06 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.479928+0000", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1440000000000003e-06 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608456+0000", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.128527734 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608642+0000", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018633000000000001 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608808+0000", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016560899999999999 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.612363+0000", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035555029999999998 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614988+0000", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026251569999999999 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617615+0000", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026267220000000002 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617667+0000", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2256999999999998e-05 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617713+0000", 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5911999999999997e-05 2024-04-25T05:38:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175170 7.3 7:c25252b4:::benchmark_data_smithi043_74356_object58301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.480317+0000", 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 13.272215815999999, 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.137798218, 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:38:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175170 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480317+0000", 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480317+0000", 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480318+0000", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6400000000000001e-07 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480323+0000", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7360000000000001e-06 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480332+0000", 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7589999999999992e-06 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480341+0000", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.585e-06 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480635+0000", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000293801 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480769+0000", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013457000000000001 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:38:16.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480892+0000", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012255100000000001 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.611141+0000", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13024924299999999 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.614801+0000", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036597349999999999 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:16.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.618032+0000", 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003230868 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.618076+0000", 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4808000000000002e-05 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.618115+0000", 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8698000000000001e-05 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:38:16.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:38:16.892 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:38:16.892 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:16.893 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:16.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:16.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:16.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:16 smithi060 ceph-mon[30898]: pgmap v199: 148 pgs: 5 unknown, 57 peering, 86 active+clean; 3.1 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s 2024-04-25T05:38:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:16 smithi060 ceph-mon[30898]: osdmap e163: 8 total, 8 up, 6 in 2024-04-25T05:38:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/141283520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:17.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:17.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:17.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:38:17.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:17.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:17.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:17.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:17.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:17.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:17.612 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:17.644 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:17.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:17.775 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:17.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:17.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:18.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:18.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:18.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:18.235 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:18.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:18.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:18.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:18.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:18.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:18.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:18.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:18.506 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:18.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:38:18.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:18.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.795 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:18.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:18.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:18.940 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:18.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:18.990 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:19.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:19.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:19.174 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7] out_osds: [6, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:38:19.174 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:38:19.174 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-25T05:38:19.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:38:19.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:19.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:19 smithi043 ceph-mon[27010]: pgmap v201: 148 pgs: 3 active+remapped+backfilling, 3 unknown, 24 peering, 118 active+clean; 3.1 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 202 op/s; 6255/151422 objects misplaced (4.131%); 41 MiB/s, 660 objects/s recovering 2024-04-25T05:38:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:19 smithi043 ceph-mon[37492]: pgmap v201: 148 pgs: 3 active+remapped+backfilling, 3 unknown, 24 peering, 118 active+clean; 3.1 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 202 op/s; 6255/151422 objects misplaced (4.131%); 41 MiB/s, 660 objects/s recovering 2024-04-25T05:38:19.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:19.441 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:19 smithi060 ceph-mon[30898]: pgmap v201: 148 pgs: 3 active+remapped+backfilling, 3 unknown, 24 peering, 118 active+clean; 3.1 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 202 op/s; 6255/151422 objects misplaced (4.131%); 41 MiB/s, 660 objects/s recovering 2024-04-25T05:38:19.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:19.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:19.599 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:19.651 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:38:19.651 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:38:19.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.651 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:187208 7.4 7:2ef3adf9:::benchmark_data_smithi043_74356_object70339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-25T05:38:19.651 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:19.639199+0000", 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.010456896, 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010487467, 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 187208 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:19.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639199+0000", 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639199+0000", 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639200+0000", 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2099999999999997e-07 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639203+0000", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2940000000000001e-06 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639210+0000", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.81e-06 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639217+0000", 2024-04-25T05:38:19.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7669999999999998e-06 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639285+0000", 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8979000000000005e-05 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639428+0000", 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014245300000000001 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:38:19.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.639650+0000", 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000221914 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.642109+0000", 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024593599999999998 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.642452+0000", 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000343118 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.646465+0000", 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040122769999999999 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:187214 7.4 7:289893e8:::benchmark_data_smithi043_74356_object70345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:19.648423+0000", 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0012328199999999999, 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013687490000000001, 2024-04-25T05:38:19.657 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 187214 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648423+0000", 2024-04-25T05:38:19.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648423+0000", 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648425+0000", 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.336e-06 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648436+0000", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1095e-05 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648450+0000", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4849e-05 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648459+0000", 2024-04-25T05:38:19.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1049999999999995e-06 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648541+0000", 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2873000000000004e-05 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648692+0000", 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015014800000000001 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:38:19.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648851+0000", 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015952800000000001 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:187208 7.4 e163/160)", 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:19.648560+0000", 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.001096126, 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001307615, 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:19.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648560+0000", 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648560+0000", 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648561+0000", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5099999999999999e-07 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648564+0000", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1140000000000002e-06 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648573+0000", 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3689999999999996e-06 2024-04-25T05:38:19.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.648580+0000", 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0809999999999996e-06 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:187216 7.3 e163/88 7:c706edf0:::benchmark_data_smithi043_74356_object70347:head v 163'11655, mlcod=163'11653)", 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:19.649063+0000", 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.00059278599999999999, 2024-04-25T05:38:19.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00085439100000000003, 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649063+0000", 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649063+0000", 2024-04-25T05:38:19.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649064+0000", 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8100000000000003e-07 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649069+0000", 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1619999999999999e-06 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:19.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649080+0000", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1148000000000001e-05 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649087+0000", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2289999999999997e-06 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649143+0000", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5816999999999998e-05 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:19.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:19.649163+0000", 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9673999999999998e-05 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 4 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:38:19.669 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:19.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:19.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:19.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:19.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:19.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:20.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.044 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:38:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:20.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:38:20.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:20.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.336 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:38:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:38:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: osdmap e164: 8 total, 8 up, 6 in 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1483446382' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: osdmap e164: 8 total, 8 up, 6 in 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1483446382' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:38:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:38:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: Health check update: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-04-25T05:38:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: osdmap e164: 8 total, 8 up, 6 in 2024-04-25T05:38:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1483446382' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:20.575 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:20.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:20.758 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:38:20.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:20.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:38:20.880 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:38:20.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:20.953 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:20.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:21.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:21.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:21.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:21.220 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:21.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:21.342 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[27010]: pgmap v202: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 502 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[27010]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[27010]: osdmap e165: 8 total, 8 up, 6 in 2024-04-25T05:38:21.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[37492]: pgmap v202: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 502 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:38:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[37492]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-25T05:38:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:21 smithi043 ceph-mon[37492]: osdmap e165: 8 total, 8 up, 6 in 2024-04-25T05:38:21.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:21.477 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:21 smithi060 ceph-mon[30898]: pgmap v202: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 502 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:21 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:38:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:21 smithi060 ceph-mon[30898]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 7 pgs peering) 2024-04-25T05:38:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:21 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:21 smithi060 ceph-mon[30898]: osdmap e165: 8 total, 8 up, 6 in 2024-04-25T05:38:21.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:21.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:21.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:21.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:21.809 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:21.851 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:21.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:21.896 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:21.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:21.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:21.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:38:21.947 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:21.987 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:22.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:22.031 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:22.081 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:22.122 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:22.133 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:22.133 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:22.133 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:22.133 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:22.133 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:22.134 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:22.134 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:22.134 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:22.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:22.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:22.243 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:22.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:22 smithi043 ceph-mon[27010]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:38:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/944216518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:22.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:22 smithi043 ceph-mon[37492]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:38:22.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/944216518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:22.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:22.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:22 smithi060 ceph-mon[30898]: from='client.24974 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:38:22.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/944216518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:22.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:22.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:22.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:22.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:22.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:22.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:22.850 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:22.864 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:22.864 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:22.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:22.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:22.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:22.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:23.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:23.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:23.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:23.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:23.366 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[27010]: pgmap v205: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 501 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[27010]: osdmap e166: 8 total, 8 up, 6 in 2024-04-25T05:38:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:38:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:38:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[37492]: pgmap v205: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 501 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[37492]: osdmap e166: 8 total, 8 up, 6 in 2024-04-25T05:38:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:38:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:23 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:38:23.468 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:23.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:23 smithi060 ceph-mon[30898]: pgmap v205: 148 pgs: 8 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 501 op/s; 19876/145107 objects misplaced (13.697%); 47 MiB/s, 771 objects/s recovering 2024-04-25T05:38:23.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:23 smithi060 ceph-mon[30898]: osdmap e166: 8 total, 8 up, 6 in 2024-04-25T05:38:23.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:23 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:38:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:23 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:38:23.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:23.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:23.676 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-04-25T05:38:20.741001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3155368499,"num_objects":48369,"num_object_clones":0,"num_object_copies":145107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19876,"num_objects_unfound":0,"num_objects_dirty":48369,"num_whiteouts":0,"num_read":1465,"num_read_kb":1279,"num_write":186402,"num_write_kb":7481886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36170,"num_bytes_recovered":2360286616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64739,"ondisk_log_size":64739,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":439,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8870568,"kb_used_data":7980964,"kb_used_omap":9,"kb_used_meta":889462,"kb_avail":553600344,"statfs":{"total":575970213888,"available":566886752256,"internally_reserved":0,"allocated":8172507136,"data_stored":8169820805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":910809765},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-197328896,"num_objects":-3011,"num_object_clones":0,"num_object_copies":-9033,"num_objects_missing_on_primary":0,"num_objects_missing":-4,"num_objects_degraded":0,"num_objects_misplaced":19876,"num_objects_unfound":0,"num_objects_dirty":-3011,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4636,"num_bytes_recovered":298191584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005812"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":199,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745646+0000","last_change":"2024-04-25T05:37:49.635619+0000","last_active":"2024-04-25T05:38:15.745646+0000","last_peered":"2024-04-25T05:38:15.745646+0000","last_clean":"2024-04-25T05:38:15.745646+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:38:15.745646+0000","last_undegraded":"2024-04-25T05:38:15.745646+0000","last_fullsized":"2024-04-25T05:38:15.745646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:49.635538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:49.635538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:55.249500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743415+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:38:15.743415+0000","last_peered":"2024-04-25T05:38:15.743415+0000","last_clean":"2024-04-25T05:38:15.743415+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:38:15.743415+0000","last_undegraded":"2024-04-25T05:38:15.743415+0000","last_fullsized":"2024-04-25T05:38:15.743415+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741511+0000","last_change":"2024-04-25T05:38:13.237819+0000","last_active":"2024-04-25T05:38:15.741511+0000","last_peered":"2024-04-25T05:38:15.741511+0000","last_clean":"2024-04-25T05:38:15.741511+0000","last_became_active":"2024-04-25T05:38:13.237423+0000","last_became_peered":"2024-04-25T05:38:13.237423+0000","last_unstale":"2024-04-25T05:38:15.741511+0000","last_undegraded":"2024-04-25T05:38:15.741511+0000","last_fullsized":"2024-04-25T05:38:15.741511+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:18.176394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.176394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":253,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744983+0000","last_change":"2024-04-25T05:36:54.265352+0000","last_active":"2024-04-25T05:38:15.744983+0000","last_peered":"2024-04-25T05:38:15.744983+0000","last_clean":"2024-04-25T05:38:15.744983+0000","last_became_active":"2024-04-25T05:36:54.243668+0000","last_became_peered":"2024-04-25T05:36:54.243668+0000","last_unstale":"2024-04-25T05:38:15.744983+0000","last_undegraded":"2024-04-25T05:38:15.744983+0000","last_fullsized":"2024-04-25T05:38:15.744983+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703349+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":198,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746896+0000","last_change":"2024-04-25T05:36:54.236185+0000","last_active":"2024-04-25T05:38:15.746896+0000","last_peered":"2024-04-25T05:38:15.746896+0000","last_clean":"2024-04-25T05:38:15.746896+0000","last_became_active":"2024-04-25T05:36:54.235695+0000","last_became_peered":"2024-04-25T05:36:54.235695+0000","last_unstale":"2024-04-25T05:38:15.746896+0000","last_undegraded":"2024-04-25T05:38:15.746896+0000","last_fullsized":"2024-04-25T05:38:15.746896+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.866961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":209,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745106+0000","last_change":"2024-04-25T05:37:47.617448+0000","last_active":"2024-04-25T05:38:15.745106+0000","last_peered":"2024-04-25T05:38:15.745106+0000","last_clean":"2024-04-25T05:38:15.745106+0000","last_became_active":"2024-04-25T05:36:54.229992+0000","last_became_peered":"2024-04-25T05:36:54.229992+0000","last_unstale":"2024-04-25T05:38:15.745106+0000","last_undegraded":"2024-04-25T05:38:15.745106+0000","last_fullsized":"2024-04-25T05:38:15.745106+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:47.617351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:47.617351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:16.683166+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749006+0000","last_change":"2024-04-25T05:36:54.230655+0000","last_active":"2024-04-25T05:38:15.749006+0000","last_peered":"2024-04-25T05:38:15.749006+0000","last_clean":"2024-04-25T05:38:15.749006+0000","last_became_active":"2024-04-25T05:36:54.230169+0000","last_became_peered":"2024-04-25T05:36:54.230169+0000","last_unstale":"2024-04-25T05:38:15.749006+0000","last_undegraded":"2024-04-25T05:38:15.749006+0000","last_fullsized":"2024-04-25T05:38:15.749006+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867176+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":303,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746876+0000","last_change":"2024-04-25T05:38:13.261747+0000","last_active":"2024-04-25T05:38:15.746876+0000","last_peered":"2024-04-25T05:38:15.746876+0000","last_clean":"2024-04-25T05:38:15.746876+0000","last_became_active":"2024-04-25T05:38:13.246798+0000","last_became_peered":"2024-04-25T05:38:13.246798+0000","last_unstale":"2024-04-25T05:38:15.746876+0000","last_undegraded":"2024-04-25T05:38:15.746876+0000","last_fullsized":"2024-04-25T05:38:15.746876+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741135+0000","last_change":"2024-04-25T05:38:13.304611+0000","last_active":"2024-04-25T05:38:15.741135+0000","last_peered":"2024-04-25T05:38:15.741135+0000","last_clean":"2024-04-25T05:38:15.741135+0000","last_became_active":"2024-04-25T05:38:13.304062+0000","last_became_peered":"2024-04-25T05:38:13.304062+0000","last_unstale":"2024-04-25T05:38:15.741135+0000","last_undegraded":"2024-04-25T05:38:15.741135+0000","last_fullsized":"2024-04-25T05:38:15.741135+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:32.284637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.284637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":210,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740622+0000","last_change":"2024-04-25T05:38:13.239184+0000","last_active":"2024-04-25T05:38:15.740622+0000","last_peered":"2024-04-25T05:38:15.740622+0000","last_clean":"2024-04-25T05:38:15.740622+0000","last_became_active":"2024-04-25T05:38:13.237479+0000","last_became_peered":"2024-04-25T05:38:13.237479+0000","last_unstale":"2024-04-25T05:38:15.740622+0000","last_undegraded":"2024-04-25T05:38:15.740622+0000","last_fullsized":"2024-04-25T05:38:15.740622+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740785+0000","last_change":"2024-04-25T05:36:54.244217+0000","last_active":"2024-04-25T05:38:15.740785+0000","last_peered":"2024-04-25T05:38:15.740785+0000","last_clean":"2024-04-25T05:38:15.740785+0000","last_became_active":"2024-04-25T05:36:54.243924+0000","last_became_peered":"2024-04-25T05:36:54.243924+0000","last_unstale":"2024-04-25T05:38:15.740785+0000","last_undegraded":"2024-04-25T05:38:15.740785+0000","last_fullsized":"2024-04-25T05:38:15.740785+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":268,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745969+0000","last_change":"2024-04-25T05:38:13.254897+0000","last_active":"2024-04-25T05:38:15.745969+0000","last_peered":"2024-04-25T05:38:15.745969+0000","last_clean":"2024-04-25T05:38:15.745969+0000","last_became_active":"2024-04-25T05:38:13.239434+0000","last_became_peered":"2024-04-25T05:38:13.239434+0000","last_unstale":"2024-04-25T05:38:15.745969+0000","last_undegraded":"2024-04-25T05:38:15.745969+0000","last_fullsized":"2024-04-25T05:38:15.745969+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.536654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":208,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.742187+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:38:15.742187+0000","last_peered":"2024-04-25T05:38:15.742187+0000","last_clean":"2024-04-25T05:38:15.742187+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:38:15.742187+0000","last_undegraded":"2024-04-25T05:38:15.742187+0000","last_fullsized":"2024-04-25T05:38:15.742187+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:55.892919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744510+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:38:15.744510+0000","last_peered":"2024-04-25T05:38:15.744510+0000","last_clean":"2024-04-25T05:38:15.744510+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:38:15.744510+0000","last_undegraded":"2024-04-25T05:38:15.744510+0000","last_fullsized":"2024-04-25T05:38:15.744510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:59.764698+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741618+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:38:15.741618+0000","last_peered":"2024-04-25T05:38:15.741618+0000","last_clean":"2024-04-25T05:38:15.741618+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:38:15.741618+0000","last_undegraded":"2024-04-25T05:38:15.741618+0000","last_fullsized":"2024-04-25T05:38:15.741618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746885+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:38:15.746885+0000","last_peered":"2024-04-25T05:38:15.746885+0000","last_clean":"2024-04-25T05:38:15.746885+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:38:15.746885+0000","last_undegraded":"2024-04-25T05:38:15.746885+0000","last_fullsized":"2024-04-25T05:38:15.746885+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703424+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":206,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746531+0000","last_change":"2024-04-25T05:36:54.245109+0000","last_active":"2024-04-25T05:38:15.746531+0000","last_peered":"2024-04-25T05:38:15.746531+0000","last_clean":"2024-04-25T05:38:15.746531+0000","last_became_active":"2024-04-25T05:36:54.244292+0000","last_became_peered":"2024-04-25T05:36:54.244292+0000","last_unstale":"2024-04-25T05:38:15.746531+0000","last_undegraded":"2024-04-25T05:38:15.746531+0000","last_fullsized":"2024-04-25T05:38:15.746531+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:19.691117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.751567+0000","last_change":"2024-04-25T05:38:13.285789+0000","last_active":"2024-04-25T05:38:15.751567+0000","last_peered":"2024-04-25T05:38:15.751567+0000","last_clean":"2024-04-25T05:38:15.751567+0000","last_became_active":"2024-04-25T05:38:13.285434+0000","last_became_peered":"2024-04-25T05:38:13.285434+0000","last_unstale":"2024-04-25T05:38:15.751567+0000","last_undegraded":"2024-04-25T05:38:15.751567+0000","last_fullsized":"2024-04-25T05:38:15.751567+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746846+0000","last_change":"2024-04-25T05:38:13.289898+0000","last_active":"2024-04-25T05:38:15.746846+0000","last_peered":"2024-04-25T05:38:15.746846+0000","last_clean":"2024-04-25T05:38:15.746846+0000","last_became_active":"2024-04-25T05:38:13.289539+0000","last_became_peered":"2024-04-25T05:38:13.289539+0000","last_unstale":"2024-04-25T05:38:15.746846+0000","last_undegraded":"2024-04-25T05:38:15.746846+0000","last_fullsized":"2024-04-25T05:38:15.746846+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":247,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749285+0000","last_change":"2024-04-25T05:37:50.872876+0000","last_active":"2024-04-25T05:38:15.749285+0000","last_peered":"2024-04-25T05:38:15.749285+0000","last_clean":"2024-04-25T05:38:15.749285+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:38:15.749285+0000","last_undegraded":"2024-04-25T05:38:15.749285+0000","last_fullsized":"2024-04-25T05:38:15.749285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:37:50.872688+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:00.785066+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":199,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.754529+0000","last_change":"2024-04-25T05:36:54.237606+0000","last_active":"2024-04-25T05:38:15.754529+0000","last_peered":"2024-04-25T05:38:15.754529+0000","last_clean":"2024-04-25T05:38:15.754529+0000","last_became_active":"2024-04-25T05:36:54.237282+0000","last_became_peered":"2024-04-25T05:36:54.237282+0000","last_unstale":"2024-04-25T05:38:15.754529+0000","last_undegraded":"2024-04-25T05:38:15.754529+0000","last_fullsized":"2024-04-25T05:38:15.754529+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.866942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":195,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748064+0000","last_change":"2024-04-25T05:34:02.706601+0000","last_active":"2024-04-25T05:38:15.748064+0000","last_peered":"2024-04-25T05:38:15.748064+0000","last_clean":"2024-04-25T05:38:15.748064+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:38:15.748064+0000","last_undegraded":"2024-04-25T05:38:15.748064+0000","last_fullsized":"2024-04-25T05:38:15.748064+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":211,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746079+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:38:15.746079+0000","last_peered":"2024-04-25T05:38:15.746079+0000","last_clean":"2024-04-25T05:38:15.746079+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:38:15.746079+0000","last_undegraded":"2024-04-25T05:38:15.746079+0000","last_fullsized":"2024-04-25T05:38:15.746079+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703487+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747775+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:38:15.747775+0000","last_peered":"2024-04-25T05:38:15.747775+0000","last_clean":"2024-04-25T05:38:15.747775+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:38:15.747775+0000","last_undegraded":"2024-04-25T05:38:15.747775+0000","last_fullsized":"2024-04-25T05:38:15.747775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.866995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":209,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743953+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:38:15.743953+0000","last_peered":"2024-04-25T05:38:15.743953+0000","last_clean":"2024-04-25T05:38:15.743953+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:38:15.743953+0000","last_undegraded":"2024-04-25T05:38:15.743953+0000","last_fullsized":"2024-04-25T05:38:15.743953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"130'9","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747585+0000","last_change":"2024-04-25T05:36:54.240381+0000","last_active":"2024-04-25T05:38:15.747585+0000","last_peered":"2024-04-25T05:38:15.747585+0000","last_clean":"2024-04-25T05:38:15.747585+0000","last_became_active":"2024-04-25T05:36:54.232031+0000","last_became_peered":"2024-04-25T05:36:54.232031+0000","last_unstale":"2024-04-25T05:38:15.747585+0000","last_undegraded":"2024-04-25T05:38:15.747585+0000","last_fullsized":"2024-04-25T05:38:15.747585+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743325+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:38:15.743325+0000","last_peered":"2024-04-25T05:38:15.743325+0000","last_clean":"2024-04-25T05:38:15.743325+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:38:15.743325+0000","last_undegraded":"2024-04-25T05:38:15.743325+0000","last_fullsized":"2024-04-25T05:38:15.743325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113501+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'16","reported_seq":78,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746105+0000","last_change":"2024-04-25T05:38:13.313436+0000","last_active":"2024-04-25T05:38:15.746105+0000","last_peered":"2024-04-25T05:38:15.746105+0000","last_clean":"2024-04-25T05:38:15.746105+0000","last_became_active":"2024-04-25T05:38:13.281967+0000","last_became_peered":"2024-04-25T05:38:13.281967+0000","last_unstale":"2024-04-25T05:38:15.746105+0000","last_undegraded":"2024-04-25T05:38:15.746105+0000","last_fullsized":"2024-04-25T05:38:15.746105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:36:15.128887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.128887+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":200,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.751300+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:38:15.751300+0000","last_peered":"2024-04-25T05:38:15.751300+0000","last_clean":"2024-04-25T05:38:15.751300+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:38:15.751300+0000","last_undegraded":"2024-04-25T05:38:15.751300+0000","last_fullsized":"2024-04-25T05:38:15.751300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745230+0000","last_change":"2024-04-25T05:38:13.282576+0000","last_active":"2024-04-25T05:38:15.745230+0000","last_peered":"2024-04-25T05:38:15.745230+0000","last_clean":"2024-04-25T05:38:15.745230+0000","last_became_active":"2024-04-25T05:38:13.282204+0000","last_became_peered":"2024-04-25T05:38:13.282204+0000","last_unstale":"2024-04-25T05:38:15.745230+0000","last_undegraded":"2024-04-25T05:38:15.745230+0000","last_fullsized":"2024-04-25T05:38:15.745230+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.171349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.171349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747001+0000","last_change":"2024-04-25T05:38:13.305988+0000","last_active":"2024-04-25T05:38:15.747001+0000","last_peered":"2024-04-25T05:38:15.747001+0000","last_clean":"2024-04-25T05:38:15.747001+0000","last_became_active":"2024-04-25T05:38:13.305466+0000","last_became_peered":"2024-04-25T05:38:13.305466+0000","last_unstale":"2024-04-25T05:38:15.747001+0000","last_undegraded":"2024-04-25T05:38:15.747001+0000","last_fullsized":"2024-04-25T05:38:15.747001+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":170,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741765+0000","last_change":"2024-04-25T05:38:13.323005+0000","last_active":"2024-04-25T05:38:15.741765+0000","last_peered":"2024-04-25T05:38:15.741765+0000","last_clean":"2024-04-25T05:38:15.741765+0000","last_became_active":"2024-04-25T05:38:13.297139+0000","last_became_peered":"2024-04-25T05:38:13.297139+0000","last_unstale":"2024-04-25T05:38:15.741765+0000","last_undegraded":"2024-04-25T05:38:15.741765+0000","last_fullsized":"2024-04-25T05:38:15.741765+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:37:35.189581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:35.189581+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:47.037055+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":209,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743314+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:38:15.743314+0000","last_peered":"2024-04-25T05:38:15.743314+0000","last_clean":"2024-04-25T05:38:15.743314+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:38:15.743314+0000","last_undegraded":"2024-04-25T05:38:15.743314+0000","last_fullsized":"2024-04-25T05:38:15.743314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:44.881845+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":85,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745221+0000","last_change":"2024-04-25T05:38:13.290337+0000","last_active":"2024-04-25T05:38:15.745221+0000","last_peered":"2024-04-25T05:38:15.745221+0000","last_clean":"2024-04-25T05:38:15.745221+0000","last_became_active":"2024-04-25T05:38:13.289541+0000","last_became_peered":"2024-04-25T05:38:13.289541+0000","last_unstale":"2024-04-25T05:38:15.745221+0000","last_undegraded":"2024-04-25T05:38:15.745221+0000","last_fullsized":"2024-04-25T05:38:15.745221+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741607+0000","last_change":"2024-04-25T05:38:13.296470+0000","last_active":"2024-04-25T05:38:15.741607+0000","last_peered":"2024-04-25T05:38:15.741607+0000","last_clean":"2024-04-25T05:38:15.741607+0000","last_became_active":"2024-04-25T05:38:13.295979+0000","last_became_peered":"2024-04-25T05:38:13.295979+0000","last_unstale":"2024-04-25T05:38:15.741607+0000","last_undegraded":"2024-04-25T05:38:15.741607+0000","last_fullsized":"2024-04-25T05:38:15.741607+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747094+0000","last_change":"2024-04-25T05:36:54.262135+0000","last_active":"2024-04-25T05:38:15.747094+0000","last_peered":"2024-04-25T05:38:15.747094+0000","last_clean":"2024-04-25T05:38:15.747094+0000","last_became_active":"2024-04-25T05:36:54.244002+0000","last_became_peered":"2024-04-25T05:36:54.244002+0000","last_unstale":"2024-04-25T05:38:15.747094+0000","last_undegraded":"2024-04-25T05:38:15.747094+0000","last_fullsized":"2024-04-25T05:38:15.747094+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867209+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748541+0000","last_change":"2024-04-25T05:38:13.246015+0000","last_active":"2024-04-25T05:38:15.748541+0000","last_peered":"2024-04-25T05:38:15.748541+0000","last_clean":"2024-04-25T05:38:15.748541+0000","last_became_active":"2024-04-25T05:38:13.245676+0000","last_became_peered":"2024-04-25T05:38:13.245676+0000","last_unstale":"2024-04-25T05:38:15.748541+0000","last_undegraded":"2024-04-25T05:38:15.748541+0000","last_fullsized":"2024-04-25T05:38:15.748541+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746529+0000","last_change":"2024-04-25T05:38:13.245873+0000","last_active":"2024-04-25T05:38:15.746529+0000","last_peered":"2024-04-25T05:38:15.746529+0000","last_clean":"2024-04-25T05:38:15.746529+0000","last_became_active":"2024-04-25T05:38:13.245293+0000","last_became_peered":"2024-04-25T05:38:13.245293+0000","last_unstale":"2024-04-25T05:38:15.746529+0000","last_undegraded":"2024-04-25T05:38:15.746529+0000","last_fullsized":"2024-04-25T05:38:15.746529+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:21.131800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740784+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:38:15.740784+0000","last_peered":"2024-04-25T05:38:15.740784+0000","last_clean":"2024-04-25T05:38:15.740784+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:38:15.740784+0000","last_undegraded":"2024-04-25T05:38:15.740784+0000","last_fullsized":"2024-04-25T05:38:15.740784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":259,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746760+0000","last_change":"2024-04-25T05:36:54.252887+0000","last_active":"2024-04-25T05:38:15.746760+0000","last_peered":"2024-04-25T05:38:15.746760+0000","last_clean":"2024-04-25T05:38:15.746760+0000","last_became_active":"2024-04-25T05:36:54.232522+0000","last_became_peered":"2024-04-25T05:36:54.232522+0000","last_unstale":"2024-04-25T05:38:15.746760+0000","last_undegraded":"2024-04-25T05:38:15.746760+0000","last_fullsized":"2024-04-25T05:38:15.746760+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867159+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740704+0000","last_change":"2024-04-25T05:38:13.306123+0000","last_active":"2024-04-25T05:38:15.740704+0000","last_peered":"2024-04-25T05:38:15.740704+0000","last_clean":"2024-04-25T05:38:15.740704+0000","last_became_active":"2024-04-25T05:38:13.305781+0000","last_became_peered":"2024-04-25T05:38:13.305781+0000","last_unstale":"2024-04-25T05:38:15.740704+0000","last_undegraded":"2024-04-25T05:38:15.740704+0000","last_fullsized":"2024-04-25T05:38:15.740704+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:34.276673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:34.276673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":202,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746788+0000","last_change":"2024-04-25T05:38:13.287078+0000","last_active":"2024-04-25T05:38:15.746788+0000","last_peered":"2024-04-25T05:38:15.746788+0000","last_clean":"2024-04-25T05:38:15.746788+0000","last_became_active":"2024-04-25T05:38:13.286756+0000","last_became_peered":"2024-04-25T05:38:13.286756+0000","last_unstale":"2024-04-25T05:38:15.746788+0000","last_undegraded":"2024-04-25T05:38:15.746788+0000","last_fullsized":"2024-04-25T05:38:15.746788+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745599+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:38:15.745599+0000","last_peered":"2024-04-25T05:38:15.745599+0000","last_clean":"2024-04-25T05:38:15.745599+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:38:15.745599+0000","last_undegraded":"2024-04-25T05:38:15.745599+0000","last_fullsized":"2024-04-25T05:38:15.745599+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":27,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745284+0000","last_change":"2024-04-25T05:38:13.316204+0000","last_active":"2024-04-25T05:38:15.745284+0000","last_peered":"2024-04-25T05:38:15.745284+0000","last_clean":"2024-04-25T05:38:15.745284+0000","last_became_active":"2024-04-25T05:38:13.296338+0000","last_became_peered":"2024-04-25T05:38:13.296338+0000","last_unstale":"2024-04-25T05:38:15.745284+0000","last_undegraded":"2024-04-25T05:38:15.745284+0000","last_fullsized":"2024-04-25T05:38:15.745284+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:46.654977+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":198,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747042+0000","last_change":"2024-04-25T05:38:13.247491+0000","last_active":"2024-04-25T05:38:15.747042+0000","last_peered":"2024-04-25T05:38:15.747042+0000","last_clean":"2024-04-25T05:38:15.747042+0000","last_became_active":"2024-04-25T05:38:13.246584+0000","last_became_peered":"2024-04-25T05:38:13.246584+0000","last_unstale":"2024-04-25T05:38:15.747042+0000","last_undegraded":"2024-04-25T05:38:15.747042+0000","last_fullsized":"2024-04-25T05:38:15.747042+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745193+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:38:15.745193+0000","last_peered":"2024-04-25T05:38:15.745193+0000","last_clean":"2024-04-25T05:38:15.745193+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:38:15.745193+0000","last_undegraded":"2024-04-25T05:38:15.745193+0000","last_fullsized":"2024-04-25T05:38:15.745193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113436+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747787+0000","last_change":"2024-04-25T05:36:54.240953+0000","last_active":"2024-04-25T05:38:15.747787+0000","last_peered":"2024-04-25T05:38:15.747787+0000","last_clean":"2024-04-25T05:38:15.747787+0000","last_became_active":"2024-04-25T05:36:54.235617+0000","last_became_peered":"2024-04-25T05:36:54.235617+0000","last_unstale":"2024-04-25T05:38:15.747787+0000","last_undegraded":"2024-04-25T05:38:15.747787+0000","last_fullsized":"2024-04-25T05:38:15.747787+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:35:02.707330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:02.707330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747114+0000","last_change":"2024-04-25T05:36:54.258480+0000","last_active":"2024-04-25T05:38:15.747114+0000","last_peered":"2024-04-25T05:38:15.747114+0000","last_clean":"2024-04-25T05:38:15.747114+0000","last_became_active":"2024-04-25T05:36:54.236292+0000","last_became_peered":"2024-04-25T05:36:54.236292+0000","last_unstale":"2024-04-25T05:38:15.747114+0000","last_undegraded":"2024-04-25T05:38:15.747114+0000","last_fullsized":"2024-04-25T05:38:15.747114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:35:07.728528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:07.728528+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744804+0000","last_change":"2024-04-25T05:38:13.246984+0000","last_active":"2024-04-25T05:38:15.744804+0000","last_peered":"2024-04-25T05:38:15.744804+0000","last_clean":"2024-04-25T05:38:15.744804+0000","last_became_active":"2024-04-25T05:38:13.246544+0000","last_became_peered":"2024-04-25T05:38:13.246544+0000","last_unstale":"2024-04-25T05:38:15.744804+0000","last_undegraded":"2024-04-25T05:38:15.744804+0000","last_fullsized":"2024-04-25T05:38:15.744804+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"160'9","reported_seq":275,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741014+0000","last_change":"2024-04-25T05:38:13.305236+0000","last_active":"2024-04-25T05:38:15.741014+0000","last_peered":"2024-04-25T05:38:15.741014+0000","last_clean":"2024-04-25T05:38:15.741014+0000","last_became_active":"2024-04-25T05:38:13.295760+0000","last_became_peered":"2024-04-25T05:38:13.295760+0000","last_unstale":"2024-04-25T05:38:15.741014+0000","last_undegraded":"2024-04-25T05:38:15.741014+0000","last_fullsized":"2024-04-25T05:38:15.741014+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:39.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.823449+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743759+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:38:15.743759+0000","last_peered":"2024-04-25T05:38:15.743759+0000","last_clean":"2024-04-25T05:38:15.743759+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:38:15.743759+0000","last_undegraded":"2024-04-25T05:38:15.743759+0000","last_fullsized":"2024-04-25T05:38:15.743759+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:51.933293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":247,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747892+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:38:15.747892+0000","last_peered":"2024-04-25T05:38:15.747892+0000","last_clean":"2024-04-25T05:38:15.747892+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:38:15.747892+0000","last_undegraded":"2024-04-25T05:38:15.747892+0000","last_fullsized":"2024-04-25T05:38:15.747892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867246+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":199,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747723+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:38:15.747723+0000","last_peered":"2024-04-25T05:38:15.747723+0000","last_clean":"2024-04-25T05:38:15.747723+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:38:15.747723+0000","last_undegraded":"2024-04-25T05:38:15.747723+0000","last_fullsized":"2024-04-25T05:38:15.747723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":256,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749262+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:38:15.749262+0000","last_peered":"2024-04-25T05:38:15.749262+0000","last_clean":"2024-04-25T05:38:15.749262+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:38:15.749262+0000","last_undegraded":"2024-04-25T05:38:15.749262+0000","last_fullsized":"2024-04-25T05:38:15.749262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744611+0000","last_change":"2024-04-25T05:38:13.282321+0000","last_active":"2024-04-25T05:38:15.744611+0000","last_peered":"2024-04-25T05:38:15.744611+0000","last_clean":"2024-04-25T05:38:15.744611+0000","last_became_active":"2024-04-25T05:38:13.282059+0000","last_became_peered":"2024-04-25T05:38:13.282059+0000","last_unstale":"2024-04-25T05:38:15.744611+0000","last_undegraded":"2024-04-25T05:38:15.744611+0000","last_fullsized":"2024-04-25T05:38:15.744611+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":241,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741675+0000","last_change":"2024-04-25T05:35:40.845196+0000","last_active":"2024-04-25T05:38:15.741675+0000","last_peered":"2024-04-25T05:38:15.741675+0000","last_clean":"2024-04-25T05:38:15.741675+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:38:15.741675+0000","last_undegraded":"2024-04-25T05:38:15.741675+0000","last_fullsized":"2024-04-25T05:38:15.741675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.845036+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":207,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749604+0000","last_change":"2024-04-25T05:38:13.947132+0000","last_active":"2024-04-25T05:38:15.749604+0000","last_peered":"2024-04-25T05:38:15.749604+0000","last_clean":"2024-04-25T05:38:15.749604+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:38:15.749604+0000","last_undegraded":"2024-04-25T05:38:15.749604+0000","last_fullsized":"2024-04-25T05:38:15.749604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:13.947044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:13.947044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:19.521993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":196,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.754158+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:38:15.754158+0000","last_peered":"2024-04-25T05:38:15.754158+0000","last_clean":"2024-04-25T05:38:15.754158+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:38:15.754158+0000","last_undegraded":"2024-04-25T05:38:15.754158+0000","last_fullsized":"2024-04-25T05:38:15.754158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743804+0000","last_change":"2024-04-25T05:38:13.241002+0000","last_active":"2024-04-25T05:38:15.743804+0000","last_peered":"2024-04-25T05:38:15.743804+0000","last_clean":"2024-04-25T05:38:15.743804+0000","last_became_active":"2024-04-25T05:38:13.240401+0000","last_became_peered":"2024-04-25T05:38:13.240401+0000","last_unstale":"2024-04-25T05:38:15.743804+0000","last_undegraded":"2024-04-25T05:38:15.743804+0000","last_fullsized":"2024-04-25T05:38:15.743804+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:30.787863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":91,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749241+0000","last_change":"2024-04-25T05:38:13.255668+0000","last_active":"2024-04-25T05:38:15.749241+0000","last_peered":"2024-04-25T05:38:15.749241+0000","last_clean":"2024-04-25T05:38:15.749241+0000","last_became_active":"2024-04-25T05:38:13.245172+0000","last_became_peered":"2024-04-25T05:38:13.245172+0000","last_unstale":"2024-04-25T05:38:15.749241+0000","last_undegraded":"2024-04-25T05:38:15.749241+0000","last_fullsized":"2024-04-25T05:38:15.749241+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":214,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745508+0000","last_change":"2024-04-25T05:37:41.064307+0000","last_active":"2024-04-25T05:38:15.745508+0000","last_peered":"2024-04-25T05:38:15.745508+0000","last_clean":"2024-04-25T05:38:15.745508+0000","last_became_active":"2024-04-25T05:36:54.232503+0000","last_became_peered":"2024-04-25T05:36:54.232503+0000","last_unstale":"2024-04-25T05:38:15.745508+0000","last_undegraded":"2024-04-25T05:38:15.745508+0000","last_fullsized":"2024-04-25T05:38:15.745508+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:59.335848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740687+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:38:15.740687+0000","last_peered":"2024-04-25T05:38:15.740687+0000","last_clean":"2024-04-25T05:38:15.740687+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:38:15.740687+0000","last_undegraded":"2024-04-25T05:38:15.740687+0000","last_fullsized":"2024-04-25T05:38:15.740687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740648+0000","last_change":"2024-04-25T05:38:13.245394+0000","last_active":"2024-04-25T05:38:15.740648+0000","last_peered":"2024-04-25T05:38:15.740648+0000","last_clean":"2024-04-25T05:38:15.740648+0000","last_became_active":"2024-04-25T05:38:13.244520+0000","last_became_peered":"2024-04-25T05:38:13.244520+0000","last_unstale":"2024-04-25T05:38:15.740648+0000","last_undegraded":"2024-04-25T05:38:15.740648+0000","last_fullsized":"2024-04-25T05:38:15.740648+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.715360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.715360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.754643+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:38:15.754643+0000","last_peered":"2024-04-25T05:38:15.754643+0000","last_clean":"2024-04-25T05:38:15.754643+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:38:15.754643+0000","last_undegraded":"2024-04-25T05:38:15.754643+0000","last_fullsized":"2024-04-25T05:38:15.754643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"163'11517","reported_seq":22418,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.940110+0000","last_change":"2024-04-25T05:37:50.115871+0000","last_active":"2024-04-25T05:38:17.940110+0000","last_peered":"2024-04-25T05:38:17.940110+0000","last_clean":"2024-04-25T05:38:17.940110+0000","last_became_active":"2024-04-25T05:37:50.089090+0000","last_became_peered":"2024-04-25T05:37:50.089090+0000","last_unstale":"2024-04-25T05:38:17.940110+0000","last_undegraded":"2024-04-25T05:38:17.940110+0000","last_fullsized":"2024-04-25T05:38:17.940110+0000","mapping_epoch":149,"log_start":"134'7932","ondisk_log_start":"134'7932","created":88,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'5609","last_scrub_stamp":"2024-04-25T05:36:18.461345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:18.461345+0000","objects_scrubbed":5551,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113469+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189333504,"num_objects":2889,"num_object_clones":0,"num_object_copies":8667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11517,"num_write_kb":460992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741475+0000","last_change":"2024-04-25T05:38:13.285501+0000","last_active":"2024-04-25T05:38:15.741475+0000","last_peered":"2024-04-25T05:38:15.741475+0000","last_clean":"2024-04-25T05:38:15.741475+0000","last_became_active":"2024-04-25T05:38:13.284963+0000","last_became_peered":"2024-04-25T05:38:13.284963+0000","last_unstale":"2024-04-25T05:38:15.741475+0000","last_undegraded":"2024-04-25T05:38:15.741475+0000","last_fullsized":"2024-04-25T05:38:15.741475+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745253+0000","last_change":"2024-04-25T05:38:13.313381+0000","last_active":"2024-04-25T05:38:15.745253+0000","last_peered":"2024-04-25T05:38:15.745253+0000","last_clean":"2024-04-25T05:38:15.745253+0000","last_became_active":"2024-04-25T05:38:13.295320+0000","last_became_peered":"2024-04-25T05:38:13.295320+0000","last_unstale":"2024-04-25T05:38:15.745253+0000","last_undegraded":"2024-04-25T05:38:15.745253+0000","last_fullsized":"2024-04-25T05:38:15.745253+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:56.506745+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740514+0000","last_change":"2024-04-25T05:36:54.267732+0000","last_active":"2024-04-25T05:38:15.740514+0000","last_peered":"2024-04-25T05:38:15.740514+0000","last_clean":"2024-04-25T05:38:15.740514+0000","last_became_active":"2024-04-25T05:36:54.244247+0000","last_became_peered":"2024-04-25T05:36:54.244247+0000","last_unstale":"2024-04-25T05:38:15.740514+0000","last_undegraded":"2024-04-25T05:38:15.740514+0000","last_fullsized":"2024-04-25T05:38:15.740514+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"130'10","reported_seq":283,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744062+0000","last_change":"2024-04-25T05:36:54.243037+0000","last_active":"2024-04-25T05:38:15.744062+0000","last_peered":"2024-04-25T05:38:15.744062+0000","last_clean":"2024-04-25T05:38:15.744062+0000","last_became_active":"2024-04-25T05:36:54.235542+0000","last_became_peered":"2024-04-25T05:36:54.235542+0000","last_unstale":"2024-04-25T05:38:15.744062+0000","last_undegraded":"2024-04-25T05:38:15.744062+0000","last_fullsized":"2024-04-25T05:38:15.744062+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"163'11623","reported_seq":11784,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.911726+0000","last_change":"2024-04-25T05:35:09.680981+0000","last_active":"2024-04-25T05:38:17.911726+0000","last_peered":"2024-04-25T05:38:17.911726+0000","last_clean":"2024-04-25T05:38:17.911726+0000","last_became_active":"2024-04-25T05:35:09.680271+0000","last_became_peered":"2024-04-25T05:35:09.680271+0000","last_unstale":"2024-04-25T05:38:17.911726+0000","last_undegraded":"2024-04-25T05:38:17.911726+0000","last_fullsized":"2024-04-25T05:38:17.911726+0000","mapping_epoch":88,"log_start":"134'8033","ondisk_log_start":"134'8033","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197197824,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11623,"num_write_kb":468224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":192,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746999+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:38:15.746999+0000","last_peered":"2024-04-25T05:38:15.746999+0000","last_clean":"2024-04-25T05:38:15.746999+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:38:15.746999+0000","last_undegraded":"2024-04-25T05:38:15.746999+0000","last_fullsized":"2024-04-25T05:38:15.746999+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.866977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745139+0000","last_change":"2024-04-25T05:36:48.522033+0000","last_active":"2024-04-25T05:38:15.745139+0000","last_peered":"2024-04-25T05:38:15.745139+0000","last_clean":"2024-04-25T05:38:15.745139+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:38:15.745139+0000","last_undegraded":"2024-04-25T05:38:15.745139+0000","last_fullsized":"2024-04-25T05:38:15.745139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:48.521908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:48.521908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":254,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746334+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:38:15.746334+0000","last_peered":"2024-04-25T05:38:15.746334+0000","last_clean":"2024-04-25T05:38:15.746334+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:38:15.746334+0000","last_undegraded":"2024-04-25T05:38:15.746334+0000","last_fullsized":"2024-04-25T05:38:15.746334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703257+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748745+0000","last_change":"2024-04-25T05:38:13.302895+0000","last_active":"2024-04-25T05:38:15.748745+0000","last_peered":"2024-04-25T05:38:15.748745+0000","last_clean":"2024-04-25T05:38:15.748745+0000","last_became_active":"2024-04-25T05:38:13.302285+0000","last_became_peered":"2024-04-25T05:38:13.302285+0000","last_unstale":"2024-04-25T05:38:15.748745+0000","last_undegraded":"2024-04-25T05:38:15.748745+0000","last_fullsized":"2024-04-25T05:38:15.748745+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:47.489244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.489244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"163'11561","reported_seq":12608,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.952347+0000","last_change":"2024-04-25T05:38:14.435534+0000","last_active":"2024-04-25T05:38:17.952347+0000","last_peered":"2024-04-25T05:38:17.952347+0000","last_clean":"2024-04-25T05:38:12.192902+0000","last_became_active":"2024-04-25T05:38:14.371489+0000","last_became_peered":"2024-04-25T05:38:14.371489+0000","last_unstale":"2024-04-25T05:38:17.952347+0000","last_undegraded":"2024-04-25T05:38:17.952347+0000","last_fullsized":"2024-04-25T05:38:17.952347+0000","mapping_epoch":160,"log_start":"134'7997","ondisk_log_start":"134'7997","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7637","last_scrub_stamp":"2024-04-25T05:36:54.004340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.004340+0000","objects_scrubbed":6885,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194052096,"num_objects":2961,"num_object_clones":0,"num_object_copies":8883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2570,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11561,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":26345472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":2961}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":200,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746936+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:38:15.746936+0000","last_peered":"2024-04-25T05:38:15.746936+0000","last_clean":"2024-04-25T05:38:15.746936+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:38:15.746936+0000","last_undegraded":"2024-04-25T05:38:15.746936+0000","last_fullsized":"2024-04-25T05:38:15.746936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748377+0000","last_change":"2024-04-25T05:38:13.236944+0000","last_active":"2024-04-25T05:38:15.748377+0000","last_peered":"2024-04-25T05:38:15.748377+0000","last_clean":"2024-04-25T05:38:15.748377+0000","last_became_active":"2024-04-25T05:38:13.236574+0000","last_became_peered":"2024-04-25T05:38:13.236574+0000","last_unstale":"2024-04-25T05:38:15.748377+0000","last_undegraded":"2024-04-25T05:38:15.748377+0000","last_fullsized":"2024-04-25T05:38:15.748377+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:51.573898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.573898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":236,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745818+0000","last_change":"2024-04-25T05:36:54.240931+0000","last_active":"2024-04-25T05:38:15.745818+0000","last_peered":"2024-04-25T05:38:15.745818+0000","last_clean":"2024-04-25T05:38:15.745818+0000","last_became_active":"2024-04-25T05:36:54.232602+0000","last_became_peered":"2024-04-25T05:36:54.232602+0000","last_unstale":"2024-04-25T05:38:15.745818+0000","last_undegraded":"2024-04-25T05:38:15.745818+0000","last_fullsized":"2024-04-25T05:38:15.745818+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":162,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745043+0000","last_change":"2024-04-25T05:36:54.260249+0000","last_active":"2024-04-25T05:38:15.745043+0000","last_peered":"2024-04-25T05:38:15.745043+0000","last_clean":"2024-04-25T05:38:15.745043+0000","last_became_active":"2024-04-25T05:36:54.235633+0000","last_became_peered":"2024-04-25T05:36:54.235633+0000","last_unstale":"2024-04-25T05:38:15.745043+0000","last_undegraded":"2024-04-25T05:38:15.745043+0000","last_fullsized":"2024-04-25T05:38:15.745043+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"163'11472","reported_seq":11703,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.894376+0000","last_change":"2024-04-25T05:36:31.742804+0000","last_active":"2024-04-25T05:38:17.894376+0000","last_peered":"2024-04-25T05:38:17.894376+0000","last_clean":"2024-04-25T05:38:17.894376+0000","last_became_active":"2024-04-25T05:35:09.680700+0000","last_became_peered":"2024-04-25T05:35:09.680700+0000","last_unstale":"2024-04-25T05:38:17.894376+0000","last_undegraded":"2024-04-25T05:38:17.894376+0000","last_fullsized":"2024-04-25T05:38:17.894376+0000","mapping_epoch":88,"log_start":"123'6554","ondisk_log_start":"123'6554","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6352","last_scrub_stamp":"2024-04-25T05:36:31.742451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.742451+0000","objects_scrubbed":6298,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743537+0000","last_change":"2024-04-25T05:38:13.240604+0000","last_active":"2024-04-25T05:38:15.743537+0000","last_peered":"2024-04-25T05:38:15.743537+0000","last_clean":"2024-04-25T05:38:15.743537+0000","last_became_active":"2024-04-25T05:38:13.240067+0000","last_became_peered":"2024-04-25T05:38:13.240067+0000","last_unstale":"2024-04-25T05:38:15.743537+0000","last_undegraded":"2024-04-25T05:38:15.743537+0000","last_fullsized":"2024-04-25T05:38:15.743537+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:27.172809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.172809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":203,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748666+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:38:15.748666+0000","last_peered":"2024-04-25T05:38:15.748666+0000","last_clean":"2024-04-25T05:38:15.748666+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:38:15.748666+0000","last_undegraded":"2024-04-25T05:38:15.748666+0000","last_fullsized":"2024-04-25T05:38:15.748666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741808+0000","last_change":"2024-04-25T05:37:30.237718+0000","last_active":"2024-04-25T05:38:15.741808+0000","last_peered":"2024-04-25T05:38:15.741808+0000","last_clean":"2024-04-25T05:38:15.741808+0000","last_became_active":"2024-04-25T05:36:54.234986+0000","last_became_peered":"2024-04-25T05:36:54.234986+0000","last_unstale":"2024-04-25T05:38:15.741808+0000","last_undegraded":"2024-04-25T05:38:15.741808+0000","last_fullsized":"2024-04-25T05:38:15.741808+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:46.447267+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":77,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746178+0000","last_change":"2024-04-25T05:38:13.317524+0000","last_active":"2024-04-25T05:38:15.746178+0000","last_peered":"2024-04-25T05:38:15.746178+0000","last_clean":"2024-04-25T05:38:15.746178+0000","last_became_active":"2024-04-25T05:38:13.294321+0000","last_became_peered":"2024-04-25T05:38:13.294321+0000","last_unstale":"2024-04-25T05:38:15.746178+0000","last_undegraded":"2024-04-25T05:38:15.746178+0000","last_fullsized":"2024-04-25T05:38:15.746178+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"163'11705","reported_seq":23273,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.966840+0000","last_change":"2024-04-25T05:38:14.402301+0000","last_active":"2024-04-25T05:38:17.966840+0000","last_peered":"2024-04-25T05:38:17.966840+0000","last_clean":"2024-04-25T05:38:12.181047+0000","last_became_active":"2024-04-25T05:38:14.357895+0000","last_became_peered":"2024-04-25T05:38:14.357895+0000","last_unstale":"2024-04-25T05:38:17.966840+0000","last_undegraded":"2024-04-25T05:38:17.966840+0000","last_fullsized":"2024-04-25T05:38:17.966840+0000","mapping_epoch":160,"log_start":"134'8052","ondisk_log_start":"134'8052","created":88,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202702848,"num_objects":3093,"num_object_clones":0,"num_object_copies":9279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2776,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11705,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5693,"num_bytes_recovered":373030912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,0],"avail_no_missing":["3","0","1"],"object_location_counts":[{"shards":"0,1,3","objects":3093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":200,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749346+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:38:15.749346+0000","last_peered":"2024-04-25T05:38:15.749346+0000","last_clean":"2024-04-25T05:38:15.749346+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:38:15.749346+0000","last_undegraded":"2024-04-25T05:38:15.749346+0000","last_fullsized":"2024-04-25T05:38:15.749346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741108+0000","last_change":"2024-04-25T05:38:13.302140+0000","last_active":"2024-04-25T05:38:15.741108+0000","last_peered":"2024-04-25T05:38:15.741108+0000","last_clean":"2024-04-25T05:38:15.741108+0000","last_became_active":"2024-04-25T05:38:13.284779+0000","last_became_peered":"2024-04-25T05:38:13.284779+0000","last_unstale":"2024-04-25T05:38:15.741108+0000","last_undegraded":"2024-04-25T05:38:15.741108+0000","last_fullsized":"2024-04-25T05:38:15.741108+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":224,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746720+0000","last_change":"2024-04-25T05:38:13.277150+0000","last_active":"2024-04-25T05:38:15.746720+0000","last_peered":"2024-04-25T05:38:15.746720+0000","last_clean":"2024-04-25T05:38:15.746720+0000","last_became_active":"2024-04-25T05:38:13.276807+0000","last_became_peered":"2024-04-25T05:38:13.276807+0000","last_unstale":"2024-04-25T05:38:15.746720+0000","last_undegraded":"2024-04-25T05:38:15.746720+0000","last_fullsized":"2024-04-25T05:38:15.746720+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:37:43.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:55.259152+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":332,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.742463+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:38:15.742463+0000","last_peered":"2024-04-25T05:38:15.742463+0000","last_clean":"2024-04-25T05:38:15.742463+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:38:15.742463+0000","last_undegraded":"2024-04-25T05:38:15.742463+0000","last_fullsized":"2024-04-25T05:38:15.742463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113420+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.751659+0000","last_change":"2024-04-25T05:38:13.286961+0000","last_active":"2024-04-25T05:38:15.751659+0000","last_peered":"2024-04-25T05:38:15.751659+0000","last_clean":"2024-04-25T05:38:15.751659+0000","last_became_active":"2024-04-25T05:38:13.286575+0000","last_became_peered":"2024-04-25T05:38:13.286575+0000","last_unstale":"2024-04-25T05:38:15.751659+0000","last_undegraded":"2024-04-25T05:38:15.751659+0000","last_fullsized":"2024-04-25T05:38:15.751659+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"163'11529","reported_seq":23049,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.712714+0000","last_change":"2024-04-25T05:38:14.379607+0000","last_active":"2024-04-25T05:38:17.712714+0000","last_peered":"2024-04-25T05:38:17.712714+0000","last_clean":"2024-04-25T05:38:12.203124+0000","last_became_active":"2024-04-25T05:38:14.333680+0000","last_became_peered":"2024-04-25T05:38:14.333680+0000","last_unstale":"2024-04-25T05:38:17.712714+0000","last_undegraded":"2024-04-25T05:38:17.712714+0000","last_fullsized":"2024-04-25T05:38:17.712714+0000","mapping_epoch":160,"log_start":"128'7276","ondisk_log_start":"128'7276","created":88,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196018176,"num_objects":2991,"num_object_clones":0,"num_object_copies":8973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1659,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11529,"num_write_kb":464640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5668,"num_bytes_recovered":371392512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":2991}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":214,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746679+0000","last_change":"2024-04-25T05:36:54.626908+0000","last_active":"2024-04-25T05:38:15.746679+0000","last_peered":"2024-04-25T05:38:15.746679+0000","last_clean":"2024-04-25T05:38:15.746679+0000","last_became_active":"2024-04-25T05:36:54.225394+0000","last_became_peered":"2024-04-25T05:36:54.225394+0000","last_unstale":"2024-04-25T05:38:15.746679+0000","last_undegraded":"2024-04-25T05:38:15.746679+0000","last_fullsized":"2024-04-25T05:38:15.746679+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.511619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744706+0000","last_change":"2024-04-25T05:38:13.307221+0000","last_active":"2024-04-25T05:38:15.744706+0000","last_peered":"2024-04-25T05:38:15.744706+0000","last_clean":"2024-04-25T05:38:15.744706+0000","last_became_active":"2024-04-25T05:38:13.306297+0000","last_became_peered":"2024-04-25T05:38:13.306297+0000","last_unstale":"2024-04-25T05:38:15.744706+0000","last_undegraded":"2024-04-25T05:38:15.744706+0000","last_fullsized":"2024-04-25T05:38:15.744706+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:52.054641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:52.054641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:17.305985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747795+0000","last_change":"2024-04-25T05:38:13.284622+0000","last_active":"2024-04-25T05:38:15.747795+0000","last_peered":"2024-04-25T05:38:15.747795+0000","last_clean":"2024-04-25T05:38:15.747795+0000","last_became_active":"2024-04-25T05:38:13.284281+0000","last_became_peered":"2024-04-25T05:38:13.284281+0000","last_unstale":"2024-04-25T05:38:15.747795+0000","last_undegraded":"2024-04-25T05:38:15.747795+0000","last_fullsized":"2024-04-25T05:38:15.747795+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":273,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743802+0000","last_change":"2024-04-25T05:38:13.314279+0000","last_active":"2024-04-25T05:38:15.743802+0000","last_peered":"2024-04-25T05:38:15.743802+0000","last_clean":"2024-04-25T05:38:15.743802+0000","last_became_active":"2024-04-25T05:38:13.295547+0000","last_became_peered":"2024-04-25T05:38:13.295547+0000","last_unstale":"2024-04-25T05:38:15.743802+0000","last_undegraded":"2024-04-25T05:38:15.743802+0000","last_fullsized":"2024-04-25T05:38:15.743802+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":203,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.750218+0000","last_change":"2024-04-25T05:35:42.672247+0000","last_active":"2024-04-25T05:38:15.750218+0000","last_peered":"2024-04-25T05:38:15.750218+0000","last_clean":"2024-04-25T05:38:15.750218+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:38:15.750218+0000","last_undegraded":"2024-04-25T05:38:15.750218+0000","last_fullsized":"2024-04-25T05:38:15.750218+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:42.672114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.672114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"163'11452","reported_seq":14202,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.704842+0000","last_change":"2024-04-25T05:38:14.381115+0000","last_active":"2024-04-25T05:38:17.704842+0000","last_peered":"2024-04-25T05:38:17.704842+0000","last_clean":"2024-04-25T05:38:12.178697+0000","last_became_active":"2024-04-25T05:38:14.339464+0000","last_became_peered":"2024-04-25T05:38:14.339464+0000","last_unstale":"2024-04-25T05:38:17.704842+0000","last_undegraded":"2024-04-25T05:38:17.704842+0000","last_fullsized":"2024-04-25T05:38:17.704842+0000","mapping_epoch":160,"log_start":"127'7137","ondisk_log_start":"127'7137","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197787648,"num_objects":3018,"num_object_clones":0,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1768,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1292,"num_bytes_recovered":84672512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":3018}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":200,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741419+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:38:15.741419+0000","last_peered":"2024-04-25T05:38:15.741419+0000","last_clean":"2024-04-25T05:38:15.741419+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:38:15.741419+0000","last_undegraded":"2024-04-25T05:38:15.741419+0000","last_fullsized":"2024-04-25T05:38:15.741419+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":494,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.754167+0000","last_change":"2024-04-25T05:38:15.754167+0000","last_active":"2024-04-25T05:38:15.754167+0000","last_peered":"2024-04-25T05:38:15.754167+0000","last_clean":"2024-04-25T05:38:15.754167+0000","last_became_active":"2024-04-25T05:38:15.753487+0000","last_became_peered":"2024-04-25T05:38:15.753487+0000","last_unstale":"2024-04-25T05:38:15.754167+0000","last_undegraded":"2024-04-25T05:38:15.754167+0000","last_fullsized":"2024-04-25T05:38:15.754167+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:36:28.364006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.364006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747737+0000","last_change":"2024-04-25T05:36:54.233589+0000","last_active":"2024-04-25T05:38:15.747737+0000","last_peered":"2024-04-25T05:38:15.747737+0000","last_clean":"2024-04-25T05:38:15.747737+0000","last_became_active":"2024-04-25T05:36:54.232305+0000","last_became_peered":"2024-04-25T05:36:54.232305+0000","last_unstale":"2024-04-25T05:38:15.747737+0000","last_undegraded":"2024-04-25T05:38:15.747737+0000","last_fullsized":"2024-04-25T05:38:15.747737+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"160'18","reported_seq":181,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745873+0000","last_change":"2024-04-25T05:38:13.256021+0000","last_active":"2024-04-25T05:38:15.745873+0000","last_peered":"2024-04-25T05:38:15.745873+0000","last_clean":"2024-04-25T05:38:15.745873+0000","last_became_active":"2024-04-25T05:38:13.240414+0000","last_became_peered":"2024-04-25T05:38:13.240414+0000","last_unstale":"2024-04-25T05:38:15.745873+0000","last_undegraded":"2024-04-25T05:38:15.745873+0000","last_fullsized":"2024-04-25T05:38:15.745873+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:27.678020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.678020+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":213,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744062+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:38:15.744062+0000","last_peered":"2024-04-25T05:38:15.744062+0000","last_clean":"2024-04-25T05:38:15.744062+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:38:15.744062+0000","last_undegraded":"2024-04-25T05:38:15.744062+0000","last_fullsized":"2024-04-25T05:38:15.744062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:31.527148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"163'11495","reported_seq":12267,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.697174+0000","last_change":"2024-04-25T05:38:14.354910+0000","last_active":"2024-04-25T05:38:17.697174+0000","last_peered":"2024-04-25T05:38:17.697174+0000","last_clean":"2024-04-25T05:38:12.223920+0000","last_became_active":"2024-04-25T05:38:14.315722+0000","last_became_peered":"2024-04-25T05:38:14.315722+0000","last_unstale":"2024-04-25T05:38:17.697174+0000","last_undegraded":"2024-04-25T05:38:17.697174+0000","last_fullsized":"2024-04-25T05:38:17.697174+0000","mapping_epoch":160,"log_start":"127'7198","ondisk_log_start":"127'7198","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7574","last_scrub_stamp":"2024-04-25T05:36:51.500427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.500427+0000","objects_scrubbed":7061,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201916416,"num_objects":3081,"num_object_clones":0,"num_object_copies":9243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2828,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11495,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":17235968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":3081}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":201,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.742223+0000","last_change":"2024-04-25T05:35:38.262254+0000","last_active":"2024-04-25T05:38:15.742223+0000","last_peered":"2024-04-25T05:38:15.742223+0000","last_clean":"2024-04-25T05:38:15.742223+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:38:15.742223+0000","last_undegraded":"2024-04-25T05:38:15.742223+0000","last_fullsized":"2024-04-25T05:38:15.742223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.261996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.261996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113327+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745418+0000","last_change":"2024-04-25T05:38:13.239349+0000","last_active":"2024-04-25T05:38:15.745418+0000","last_peered":"2024-04-25T05:38:15.745418+0000","last_clean":"2024-04-25T05:38:15.745418+0000","last_became_active":"2024-04-25T05:38:13.238969+0000","last_became_peered":"2024-04-25T05:38:13.238969+0000","last_unstale":"2024-04-25T05:38:15.745418+0000","last_undegraded":"2024-04-25T05:38:15.745418+0000","last_fullsized":"2024-04-25T05:38:15.745418+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":495,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:16.308518+0000","last_change":"2024-04-25T05:38:13.325017+0000","last_active":"2024-04-25T05:38:16.308518+0000","last_peered":"2024-04-25T05:38:16.308518+0000","last_clean":"2024-04-25T05:38:16.308518+0000","last_became_active":"2024-04-25T05:38:13.305327+0000","last_became_peered":"2024-04-25T05:38:13.305327+0000","last_unstale":"2024-04-25T05:38:16.308518+0000","last_undegraded":"2024-04-25T05:38:16.308518+0000","last_fullsized":"2024-04-25T05:38:16.308518+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:48.471871+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":47,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740570+0000","last_change":"2024-04-25T05:37:56.295161+0000","last_active":"2024-04-25T05:38:15.740570+0000","last_peered":"2024-04-25T05:38:15.740570+0000","last_clean":"2024-04-25T05:38:15.740570+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:38:15.740570+0000","last_undegraded":"2024-04-25T05:38:15.740570+0000","last_fullsized":"2024-04-25T05:38:15.740570+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741971+0000","last_change":"2024-04-25T05:38:13.296713+0000","last_active":"2024-04-25T05:38:15.741971+0000","last_peered":"2024-04-25T05:38:15.741971+0000","last_clean":"2024-04-25T05:38:15.741971+0000","last_became_active":"2024-04-25T05:38:13.296013+0000","last_became_peered":"2024-04-25T05:38:13.296013+0000","last_unstale":"2024-04-25T05:38:15.741971+0000","last_undegraded":"2024-04-25T05:38:15.741971+0000","last_fullsized":"2024-04-25T05:38:15.741971+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"163'11632","reported_seq":11872,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.635138+0000","last_change":"2024-04-25T05:36:51.881839+0000","last_active":"2024-04-25T05:38:17.635138+0000","last_peered":"2024-04-25T05:38:17.635138+0000","last_clean":"2024-04-25T05:38:17.635138+0000","last_became_active":"2024-04-25T05:35:09.676494+0000","last_became_peered":"2024-04-25T05:35:09.676494+0000","last_unstale":"2024-04-25T05:38:17.635138+0000","last_undegraded":"2024-04-25T05:38:17.635138+0000","last_fullsized":"2024-04-25T05:38:17.635138+0000","mapping_epoch":88,"log_start":"127'7225","ondisk_log_start":"127'7225","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7619","last_scrub_stamp":"2024-04-25T05:36:51.881693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.881693+0000","objects_scrubbed":7027,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194772992,"num_objects":2972,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":207,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743733+0000","last_change":"2024-04-25T05:36:54.234953+0000","last_active":"2024-04-25T05:38:15.743733+0000","last_peered":"2024-04-25T05:38:15.743733+0000","last_clean":"2024-04-25T05:38:15.743733+0000","last_became_active":"2024-04-25T05:36:54.234602+0000","last_became_peered":"2024-04-25T05:36:54.234602+0000","last_unstale":"2024-04-25T05:38:15.743733+0000","last_undegraded":"2024-04-25T05:38:15.743733+0000","last_fullsized":"2024-04-25T05:38:15.743733+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113403+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.742378+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:38:15.742378+0000","last_peered":"2024-04-25T05:38:15.742378+0000","last_clean":"2024-04-25T05:38:15.742378+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:38:15.742378+0000","last_undegraded":"2024-04-25T05:38:15.742378+0000","last_fullsized":"2024-04-25T05:38:15.742378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113484+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":105,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740718+0000","last_change":"2024-04-25T05:38:13.323034+0000","last_active":"2024-04-25T05:38:15.740718+0000","last_peered":"2024-04-25T05:38:15.740718+0000","last_clean":"2024-04-25T05:38:15.740718+0000","last_became_active":"2024-04-25T05:38:13.296480+0000","last_became_peered":"2024-04-25T05:38:13.296480+0000","last_unstale":"2024-04-25T05:38:15.740718+0000","last_undegraded":"2024-04-25T05:38:15.740718+0000","last_fullsized":"2024-04-25T05:38:15.740718+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:35.248443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.248443+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741344+0000","last_change":"2024-04-25T05:38:13.239510+0000","last_active":"2024-04-25T05:38:15.741344+0000","last_peered":"2024-04-25T05:38:15.741344+0000","last_clean":"2024-04-25T05:38:15.741344+0000","last_became_active":"2024-04-25T05:38:13.237844+0000","last_became_peered":"2024-04-25T05:38:13.237844+0000","last_unstale":"2024-04-25T05:38:15.741344+0000","last_undegraded":"2024-04-25T05:38:15.741344+0000","last_fullsized":"2024-04-25T05:38:15.741344+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"163'11761","reported_seq":23363,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.695066+0000","last_change":"2024-04-25T05:37:42.904598+0000","last_active":"2024-04-25T05:38:17.695066+0000","last_peered":"2024-04-25T05:38:17.695066+0000","last_clean":"2024-04-25T05:38:17.695066+0000","last_became_active":"2024-04-25T05:37:42.889210+0000","last_became_peered":"2024-04-25T05:37:42.889210+0000","last_unstale":"2024-04-25T05:38:17.695066+0000","last_undegraded":"2024-04-25T05:38:17.695066+0000","last_fullsized":"2024-04-25T05:38:17.695066+0000","mapping_epoch":147,"log_start":"128'7498","ondisk_log_start":"128'7498","created":88,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198639616,"num_objects":3031,"num_object_clones":0,"num_object_copies":9093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":374472704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746902+0000","last_change":"2024-04-25T05:38:13.244062+0000","last_active":"2024-04-25T05:38:15.746902+0000","last_peered":"2024-04-25T05:38:15.746902+0000","last_clean":"2024-04-25T05:38:15.746902+0000","last_became_active":"2024-04-25T05:38:13.243650+0000","last_became_peered":"2024-04-25T05:38:13.243650+0000","last_unstale":"2024-04-25T05:38:15.746902+0000","last_undegraded":"2024-04-25T05:38:15.746902+0000","last_fullsized":"2024-04-25T05:38:15.746902+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":213,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741121+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:38:15.741121+0000","last_peered":"2024-04-25T05:38:15.741121+0000","last_clean":"2024-04-25T05:38:15.741121+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:38:15.741121+0000","last_undegraded":"2024-04-25T05:38:15.741121+0000","last_fullsized":"2024-04-25T05:38:15.741121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":166,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.748280+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:38:15.748280+0000","last_peered":"2024-04-25T05:38:15.748280+0000","last_clean":"2024-04-25T05:38:15.748280+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:38:15.748280+0000","last_undegraded":"2024-04-25T05:38:15.748280+0000","last_fullsized":"2024-04-25T05:38:15.748280+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"160'10","reported_seq":48,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740954+0000","last_change":"2024-04-25T05:38:13.308179+0000","last_active":"2024-04-25T05:38:15.740954+0000","last_peered":"2024-04-25T05:38:15.740954+0000","last_clean":"2024-04-25T05:38:15.740954+0000","last_became_active":"2024-04-25T05:38:13.295784+0000","last_became_peered":"2024-04-25T05:38:13.295784+0000","last_unstale":"2024-04-25T05:38:15.740954+0000","last_undegraded":"2024-04-25T05:38:15.740954+0000","last_fullsized":"2024-04-25T05:38:15.740954+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:35:42.311077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:42.311077+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"163'11632","reported_seq":11793,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.949989+0000","last_change":"2024-04-25T05:35:09.681219+0000","last_active":"2024-04-25T05:38:17.949989+0000","last_peered":"2024-04-25T05:38:17.949989+0000","last_clean":"2024-04-25T05:38:17.949989+0000","last_became_active":"2024-04-25T05:35:09.680692+0000","last_became_peered":"2024-04-25T05:35:09.680692+0000","last_unstale":"2024-04-25T05:38:17.949989+0000","last_undegraded":"2024-04-25T05:38:17.949989+0000","last_fullsized":"2024-04-25T05:38:17.949989+0000","mapping_epoch":88,"log_start":"134'8057","ondisk_log_start":"134'8057","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3575,"log_dups_size":0,"ondisk_log_size":3575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198180864,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11632,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":215,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744134+0000","last_change":"2024-04-25T05:36:59.627764+0000","last_active":"2024-04-25T05:38:15.744134+0000","last_peered":"2024-04-25T05:38:15.744134+0000","last_clean":"2024-04-25T05:38:15.744134+0000","last_became_active":"2024-04-25T05:36:54.234860+0000","last_became_peered":"2024-04-25T05:36:54.234860+0000","last_unstale":"2024-04-25T05:38:15.744134+0000","last_undegraded":"2024-04-25T05:38:15.744134+0000","last_fullsized":"2024-04-25T05:38:15.744134+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:26.357457+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":99,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.742998+0000","last_change":"2024-04-25T05:38:13.301084+0000","last_active":"2024-04-25T05:38:15.742998+0000","last_peered":"2024-04-25T05:38:15.742998+0000","last_clean":"2024-04-25T05:38:15.742998+0000","last_became_active":"2024-04-25T05:38:13.289662+0000","last_became_peered":"2024-04-25T05:38:13.289662+0000","last_unstale":"2024-04-25T05:38:15.742998+0000","last_undegraded":"2024-04-25T05:38:15.742998+0000","last_fullsized":"2024-04-25T05:38:15.742998+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745539+0000","last_change":"2024-04-25T05:38:13.255913+0000","last_active":"2024-04-25T05:38:15.745539+0000","last_peered":"2024-04-25T05:38:15.745539+0000","last_clean":"2024-04-25T05:38:15.745539+0000","last_became_active":"2024-04-25T05:38:13.244214+0000","last_became_peered":"2024-04-25T05:38:13.244214+0000","last_unstale":"2024-04-25T05:38:15.745539+0000","last_undegraded":"2024-04-25T05:38:15.745539+0000","last_fullsized":"2024-04-25T05:38:15.745539+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":265,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747445+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:38:15.747445+0000","last_peered":"2024-04-25T05:38:15.747445+0000","last_clean":"2024-04-25T05:38:15.747445+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:38:15.747445+0000","last_undegraded":"2024-04-25T05:38:15.747445+0000","last_fullsized":"2024-04-25T05:38:15.747445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"163'11570","reported_seq":11731,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.960750+0000","last_change":"2024-04-25T05:35:09.680621+0000","last_active":"2024-04-25T05:38:17.960750+0000","last_peered":"2024-04-25T05:38:17.960750+0000","last_clean":"2024-04-25T05:38:17.960750+0000","last_became_active":"2024-04-25T05:35:09.680294+0000","last_became_peered":"2024-04-25T05:35:09.680294+0000","last_unstale":"2024-04-25T05:38:17.960750+0000","last_undegraded":"2024-04-25T05:38:17.960750+0000","last_fullsized":"2024-04-25T05:38:17.960750+0000","mapping_epoch":88,"log_start":"134'7906","ondisk_log_start":"134'7906","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3664,"log_dups_size":0,"ondisk_log_size":3664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":200,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745820+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:38:15.745820+0000","last_peered":"2024-04-25T05:38:15.745820+0000","last_clean":"2024-04-25T05:38:15.745820+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:38:15.745820+0000","last_undegraded":"2024-04-25T05:38:15.745820+0000","last_fullsized":"2024-04-25T05:38:15.745820+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":219,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.754076+0000","last_change":"2024-04-25T05:38:13.247469+0000","last_active":"2024-04-25T05:38:15.754076+0000","last_peered":"2024-04-25T05:38:15.754076+0000","last_clean":"2024-04-25T05:38:15.754076+0000","last_became_active":"2024-04-25T05:38:13.246750+0000","last_became_peered":"2024-04-25T05:38:13.246750+0000","last_unstale":"2024-04-25T05:38:15.754076+0000","last_undegraded":"2024-04-25T05:38:15.754076+0000","last_fullsized":"2024-04-25T05:38:15.754076+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":293,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747283+0000","last_change":"2024-04-25T05:38:13.326846+0000","last_active":"2024-04-25T05:38:15.747283+0000","last_peered":"2024-04-25T05:38:15.747283+0000","last_clean":"2024-04-25T05:38:15.747283+0000","last_became_active":"2024-04-25T05:38:13.305935+0000","last_became_peered":"2024-04-25T05:38:13.305935+0000","last_unstale":"2024-04-25T05:38:15.747283+0000","last_undegraded":"2024-04-25T05:38:15.747283+0000","last_fullsized":"2024-04-25T05:38:15.747283+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":204,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745331+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:38:15.745331+0000","last_peered":"2024-04-25T05:38:15.745331+0000","last_clean":"2024-04-25T05:38:15.745331+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:38:15.745331+0000","last_undegraded":"2024-04-25T05:38:15.745331+0000","last_fullsized":"2024-04-25T05:38:15.745331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"163'11531","reported_seq":22421,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:17.911452+0000","last_change":"2024-04-25T05:37:52.135113+0000","last_active":"2024-04-25T05:38:17.911452+0000","last_peered":"2024-04-25T05:38:17.911452+0000","last_clean":"2024-04-25T05:38:17.911452+0000","last_became_active":"2024-04-25T05:37:52.108302+0000","last_became_peered":"2024-04-25T05:37:52.108302+0000","last_unstale":"2024-04-25T05:38:17.911452+0000","last_undegraded":"2024-04-25T05:38:17.911452+0000","last_fullsized":"2024-04-25T05:38:17.911452+0000","mapping_epoch":151,"log_start":"134'7877","ondisk_log_start":"134'7877","created":88,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3654,"log_dups_size":0,"ondisk_log_size":3654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195362816,"num_objects":2981,"num_object_clones":0,"num_object_copies":8943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":464384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5358,"num_bytes_recovered":351141888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":214,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.740676+0000","last_change":"2024-04-25T05:38:13.284810+0000","last_active":"2024-04-25T05:38:15.740676+0000","last_peered":"2024-04-25T05:38:15.740676+0000","last_clean":"2024-04-25T05:38:15.740676+0000","last_became_active":"2024-04-25T05:38:13.284451+0000","last_became_peered":"2024-04-25T05:38:13.284451+0000","last_unstale":"2024-04-25T05:38:15.740676+0000","last_undegraded":"2024-04-25T05:38:15.740676+0000","last_fullsized":"2024-04-25T05:38:15.740676+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.984070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"160'9","reported_seq":274,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746625+0000","last_change":"2024-04-25T05:38:13.244991+0000","last_active":"2024-04-25T05:38:15.746625+0000","last_peered":"2024-04-25T05:38:15.746625+0000","last_clean":"2024-04-25T05:38:15.746625+0000","last_became_active":"2024-04-25T05:38:13.238743+0000","last_became_peered":"2024-04-25T05:38:13.238743+0000","last_unstale":"2024-04-25T05:38:15.746625+0000","last_undegraded":"2024-04-25T05:38:15.746625+0000","last_fullsized":"2024-04-25T05:38:15.746625+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:37:48.604653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:48.604653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:55.398396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747196+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:38:15.747196+0000","last_peered":"2024-04-25T05:38:15.747196+0000","last_clean":"2024-04-25T05:38:15.747196+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:38:15.747196+0000","last_undegraded":"2024-04-25T05:38:15.747196+0000","last_fullsized":"2024-04-25T05:38:15.747196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867191+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":233,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.744123+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:38:15.744123+0000","last_peered":"2024-04-25T05:38:15.744123+0000","last_clean":"2024-04-25T05:38:15.744123+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:38:15.744123+0000","last_undegraded":"2024-04-25T05:38:15.744123+0000","last_fullsized":"2024-04-25T05:38:15.744123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:18.113387+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"163'11633","reported_seq":23295,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.947893+0000","last_change":"2024-04-25T05:38:14.419090+0000","last_active":"2024-04-25T05:38:17.947893+0000","last_peered":"2024-04-25T05:38:17.947893+0000","last_clean":"2024-04-25T05:38:12.215020+0000","last_became_active":"2024-04-25T05:38:14.362008+0000","last_became_peered":"2024-04-25T05:38:14.362008+0000","last_unstale":"2024-04-25T05:38:17.947893+0000","last_undegraded":"2024-04-25T05:38:17.947893+0000","last_fullsized":"2024-04-25T05:38:17.947893+0000","mapping_epoch":160,"log_start":"134'8052","ondisk_log_start":"134'8052","created":88,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":4899,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200409112,"num_objects":3059,"num_object_clones":0,"num_object_copies":9177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2714,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11633,"num_write_kb":470081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":374407192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,0],"avail_no_missing":["1","0","3"],"object_location_counts":[{"shards":"0,1,3","objects":3059}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747780+0000","last_change":"2024-04-25T05:36:54.232549+0000","last_active":"2024-04-25T05:38:15.747780+0000","last_peered":"2024-04-25T05:38:15.747780+0000","last_clean":"2024-04-25T05:38:15.747780+0000","last_became_active":"2024-04-25T05:36:54.231886+0000","last_became_peered":"2024-04-25T05:36:54.231886+0000","last_unstale":"2024-04-25T05:38:15.747780+0000","last_undegraded":"2024-04-25T05:38:15.747780+0000","last_fullsized":"2024-04-25T05:38:15.747780+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.867048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.741871+0000","last_change":"2024-04-25T05:37:08.197220+0000","last_active":"2024-04-25T05:38:15.741871+0000","last_peered":"2024-04-25T05:38:15.741871+0000","last_clean":"2024-04-25T05:38:15.741871+0000","last_became_active":"2024-04-25T05:36:54.244909+0000","last_became_peered":"2024-04-25T05:36:54.244909+0000","last_unstale":"2024-04-25T05:38:15.741871+0000","last_undegraded":"2024-04-25T05:38:15.741871+0000","last_fullsized":"2024-04-25T05:38:15.741871+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:24.881588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":212,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745431+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:38:15.745431+0000","last_peered":"2024-04-25T05:38:15.745431+0000","last_clean":"2024-04-25T05:38:15.745431+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:38:15.745431+0000","last_undegraded":"2024-04-25T05:38:15.745431+0000","last_fullsized":"2024-04-25T05:38:15.745431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":258,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746434+0000","last_change":"2024-04-25T05:36:54.269613+0000","last_active":"2024-04-25T05:38:15.746434+0000","last_peered":"2024-04-25T05:38:15.746434+0000","last_clean":"2024-04-25T05:38:15.746434+0000","last_became_active":"2024-04-25T05:36:54.244739+0000","last_became_peered":"2024-04-25T05:36:54.244739+0000","last_unstale":"2024-04-25T05:38:15.746434+0000","last_undegraded":"2024-04-25T05:38:15.746434+0000","last_fullsized":"2024-04-25T05:38:15.746434+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703313+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"163'11570","reported_seq":12322,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.892328+0000","last_change":"2024-04-25T05:38:14.378180+0000","last_active":"2024-04-25T05:38:17.892328+0000","last_peered":"2024-04-25T05:38:17.892328+0000","last_clean":"2024-04-25T05:38:12.217756+0000","last_became_active":"2024-04-25T05:38:14.325480+0000","last_became_peered":"2024-04-25T05:38:14.325480+0000","last_unstale":"2024-04-25T05:38:17.892328+0000","last_undegraded":"2024-04-25T05:38:17.892328+0000","last_fullsized":"2024-04-25T05:38:17.892328+0000","mapping_epoch":160,"log_start":"123'6640","ondisk_log_start":"123'6640","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7661","last_scrub_stamp":"2024-04-25T05:36:53.309266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.309266+0000","objects_scrubbed":6949,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194641920,"num_objects":2970,"num_object_clones":0,"num_object_copies":8910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2754,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":465280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":14614528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,0],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":2970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":199,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749610+0000","last_change":"2024-04-25T05:35:39.752489+0000","last_active":"2024-04-25T05:38:15.749610+0000","last_peered":"2024-04-25T05:38:15.749610+0000","last_clean":"2024-04-25T05:38:15.749610+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:38:15.749610+0000","last_undegraded":"2024-04-25T05:38:15.749610+0000","last_fullsized":"2024-04-25T05:38:15.749610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:39.752319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.752319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.743447+0000","last_change":"2024-04-25T05:38:13.289683+0000","last_active":"2024-04-25T05:38:15.743447+0000","last_peered":"2024-04-25T05:38:15.743447+0000","last_clean":"2024-04-25T05:38:15.743447+0000","last_became_active":"2024-04-25T05:38:13.289313+0000","last_became_peered":"2024-04-25T05:38:13.289313+0000","last_unstale":"2024-04-25T05:38:15.743447+0000","last_undegraded":"2024-04-25T05:38:15.743447+0000","last_fullsized":"2024-04-25T05:38:15.743447+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":79,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.745872+0000","last_change":"2024-04-25T05:36:54.226814+0000","last_active":"2024-04-25T05:38:15.745872+0000","last_peered":"2024-04-25T05:38:15.745872+0000","last_clean":"2024-04-25T05:38:15.745872+0000","last_became_active":"2024-04-25T05:36:54.226138+0000","last_became_peered":"2024-04-25T05:36:54.226138+0000","last_unstale":"2024-04-25T05:38:15.745872+0000","last_undegraded":"2024-04-25T05:38:15.745872+0000","last_fullsized":"2024-04-25T05:38:15.745872+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746044+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:38:15.746044+0000","last_peered":"2024-04-25T05:38:15.746044+0000","last_clean":"2024-04-25T05:38:15.746044+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:38:15.746044+0000","last_undegraded":"2024-04-25T05:38:15.746044+0000","last_fullsized":"2024-04-25T05:38:15.746044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703291+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"163'11912","reported_seq":12701,"reported_epoch":163,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:38:17.965564+0000","last_change":"2024-04-25T05:38:14.402575+0000","last_active":"2024-04-25T05:38:17.965564+0000","last_peered":"2024-04-25T05:38:17.965564+0000","last_clean":"2024-04-25T05:38:12.225884+0000","last_became_active":"2024-04-25T05:38:14.357726+0000","last_became_peered":"2024-04-25T05:38:14.357726+0000","last_unstale":"2024-04-25T05:38:17.965564+0000","last_undegraded":"2024-04-25T05:38:17.965564+0000","last_fullsized":"2024-04-25T05:38:17.965564+0000","mapping_epoch":160,"log_start":"134'8255","ondisk_log_start":"134'8255","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:09.212225+0000","last_clean_scrub_stamp":"2024-04-25T05:35:09.212225+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2807,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":480768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":20512768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":3112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":199,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746720+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:38:15.746720+0000","last_peered":"2024-04-25T05:38:15.746720+0000","last_clean":"2024-04-25T05:38:15.746720+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:38:15.746720+0000","last_undegraded":"2024-04-25T05:38:15.746720+0000","last_fullsized":"2024-04-25T05:38:15.746720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703366+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":267,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.749306+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:38:15.749306+0000","last_peered":"2024-04-25T05:38:15.749306+0000","last_clean":"2024-04-25T05:38:15.749306+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:38:15.749306+0000","last_undegraded":"2024-04-25T05:38:15.749306+0000","last_fullsized":"2024-04-25T05:38:15.749306+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.746686+0000","last_change":"2024-04-25T05:38:13.286297+0000","last_active":"2024-04-25T05:38:15.746686+0000","last_peered":"2024-04-25T05:38:15.746686+0000","last_clean":"2024-04-25T05:38:15.746686+0000","last_became_active":"2024-04-25T05:38:13.285812+0000","last_became_peered":"2024-04-25T05:38:13.285812+0000","last_unstale":"2024-04-25T05:38:15.746686+0000","last_undegraded":"2024-04-25T05:38:15.746686+0000","last_fullsized":"2024-04-25T05:38:15.746686+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:23.154413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":240,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-04-25T05:38:15.747954+0000","last_change":"2024-04-25T05:36:54.254870+0000","last_active":"2024-04-25T05:38:15.747954+0000","last_peered":"2024-04-25T05:38:15.747954+0000","last_clean":"2024-04-25T05:38:15.747954+0000","last_became_active":"2024-04-25T05:36:54.234440+0000","last_became_peered":"2024-04-25T05:36:54.234440+0000","last_unstale":"2024-04-25T05:38:15.747954+0000","last_undegraded":"2024-04-25T05:38:15.747954+0000","last_fullsized":"2024-04-25T05:38:15.747954+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:38:17.703473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3154771992,"num_objects":48139,"num_object_clones":0,"num_object_copies":144417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19876,"num_objects_unfound":0,"num_objects_dirty":48139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185595,"num_write_kb":7479425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35973,"num_bytes_recovered":2357329944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10027413504,"data_stored":10027401288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63906,"ondisk_log_size":63906,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":13834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":617,"ondisk_log_size":617,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070521,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142028,"kb_used_data":1034492,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92603124,"statfs":{"total":95995035648,"available":94825598976,"internally_reserved":0,"allocated":1059319808,"data_stored":1058872554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1572792,"kb_used_data":1379112,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92172360,"statfs":{"total":95995035648,"available":94384496640,"internally_reserved":0,"allocated":1412210688,"data_stored":1411717816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712420,"kb_used_data":1532700,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92032732,"statfs":{"total":95995035648,"available":94241517568,"internally_reserved":0,"allocated":1569484800,"data_stored":1569068895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2355160,"kb_used_data":2139344,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91389992,"statfs":{"total":95995035648,"available":93583351808,"internally_reserved":0,"allocated":2190688256,"data_stored":2190219428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220985799},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081084,"kb_used_data":977712,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92664068,"statfs":{"total":95995035648,"available":94888005632,"internally_reserved":0,"allocated":1001177088,"data_stored":1000740396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1007084,"kb_used_data":917604,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92738068,"statfs":{"total":95995035648,"available":94963781632,"internally_reserved":0,"allocated":939626496,"data_stored":939201716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1580732416,"data_stored":1580728344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1393889280,"data_stored":1393885208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1566769152,"data_stored":1566769152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2188840960,"data_stored":2188836888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1000013824,"data_stored":1000013824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":947847168,"data_stored":947847168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289538048,"data_stored":289538048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1059782656,"data_stored":1059782656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:38:23.683 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-25T05:38:23.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-25T05:38:23.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:38:23.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:23.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:23.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:23.931 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:23.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:24.026 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:24.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:24.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:24.152 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:24.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:24.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:38:24.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:24.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:24.363 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:24.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:24 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:38:24.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:24 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:38:24.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:24.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:24 smithi060 ceph-mon[30898]: osdmap e167: 8 total, 8 up, 6 in 2024-04-25T05:38:24.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:24.490 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:24.582 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:24.582 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:24.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:24.624 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:38:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:38:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:24.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[27010]: osdmap e167: 8 total, 8 up, 6 in 2024-04-25T05:38:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:38:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:38:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:24 smithi043 ceph-mon[37492]: osdmap e167: 8 total, 8 up, 6 in 2024-04-25T05:38:24.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:24.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:24.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:24.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:24.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:24.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:38:24.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:25.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:25.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:25.215 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:25.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:25.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:25.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:38:25.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[27010]: pgmap v208: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 10863/133833 objects misplaced (8.117%); 121 MiB/s, 1.93k objects/s recovering 2024-04-25T05:38:25.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:38:25.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:38:25.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[27010]: osdmap e168: 8 total, 8 up, 6 in 2024-04-25T05:38:25.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[37492]: pgmap v208: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 10863/133833 objects misplaced (8.117%); 121 MiB/s, 1.93k objects/s recovering 2024-04-25T05:38:25.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:38:25.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:38:25.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:25 smithi043 ceph-mon[37492]: osdmap e168: 8 total, 8 up, 6 in 2024-04-25T05:38:25.421 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:25.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:25 smithi060 ceph-mon[30898]: pgmap v208: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 10863/133833 objects misplaced (8.117%); 121 MiB/s, 1.93k objects/s recovering 2024-04-25T05:38:25.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:25 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:38:25.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:25 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:38:25.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:25 smithi060 ceph-mon[30898]: osdmap e168: 8 total, 8 up, 6 in 2024-04-25T05:38:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:25.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:25.553 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:25.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:25.681 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:25.716 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:25.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:25.809 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:25.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:25.971 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:25.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:25.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:26.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:26.111 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:26.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:26.216 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pg_num to 11 2024-04-25T05:38:26.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:26.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:26.385 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:38:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:38:26.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:26.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:38:26.508 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:26.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:26.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:26.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.767 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:26.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:26.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:26.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:26.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:27.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:27.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:27.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:27.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:27.224 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:38:27.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:27.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: pgmap v210: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 752 op/s; 10863/133833 objects misplaced (8.117%); 97 MiB/s, 1.55k objects/s recovering 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: osdmap e169: 8 total, 8 up, 6 in 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1446062036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: pgmap v210: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 752 op/s; 10863/133833 objects misplaced (8.117%); 97 MiB/s, 1.55k objects/s recovering 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: osdmap e169: 8 total, 8 up, 6 in 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1446062036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:38:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:27 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:38:27.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:27.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: pgmap v210: 148 pgs: 2 peering, 6 active+remapped+backfilling, 140 active+clean; 2.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1 KiB/s rd, 0 B/s wr, 752 op/s; 10863/133833 objects misplaced (8.117%); 97 MiB/s, 1.55k objects/s recovering 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3605670798' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: osdmap e169: 8 total, 8 up, 6 in 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1446062036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:38:27.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:27 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:38:27.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:27.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:27.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:27.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:27.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:27.822 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:27.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:27.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:27.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:28.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:28.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:28.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:28.206 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:28.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:28.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:28.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:28.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:28.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:28 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:28 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:38:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:28 smithi060 ceph-mon[30898]: osdmap e170: 8 total, 8 up, 6 in 2024-04-25T05:38:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:28 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:28.532 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:28.533 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:28.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:28.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:38:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[27010]: osdmap e170: 8 total, 8 up, 6 in 2024-04-25T05:38:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:38:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[37492]: osdmap e170: 8 total, 8 up, 6 in 2024-04-25T05:38:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:28 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:28.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:28.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:28.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:28.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:28.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:28.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:29.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:29.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:29.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:29.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:29.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:29.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:29.479 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[27010]: pgmap v213: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 247 op/s; 10187/130428 objects misplaced (7.810%); 7.6 MiB/s, 121 objects/s recovering 2024-04-25T05:38:29.480 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:38:29.480 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[27010]: osdmap e171: 8 total, 8 up, 6 in 2024-04-25T05:38:29.480 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[37492]: pgmap v213: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 247 op/s; 10187/130428 objects misplaced (7.810%); 7.6 MiB/s, 121 objects/s recovering 2024-04-25T05:38:29.480 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:38:29.480 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-mon[37492]: osdmap e171: 8 total, 8 up, 6 in 2024-04-25T05:38:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:29 smithi060 ceph-mon[30898]: pgmap v213: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 2.6 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 247 op/s; 10187/130428 objects misplaced (7.810%); 7.6 MiB/s, 121 objects/s recovering 2024-04-25T05:38:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:29 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:38:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:29 smithi060 ceph-mon[30898]: osdmap e171: 8 total, 8 up, 6 in 2024-04-25T05:38:29.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:29.554 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:29.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:29.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:29.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:29.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:38:29.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:29.969 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:29.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:30.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.099 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:30.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:30.118 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:30.164 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:30.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:30.204 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:30.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:30.261 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:30.304 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:30.324 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:30.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:30.345 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:30.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:30.372 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:30.378 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:30.379 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:30.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:30.484 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:30.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:30 smithi060 ceph-mon[30898]: osdmap e172: 8 total, 8 up, 6 in 2024-04-25T05:38:30.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:30 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:30.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:30 smithi060 ceph-mon[30898]: osdmap e173: 8 total, 8 up, 6 in 2024-04-25T05:38:30.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/961467452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:30.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:30.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[27010]: osdmap e172: 8 total, 8 up, 6 in 2024-04-25T05:38:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[27010]: osdmap e173: 8 total, 8 up, 6 in 2024-04-25T05:38:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/961467452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[37492]: osdmap e172: 8 total, 8 up, 6 in 2024-04-25T05:38:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:38:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[37492]: osdmap e173: 8 total, 8 up, 6 in 2024-04-25T05:38:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/961467452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:30.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:30.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:30.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:30.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:30.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:30.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:31.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:31.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:31.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:31.235 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[27010]: pgmap v216: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 6773/122535 objects misplaced (5.527%); 52 MiB/s, 836 objects/s recovering 2024-04-25T05:38:31.235 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:38:31.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:31.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:31.415 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:31.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:31 smithi060 ceph-mon[30898]: pgmap v216: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 6773/122535 objects misplaced (5.527%); 52 MiB/s, 836 objects/s recovering 2024-04-25T05:38:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:38:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:31 smithi060 ceph-mon[30898]: osdmap e174: 8 total, 8 up, 6 in 2024-04-25T05:38:31.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:31.545 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:31.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[27010]: osdmap e174: 8 total, 8 up, 6 in 2024-04-25T05:38:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[37492]: pgmap v216: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 6773/122535 objects misplaced (5.527%); 52 MiB/s, 836 objects/s recovering 2024-04-25T05:38:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:38:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:31 smithi043 ceph-mon[37492]: osdmap e174: 8 total, 8 up, 6 in 2024-04-25T05:38:31.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:31.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:31.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:31.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:31.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:32.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:32.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:32.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:38:32.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:32.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:32.347 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:32.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:32.459 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:32.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:32.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:32.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:32.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:32.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:32.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:32.853 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:32 smithi060 ceph-mon[30898]: pgmap v219: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 6773/122535 objects misplaced (5.527%); 44 MiB/s, 697 objects/s recovering 2024-04-25T05:38:32.853 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:32 smithi060 ceph-mon[30898]: osdmap e175: 8 total, 8 up, 6 in 2024-04-25T05:38:32.853 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:32.853 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:32.874 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:32.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:32.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:33.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:33.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[37492]: pgmap v219: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 6773/122535 objects misplaced (5.527%); 44 MiB/s, 697 objects/s recovering 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[37492]: osdmap e175: 8 total, 8 up, 6 in 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[27010]: pgmap v219: 158 pgs: 10 unknown, 6 active+remapped+backfilling, 142 active+clean; 2.5 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 6773/122535 objects misplaced (5.527%); 44 MiB/s, 697 objects/s recovering 2024-04-25T05:38:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[27010]: osdmap e175: 8 total, 8 up, 6 in 2024-04-25T05:38:33.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:33.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:33.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:33.161 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:38:33.161 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:38:33.161 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:38:33.161 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:38:33.161 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:38:33.161 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:33.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:33.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:33.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:33.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:33.539 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:33.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:33.661 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:33.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:33.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:33.941 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:33 smithi060 ceph-mon[30898]: osdmap e176: 8 total, 8 up, 6 in 2024-04-25T05:38:33.941 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:33.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:33.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:33.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:38:34.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:34.071 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:33 smithi043 ceph-mon[27010]: osdmap e176: 8 total, 8 up, 6 in 2024-04-25T05:38:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:33 smithi043 ceph-mon[37492]: osdmap e176: 8 total, 8 up, 6 in 2024-04-25T05:38:34.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:34.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:34.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:34.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:34.303 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:34.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:34.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:34.530 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7] out_osds: [6, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:38:34.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:38:34.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:34.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:34.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:34.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:34.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:34.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:34.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:34.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[27010]: pgmap v222: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:34.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[27010]: osdmap e177: 8 total, 8 up, 6 in 2024-04-25T05:38:34.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[27010]: osdmap e178: 8 total, 8 up, 6 in 2024-04-25T05:38:34.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[37492]: pgmap v222: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:34.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[37492]: osdmap e177: 8 total, 8 up, 6 in 2024-04-25T05:38:34.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:34 smithi043 ceph-mon[37492]: osdmap e178: 8 total, 8 up, 6 in 2024-04-25T05:38:34.896 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:45788 7.0 7:0a9817e4:::benchmark_data_smithi043_74356_object45787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:40.468556+0000", 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 174.425265594, 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.076130657000000004, 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.897 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 45788 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468556+0000", 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468556+0000", 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468557+0000", 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4200000000000005e-07 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468586+0000", 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8413999999999999e-05 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468593+0000", 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2350000000000002e-06 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468600+0000", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0119999999999999e-06 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468759+0000", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000159245 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.468890+0000", 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013102 2024-04-25T05:38:34.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.469014+0000", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012375 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.470543+0000", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001529288 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.470754+0000", 2024-04-25T05:38:34.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021135699999999999 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.544606+0000", 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073851268999999997 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.544643+0000", 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7125000000000001e-05 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:40.544687+0000", 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4100000000000001e-05 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:129259 7.0 7:050cda31:::benchmark_data_smithi043_74356_object12390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 92.546579442999999, 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098052218999999996, 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.903 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129259 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347244+0000", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.364e-06 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347250+0000", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1349999999999997e-06 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347266+0000", 2024-04-25T05:38:34.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5911000000000001e-05 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347274+0000", 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1769999999999998e-06 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347332+0000", 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7521e-05 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347467+0000", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013588000000000001 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347601+0000", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013331399999999999 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.350192+0000", 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002590958 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.353155+0000", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029637579999999999 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445182+0000", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.092026636999999994 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445245+0000", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3274000000000004e-05 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445295+0000", 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-05 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:129669 7.0 7:0aa791da:::benchmark_data_smithi043_74356_object12800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:02.978880+0000", 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 91.914941498000005, 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078216114000000003, 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.909 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129669 2024-04-25T05:38:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978880+0000", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978880+0000", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978882+0000", 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3370000000000001e-06 2024-04-25T05:38:34.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978888+0000", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1630000000000001e-06 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978900+0000", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1639e-05 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978906+0000", 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1079999999999998e-06 2024-04-25T05:38:34.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.978956+0000", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0875000000000003e-05 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.979063+0000", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010644 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.979143+0000", 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9983999999999999e-05 2024-04-25T05:38:34.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.981156+0000", 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002013202 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.983473+0000", 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023167230000000001 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.056943+0000", 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.073470267000000006 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.057033+0000", 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0336999999999999e-05 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.057096+0000", 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3039000000000004e-05 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130360 7.0 7:05078405:::benchmark_data_smithi043_74356_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.981090378000005, 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100930387, 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130360 2024-04-25T05:38:34.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912733+0000", 2024-04-25T05:38:34.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.516e-06 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912738+0000", 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0490000000000003e-06 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912749+0000", 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0635999999999999e-05 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912755+0000", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9429999999999997e-06 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912807+0000", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2732000000000003e-05 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912916+0000", 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109107 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913025+0000", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108824 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919839+0000", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068137730000000004 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998798+0000", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078959219999999997 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013581+0000", 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014783053000000001 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013621+0000", 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9190999999999998e-05 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013662+0000", 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1343e-05 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130361 7.0 7:0c1643bc:::benchmark_data_smithi043_74356_object13492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.981026876000001, 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.101036217, 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130361 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912796+0000", 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7999999999999995e-07 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912802+0000", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6789999999999999e-06 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912809+0000", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9929999999999998e-06 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912821+0000", 2024-04-25T05:38:34.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1637999999999999e-05 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915433+0000", 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026120610000000002 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915541+0000", 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108274 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915609+0000", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8141999999999999e-05 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919843+0000", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042331419999999996 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998857+0000", 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079014847999999999 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013723+0000", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014866047 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013784+0000", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0198999999999997e-05 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013831+0000", 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7614000000000002e-05 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.927 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130363 7.0 7:0185dbbe:::benchmark_data_smithi043_74356_object13494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.980495911000006, 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10060899700000001, 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130363 2024-04-25T05:38:34.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913327+0000", 2024-04-25T05:38:34.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1999999999999999e-07 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913331+0000", 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0940000000000001e-06 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913342+0000", 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0767999999999999e-05 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913359+0000", 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7198999999999999e-05 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917317+0000", 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039578080000000002 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917403+0000", 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6774999999999995e-05 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917472+0000", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8664000000000005e-05 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919845+0000", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002373165 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998895+0000", 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079049509000000004 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013869+0000", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014974315 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013898+0000", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9014999999999999e-05 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013935+0000", 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6965000000000003e-05 2024-04-25T05:38:34.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130364 7.0 7:00060f6c:::benchmark_data_smithi043_74356_object13495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.979375598999994, 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099641968999999997, 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130364 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914447+0000", 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6599999999999995e-07 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914451+0000", 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7570000000000002e-06 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914460+0000", 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2329999999999995e-06 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914467+0000", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3130000000000001e-06 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918588+0000", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041202340000000004 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918660+0000", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2220999999999999e-05 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918726+0000", 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5771000000000004e-05 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919932+0000", 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012063740000000001 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.002732+0000", 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082799625000000002 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013993+0000", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011261567 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.014048+0000", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4951999999999998e-05 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.014088+0000", 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0156e-05 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130499 7.0 7:06be1c12:::benchmark_data_smithi043_74356_object13630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 90.766664253000002, 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.113647552, 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130499 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.940 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127159+0000", 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1689999999999999e-06 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127163+0000", 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4950000000000002e-06 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127173+0000", 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086e-05 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127179+0000", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127225+0000", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5946999999999999e-05 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127358+0000", 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013307600000000001 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127504+0000", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014553799999999999 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.130178+0000", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002674091 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.233209+0000", 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10303095499999999 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240692+0000", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0074828560000000004 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240751+0000", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9342000000000002e-05 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240805+0000", 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4143e-05 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131702 7.0 7:027c59a2:::benchmark_data_smithi043_74356_object14833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.933094+0000", 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 88.960727728999998, 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.074961780000000006, 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.946 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131702 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933094+0000", 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933094+0000", 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933096+0000", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7990000000000001e-06 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933103+0000", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5729999999999996e-06 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933116+0000", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3837e-05 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933126+0000", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6299999999999993e-06 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933186+0000", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0216000000000001e-05 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933334+0000", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014749199999999999 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933476+0000", 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000142767 2024-04-25T05:38:34.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.938013+0000", 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045368209999999999 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.940291+0000", 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002277575 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.007924+0000", 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.067633132999999998 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.008003+0000", 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9078000000000004e-05 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.008056+0000", 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2859000000000003e-05 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131704 7.0 7:0e395443:::benchmark_data_smithi043_74356_object14835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.933800+0000", 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 88.960021436999995, 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.074431347999999994, 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131704 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.953 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933800+0000", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933800+0000", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933801+0000", 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4900000000000005e-07 2024-04-25T05:38:34.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933806+0000", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0939999999999998e-06 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933820+0000", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3518e-05 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.933828+0000", 2024-04-25T05:38:34.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4679999999999992e-06 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.935798+0000", 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019691460000000002 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.935919+0000", 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012138000000000001 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.936048+0000", 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128978 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.938138+0000", 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00208998 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.940341+0000", 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022028719999999998 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.008111+0000", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.067770661999999995 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.008184+0000", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2132000000000007e-05 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:06.008232+0000", 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8269000000000002e-05 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.958 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:143222 7.0 7:0973855a:::benchmark_data_smithi043_74356_object26353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:20.396690+0000", 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 74.497132227999998, 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.075424872000000004, 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 143222 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396690+0000", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396690+0000", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396691+0000", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.584e-06 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396698+0000", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1210000000000001e-06 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396712+0000", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3465999999999999e-05 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396721+0000", 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7620000000000001e-06 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396787+0000", 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5939999999999995e-05 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.396912+0000", 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012533600000000001 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.397039+0000", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012721799999999999 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.399351+0000", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023123649999999998 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.400118+0000", 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076656000000000001 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.471937+0000", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.071818678999999996 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.472071+0000", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013433499999999999 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.472115+0000", 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3506e-05 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.964 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:143465 7.0 7:0114b0cc:::benchmark_data_smithi043_74356_object26596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 74.118791449, 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098687036000000006, 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 143465 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.965 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775032+0000", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.113e-06 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775038+0000", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0249999999999999e-06 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775050+0000", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2473999999999999e-05 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775059+0000", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4199999999999996e-06 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775109+0000", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9225999999999999e-05 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775244+0000", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000135537 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775370+0000", 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012596100000000001 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.777416+0000", 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020459409999999999 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.833867+0000", 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.056450475 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873607+0000", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039740635000000003 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873672+0000", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4692999999999999e-05 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873717+0000", 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5535999999999999e-05 2024-04-25T05:38:34.970 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:163814 7.0 7:09b1f4e8:::benchmark_data_smithi043_74356_object46945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.539700772000003, 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082701040000000003, 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.971 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 163814 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354123+0000", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5579999999999999e-06 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354130+0000", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3900000000000004e-06 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354148+0000", 2024-04-25T05:38:34.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8185999999999998e-05 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354159+0000", 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0832999999999999e-05 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354211+0000", 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1708e-05 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354308+0000", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7609999999999995e-05 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354403+0000", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4476999999999997e-05 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.362773+0000", 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0083697449999999996 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.424762+0000", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061988986000000003 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436757+0000", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011995176999999999 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436792+0000", 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5092000000000002e-05 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436822+0000", 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0278e-05 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:163817 7.0 7:04abd7d8:::benchmark_data_smithi043_74356_object46948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.538744000999998, 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.081835871000000004, 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.977 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 163817 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355079+0000", 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.61e-07 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355084+0000", 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9230000000000001e-06 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355094+0000", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0546e-05 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355100+0000", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7250000000000002e-06 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357186+0000", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020863679999999999 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357361+0000", 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175024 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357481+0000", 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119339 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.362780+0000", 2024-04-25T05:38:34.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005299158 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.424826+0000", 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.062046077999999998 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436861+0000", 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012035426 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436891+0000", 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9765e-05 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436914+0000", 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2758000000000001e-05 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.983 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:163818 7.0 7:036445d7:::benchmark_data_smithi043_74356_object46949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:48.355853+0000", 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.537969101999998, 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.081155153999999993, 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 163818 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355853+0000", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355853+0000", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355853+0000", 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2600000000000002e-07 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355857+0000", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6270000000000002e-06 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355866+0000", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3160000000000003e-06 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355872+0000", 2024-04-25T05:38:34.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5809999999999996e-06 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.360112+0000", 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042402489999999998 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.360255+0000", 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285799999999999 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.360383+0000", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012764300000000001 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.362886+0000", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002503113 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.424875+0000", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061989181999999997 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436950+0000", 2024-04-25T05:38:34.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012075059000000001 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436986+0000", 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6025999999999997e-05 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.437008+0000", 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1874000000000001e-05 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.989 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:175169 7.0 7:0d4e44d2:::benchmark_data_smithi043_74356_object58300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 31.414291048999999, 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13254632699999999, 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175169 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479532+0000", 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.288e-06 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479537+0000", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9239999999999998e-06 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479548+0000", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185e-05 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479556+0000", 2024-04-25T05:38:34.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3139999999999998e-06 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479603+0000", 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7651000000000002e-05 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479689+0000", 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5328999999999996e-05 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479769+0000", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0508999999999998e-05 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.481886+0000", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021172999999999999 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.492164+0000", 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010277632 2024-04-25T05:38:34.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612007+0000", 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.119842849 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612041+0000", 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3958999999999998e-05 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:34.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612077+0000", 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6387000000000002e-05 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195392 7.0 7:00172623:::benchmark_data_smithi043_74356_object78523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.1463157490000002, 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.091797856999999997, 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:34.996 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195392 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:34.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747508+0000", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2330000000000001e-06 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747518+0000", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3039999999999994e-06 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:34.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760394+0000", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012876258 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760415+0000", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0922000000000001e-05 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760811+0000", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039616199999999998 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:34.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760938+0000", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012737299999999999 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.761064+0000", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012562900000000001 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.766816+0000", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057523770000000004 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.831492+0000", 2024-04-25T05:38:35.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.064675361000000001 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839182+0000", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0076902949999999998 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839272+0000", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0019000000000007e-05 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839304+0000", 2024-04-25T05:38:35.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1924000000000001e-05 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195400 7.0 7:0df93f00:::benchmark_data_smithi043_74356_object78531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:30.767706+0000", 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4.126115778, 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.080315955999999994, 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:35.002 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195400 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767706+0000", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767706+0000", 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767707+0000", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.085e-06 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767712+0000", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7210000000000002e-06 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767723+0000", 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1163e-05 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767729+0000", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.246e-06 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767795+0000", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6097000000000001e-05 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.767912+0000", 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011707800000000001 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.768294+0000", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000381386 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.772473+0000", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041794270000000003 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.831762+0000", 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.059288630000000002 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.847889+0000", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.016126807 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.847952+0000", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3200999999999993e-05 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.848022+0000", 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0115000000000002e-05 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:35.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195836 7.0 7:0dc9af6b:::benchmark_data_smithi043_74356_object78967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 3.638008868, 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.103185819, 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195836 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.008 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255814+0000", 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0240000000000001e-06 2024-04-25T05:38:35.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255819+0000", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8130000000000003e-06 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255830+0000", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0781000000000001e-05 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255836+0000", 2024-04-25T05:38:35.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4729999999999997e-06 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255883+0000", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6786999999999998e-05 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255989+0000", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010607600000000001 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256077+0000", 2024-04-25T05:38:35.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8104999999999998e-05 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.257931+0000", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018537289999999999 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.261718+0000", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003787159 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358884+0000", 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097166081000000001 2024-04-25T05:38:35.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358938+0000", 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4336999999999998e-05 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358999+0000", 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0454000000000001e-05 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.013 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:197365 7.0 7:03bbe225:::benchmark_data_smithi043_74356_object80496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:33.113351+0000", 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.7804705320000001, 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078462422000000004, 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 197365 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113351+0000", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113351+0000", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113353+0000", 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6780000000000001e-06 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113361+0000", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4900000000000003e-06 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113383+0000", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2889000000000001e-05 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113393+0000", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8220000000000002e-06 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:38:35.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113479+0000", 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5824999999999999e-05 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113612+0000", 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013339299999999999 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:38:35.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.113734+0000", 2024-04-25T05:38:35.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012155999999999999 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.115920+0000", 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021862610000000001 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.119399+0000", 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034789220000000002 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.191555+0000", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072156313 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.191754+0000", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019834299999999999 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:33.191814+0000", 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9926000000000002e-05 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:38:35.021 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:38:35.021 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:38:35.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:34 smithi060 ceph-mon[30898]: pgmap v222: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:35.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:34 smithi060 ceph-mon[30898]: osdmap e177: 8 total, 8 up, 6 in 2024-04-25T05:38:35.021 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:34 smithi060 ceph-mon[30898]: osdmap e178: 8 total, 8 up, 6 in 2024-04-25T05:38:35.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:35.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:35.172 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:35.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:35.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:35.422 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:35.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:35.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:35.668 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:35.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:35.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:35.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:38:35.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:35.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:36.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:36.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:36.072 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:36.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:36.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:36.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:36.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:36.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:36.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:36.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:36.539 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:36.582 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:38:36.583 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:38:36.583 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:38:36.583 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:38:36.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:36.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:38:36.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:36.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:36.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:36.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:36.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:37.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:37.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:37.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:38:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[27010]: pgmap v225: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[27010]: osdmap e179: 8 total, 8 up, 6 in 2024-04-25T05:38:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2406917747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[37492]: pgmap v225: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[37492]: osdmap e179: 8 total, 8 up, 6 in 2024-04-25T05:38:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2406917747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:36 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:37.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:36 smithi060 ceph-mon[30898]: pgmap v225: 158 pgs: 2 remapped+peering, 8 active+undersized+remapped, 6 active+remapped+backfilling, 142 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 999 op/s; 3550/110556 objects misplaced (3.211%); 42 MiB/s, 673 objects/s recovering 2024-04-25T05:38:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:36 smithi060 ceph-mon[30898]: osdmap e179: 8 total, 8 up, 6 in 2024-04-25T05:38:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2406917747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:36 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:38:37.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:37.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:37.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:38:37.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:37.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:37.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:38:37.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:37.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:38:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.592 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:37.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:37.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:37.702 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:37.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:37.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:37 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:38:37.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:37 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:38:37.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:37 smithi043 ceph-mon[37492]: osdmap e180: 8 total, 8 up, 6 in 2024-04-25T05:38:37.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:37.875 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:37.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:37.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:38.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:38.092 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:37 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:38:38.092 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:37 smithi060 ceph-mon[30898]: osdmap e180: 8 total, 8 up, 6 in 2024-04-25T05:38:38.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:38.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:38.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:37 smithi043 ceph-mon[27010]: osdmap e180: 8 total, 8 up, 6 in 2024-04-25T05:38:38.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:38.254 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:38:38.254 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:38:38.254 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:38:38.254 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:38:38.255 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:38:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.264 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:38.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:38.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:38.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:38.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:38.521 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:38.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:38.648 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:38.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.779 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:38.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:38.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:38.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:38.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:38.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:38:39.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:39.098 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:38 smithi043 ceph-mon[27010]: pgmap v228: 158 pgs: 4 active+undersized+remapped, 5 active+remapped+backfilling, 149 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 225 op/s; 2657/107865 objects misplaced (2.463%); 13 MiB/s, 212 objects/s recovering 2024-04-25T05:38:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:38 smithi043 ceph-mon[37492]: pgmap v228: 158 pgs: 4 active+undersized+remapped, 5 active+remapped+backfilling, 149 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 225 op/s; 2657/107865 objects misplaced (2.463%); 13 MiB/s, 212 objects/s recovering 2024-04-25T05:38:39.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:39.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:39.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:38 smithi060 ceph-mon[30898]: pgmap v228: 158 pgs: 4 active+undersized+remapped, 5 active+remapped+backfilling, 149 active+clean; 2.2 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 225 op/s; 2657/107865 objects misplaced (2.463%); 13 MiB/s, 212 objects/s recovering 2024-04-25T05:38:39.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:39.341 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:39.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:39.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:39.522 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:39.568 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:39.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:39.617 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:39.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:39.681 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:39.721 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:39.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:39.764 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:39.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:39.810 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:39.854 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:39.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:39.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:39.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:39.956 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:39.957 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:40.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:40.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/583729856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:40.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:39 smithi043 ceph-mon[27010]: osdmap e181: 8 total, 8 up, 6 in 2024-04-25T05:38:40.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:38:40.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/583729856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:40.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:39 smithi043 ceph-mon[37492]: osdmap e181: 8 total, 8 up, 6 in 2024-04-25T05:38:40.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:40.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:40.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/583729856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:40.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:39 smithi060 ceph-mon[30898]: osdmap e181: 8 total, 8 up, 6 in 2024-04-25T05:38:40.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:40.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:40.470 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:40.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:40.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:40.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:38:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:40.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:40.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:40.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:40.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:40.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:40.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:40 smithi060 ceph-mon[30898]: pgmap v229: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 473 op/s; 36 MiB/s, 577 objects/s recovering 2024-04-25T05:38:41.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:41.087 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:41.104 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:40 smithi043 ceph-mon[27010]: pgmap v229: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 473 op/s; 36 MiB/s, 577 objects/s recovering 2024-04-25T05:38:41.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:40 smithi043 ceph-mon[37492]: pgmap v229: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 473 op/s; 36 MiB/s, 577 objects/s recovering 2024-04-25T05:38:41.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:41.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:41.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:41.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:41.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:41.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:41.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:41.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:41.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:41.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:41.727 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:41.727 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:41.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:41.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:41.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:41.966 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:41.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:42.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:42.094 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:42.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:42.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:42.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:42.374 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:42.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:42.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:42.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:42.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:42.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:42.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:42.797 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:38:42.803 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:38:42.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:38:42.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:42.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:42.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:43.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:43.031 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[27010]: pgmap v231: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 470 op/s; 36 MiB/s, 573 objects/s recovering 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1786632725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3878205361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[37492]: pgmap v231: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 470 op/s; 36 MiB/s, 573 objects/s recovering 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1786632725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3878205361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.032 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.061 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:43.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:43.171 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:43.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:43.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:43.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-mon[30898]: pgmap v231: 158 pgs: 158 active+clean; 2.1 GiB data, 7.2 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 470 op/s; 36 MiB/s, 573 objects/s recovering 2024-04-25T05:38:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1786632725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:38:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3878205361' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:38:43.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:43.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:43.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:43.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:43.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:43.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:43.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:43.740 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:43.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:38:43.760 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:43.760 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:43.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:43.868 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:43.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:43.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:43.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:43 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:38:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:43 smithi060 ceph-mon[30898]: osdmap e182: 8 total, 8 up, 6 in 2024-04-25T05:38:43.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:44.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:44.136 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:38:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[27010]: osdmap e182: 8 total, 8 up, 6 in 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[37492]: osdmap e182: 8 total, 8 up, 6 in 2024-04-25T05:38:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:44.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:44.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:44.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:38:44.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:44.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:44.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:44.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:38:44.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:44.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:44.672 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:44.738 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:44.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:44.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:44.881 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:38:44.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:44 smithi043 ceph-mon[27010]: pgmap v233: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 31 MiB/s, 491 objects/s recovering 2024-04-25T05:38:44.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:44 smithi043 ceph-mon[27010]: osdmap e183: 8 total, 8 up, 6 in 2024-04-25T05:38:44.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:44 smithi043 ceph-mon[37492]: pgmap v233: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 31 MiB/s, 491 objects/s recovering 2024-04-25T05:38:44.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:44 smithi043 ceph-mon[37492]: osdmap e183: 8 total, 8 up, 6 in 2024-04-25T05:38:44.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:44.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:44.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:44.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:44 smithi060 ceph-mon[30898]: pgmap v233: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 31 MiB/s, 491 objects/s recovering 2024-04-25T05:38:44.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:44 smithi060 ceph-mon[30898]: osdmap e183: 8 total, 8 up, 6 in 2024-04-25T05:38:45.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:45.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:45.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:45.252 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:45.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:45.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:45.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:45.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:45.555 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:45.555 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:45.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:45.673 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:45.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:45.809 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:45.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:45.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:45.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:46.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:46.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:46.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:46.221 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:46.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:46.260 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:46.300 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:46.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:38:46.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:46.344 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:46.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:46.388 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:46.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:46.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:46.472 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:46.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:46.517 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:46.524 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:46.525 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:46.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:46.632 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:46.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:38:46.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.791 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:46.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:46.846 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:38:46.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:46.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:46.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:47.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:47.060 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[27010]: pgmap v235: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1031083812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3014168911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[37492]: pgmap v235: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-04-25T05:38:47.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1031083812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3014168911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:47.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:47.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:47.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:46 smithi060 ceph-mon[30898]: pgmap v235: 158 pgs: 158 active+clean; 1.7 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.11k op/s 2024-04-25T05:38:47.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1031083812' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:38:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3014168911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:47.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:47.344 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:47.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:47.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:47.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:47.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:47.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:47.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:38:47.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:47.884 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:47.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:47.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:48.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:38:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[27010]: osdmap e184: 8 total, 8 up, 6 in 2024-04-25T05:38:48.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:38:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[37492]: osdmap e184: 8 total, 8 up, 6 in 2024-04-25T05:38:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:48.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:38:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:47 smithi060 ceph-mon[30898]: osdmap e184: 8 total, 8 up, 6 in 2024-04-25T05:38:48.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:38:48.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:48.264 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:48.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:48.355 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:48.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:48.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:48.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:48.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:48.617 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:48.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:48.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:48.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:48.839 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:48.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:48.967 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:48.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:49.065 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:48 smithi043 ceph-mon[27010]: pgmap v237: 158 pgs: 158 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-04-25T05:38:49.066 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:48 smithi043 ceph-mon[37492]: pgmap v237: 158 pgs: 158 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-04-25T05:38:49.066 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:38:49.066 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:38:49.066 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:38:49.067 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:38:49.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:49.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:49.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:49.240 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:49.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:48 smithi060 ceph-mon[30898]: pgmap v237: 158 pgs: 158 active+clean; 1.5 GiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-04-25T05:38:49.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:49.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:49.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:49.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:49.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:49.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:49.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:49.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:49.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:49.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:50.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:50.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:50.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:50.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:38:50.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:50.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:38:50.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.288 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:50.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:50.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:50.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:50.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:50.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:50.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:50.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:50.848 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:50.949 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:50.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:50.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:51.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:51.122 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:50 smithi043 ceph-mon[27010]: pgmap v238: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 1.63k op/s 2024-04-25T05:38:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:50 smithi043 ceph-mon[27010]: osdmap e185: 8 total, 8 up, 6 in 2024-04-25T05:38:51.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:50 smithi043 ceph-mon[37492]: pgmap v238: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 1.63k op/s 2024-04-25T05:38:51.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:50 smithi043 ceph-mon[37492]: osdmap e185: 8 total, 8 up, 6 in 2024-04-25T05:38:51.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:51.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:50 smithi060 ceph-mon[30898]: pgmap v238: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 1.63k op/s 2024-04-25T05:38:51.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:50 smithi060 ceph-mon[30898]: osdmap e185: 8 total, 8 up, 6 in 2024-04-25T05:38:51.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:51.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:51.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:51.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:51.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:51.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:51.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:51.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:51.792 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:51.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:51.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:51.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:52.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:52.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:52.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:52.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:38:52.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:52.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:52.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:52.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:52.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:52.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:52.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:52.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:52.768 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:52 smithi043 ceph-mon[27010]: pgmap v240: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 962 op/s 2024-04-25T05:38:52.769 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:52 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:38:52.789 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:52.829 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:52.865 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:52.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:52.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:52.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:53.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:53.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:52 smithi043 ceph-mon[37492]: pgmap v240: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 962 op/s 2024-04-25T05:38:53.028 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:52 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:38:53.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:53.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:53.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:53.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:52 smithi060 ceph-mon[30898]: pgmap v240: 158 pgs: 158 active+clean; 1.2 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 962 op/s 2024-04-25T05:38:53.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:52 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:38:53.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:38:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:38:53.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:53.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:53.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.487 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:53.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:53.587 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:53.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:53.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:53.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:53.799 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:38:53.808 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:38:53.810 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:38:53.810 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:38:53.810 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:38:53.811 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:38:53.811 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:38:53.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/728010590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/728010590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:53.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:38:53.833 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:38:53.833 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:53.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:53.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:53.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:54.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.054 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:54.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:54.061 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:38:54.106 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:38:54.147 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:38:54.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:54.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:54.193 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:38:54.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:38:54.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/728010590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:38:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:38:54.274 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:38:54.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:54.316 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:38:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.317 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:54.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:54.354 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:38:54.357 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:38:54.358 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:38:54.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:54.443 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:54.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:54.571 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:54.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:54.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:54.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:54.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:54.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:54.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:55.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: pgmap v241: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:38:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: osdmap e186: 8 total, 8 up, 6 in 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2107474438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: 7.d scrub starts 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[27010]: osdmap e187: 8 total, 8 up, 6 in 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: pgmap v241: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: osdmap e186: 8 total, 8 up, 6 in 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2107474438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:55.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: 7.d scrub starts 2024-04-25T05:38:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:54 smithi043 ceph-mon[37492]: osdmap e187: 8 total, 8 up, 6 in 2024-04-25T05:38:55.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:55.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: pgmap v241: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2196108328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: osdmap e186: 8 total, 8 up, 6 in 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2107474438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: 7.d scrub starts 2024-04-25T05:38:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:54 smithi060 ceph-mon[30898]: osdmap e187: 8 total, 8 up, 6 in 2024-04-25T05:38:55.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:55.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:55.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:55.642 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:55.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:55.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:55.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:38:55.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:55.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:38:56.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:56.046 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:38:56.046 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:38:56.046 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: 7.d scrub ok 2024-04-25T05:38:56.046 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: 7.d scrub ok 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:38:56.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:55 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:38:56.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:56.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:56.196 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:56.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:38:56.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:38:56.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: 7.d scrub ok 2024-04-25T05:38:56.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:38:56.218 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:38:56.218 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:38:56.218 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:55 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:38:56.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:56.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:56.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:56.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:56.566 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:56.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:56.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:56.732 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:56.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:56.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:56.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:56.985 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:56.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[27010]: pgmap v244: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:38:57.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:56 smithi043 ceph-mon[37492]: pgmap v244: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:38:57.078 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:56 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:38:57.078 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:56 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:38:57.078 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:56 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:38:57.078 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:56 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:38:57.078 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:56 smithi060 ceph-mon[30898]: pgmap v244: 158 pgs: 158 active+clean; 876 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:38:57.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:57.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:57.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:57.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:57.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:57.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:57.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:57.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:38:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:57.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:57.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:57.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:57.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:57.792 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:57.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:57.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:57.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:58.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:38:58.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 7.1 scrub starts 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 7.1 scrub ok 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: osdmap e188: 8 total, 8 up, 6 in 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 7.6 scrub starts 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:38:58.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 7.1 scrub starts 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 7.1 scrub ok 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: osdmap e188: 8 total, 8 up, 6 in 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 7.6 scrub starts 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:38:58.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:38:58.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:38:58.058 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:57 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:38:58.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:38:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:58.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:58.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:58.229 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:58.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 7.1 scrub starts 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 7.1 scrub ok 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: osdmap e188: 8 total, 8 up, 6 in 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 7.6 scrub starts 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:38:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:57 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:38:58.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:58.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:58.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:38:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.488 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:38:58.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:58.614 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:58.621 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:58.671 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:58.683 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:38:58.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:58.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:58.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:58.873 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:58.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:58.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:38:59.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 7.6 scrub ok 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: pgmap v246: 158 pgs: 158 active+clean; 751 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:38:59.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 7.6 scrub ok 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: pgmap v246: 158 pgs: 158 active+clean; 751 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:38:59.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:58 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:38:59.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:38:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:59.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:38:59.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 7.6 scrub ok 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: pgmap v246: 158 pgs: 158 active+clean; 751 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 1.39k op/s 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:38:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:58 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:38:59.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:38:59.289 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:38:59.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:38:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:38:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:38:59.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:59.580 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:59.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:59.707 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:38:59.707 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:38:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:59.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:38:59.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:38:59.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:38:59.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:38:59.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:38:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:38:59.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:00.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[27010]: osdmap e189: 8 total, 8 up, 6 in 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:38:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:39:00.011 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:38:59 smithi043 ceph-mon[37492]: osdmap e189: 8 total, 8 up, 6 in 2024-04-25T05:39:00.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:00.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:00.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:00.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:00.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:59 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:39:00.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:59 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:39:00.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:59 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:39:00.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:59 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:39:00.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:38:59 smithi060 ceph-mon[30898]: osdmap e189: 8 total, 8 up, 6 in 2024-04-25T05:39:00.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:00.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:00.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:00.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:00.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:39:00.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:39:00.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:00.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:00.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:00.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:39:00.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:00.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:01.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:01.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: pgmap v247: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:39:01.127 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:39:01.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:39:01.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:39:01.128 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:39:01.129 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:39:01.129 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:39:01.129 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: pgmap v247: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:39:01.130 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:00 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:39:01.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:01.186 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:01.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: pgmap v247: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:39:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:00 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:39:01.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:01.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:01.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:01.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:01.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:01.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:01.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:01.715 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:01.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:39:01.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:39:01.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:01.982 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:01.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:02.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:02.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: osdmap e190: 8 total, 8 up, 6 in 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:39:02.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/301711319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: osdmap e190: 8 total, 8 up, 6 in 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:02.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/301711319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:01 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3667149581' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: osdmap e190: 8 total, 8 up, 6 in 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/301711319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:01 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:02.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:02.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:02.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:02.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:02.526 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:02.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:02.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:02.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:02.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:02.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:02.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:03.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:03.046 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:03.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:39:03.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: pgmap v250: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: osdmap e191: 8 total, 8 up, 6 in 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:39:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: pgmap v250: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: osdmap e191: 8 total, 8 up, 6 in 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:03.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:03.173 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:03.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: pgmap v250: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 460 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: osdmap e191: 8 total, 8 up, 6 in 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:03.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:03.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:03.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:03.429 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:03.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:03.559 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:03.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:03.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:03.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:03.825 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:03.877 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 116866 objects 2024-04-25T05:39:03.877 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :137.125 2024-04-25T05:39:03.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:03.946 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:03.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:04.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:04.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: 4.1e deep-scrub starts 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: 4.1e deep-scrub ok 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: osdmap e192: 8 total, 8 up, 6 in 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: 6.13 deep-scrub starts 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[27010]: 6.13 deep-scrub ok 2024-04-25T05:39:04.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: 4.1e deep-scrub starts 2024-04-25T05:39:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: 4.1e deep-scrub ok 2024-04-25T05:39:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: osdmap e192: 8 total, 8 up, 6 in 2024-04-25T05:39:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: 6.13 deep-scrub starts 2024-04-25T05:39:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:03 smithi043 ceph-mon[37492]: 6.13 deep-scrub ok 2024-04-25T05:39:04.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: 4.1e deep-scrub starts 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: 4.1e deep-scrub ok 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: osdmap e192: 8 total, 8 up, 6 in 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: 6.13 deep-scrub starts 2024-04-25T05:39:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:03 smithi060 ceph-mon[30898]: 6.13 deep-scrub ok 2024-04-25T05:39:04.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:04.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.439 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:04.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:04.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:04.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:04.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:04.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:04.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:04.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:05.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:05.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:39:05.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:39:05.070 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: pgmap v253: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[37492]: osdmap e193: 8 total, 8 up, 6 in 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: pgmap v253: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:39:05.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:39:05.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:04 smithi043 ceph-mon[27010]: osdmap e193: 8 total, 8 up, 6 in 2024-04-25T05:39:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.072 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:05.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:05.120 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:05.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:05.198 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:05.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:39:05.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: pgmap v253: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.51k op/s 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:39:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:04 smithi060 ceph-mon[30898]: osdmap e193: 8 total, 8 up, 6 in 2024-04-25T05:39:05.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:05.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:05.413 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:05.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:05.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:05.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:05.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:05.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:05.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:05.935 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:05.935 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:05 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:39:05.935 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:05 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:39:05.935 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:05 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:39:05.936 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:05 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:39:05.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:05.942 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:05.961 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:05.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:05.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:05.981 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:06.002 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:06.024 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:06.045 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:06.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:06.066 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:06.068 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:06.088 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:06.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:06.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:06.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:05 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:39:06.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:05 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:39:06.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:06.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:06.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:06.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:06.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:06.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:06.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:06.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:06.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:06.829 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:06.890 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:39:06.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:06.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:06.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:07.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:07.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: pgmap v255: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3961288004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: 6.1f scrub starts 2024-04-25T05:39:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: 6.1f scrub ok 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: pgmap v255: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3961288004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: 6.1f scrub starts 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: 6.1f scrub ok 2024-04-25T05:39:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:39:07.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:07.201 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: pgmap v255: 158 pgs: 158 active+clean; 76 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.22k op/s 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3961288004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: 6.1f scrub starts 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: 6.1f scrub ok 2024-04-25T05:39:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:39:07.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:07.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:07.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:07.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:07.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:07.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:07.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:07.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:07.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:07.823 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:07.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:07.951 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:07.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:08.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:08.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:08.080 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:08.080 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:39:08.080 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:39:08.081 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:08.175 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:39:08.175 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:07 smithi043 ceph-mon[37492]: osdmap e194: 8 total, 8 up, 6 in 2024-04-25T05:39:08.175 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:39:08.175 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:07 smithi043 ceph-mon[27010]: osdmap e194: 8 total, 8 up, 6 in 2024-04-25T05:39:08.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3956684988' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:39:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:07 smithi060 ceph-mon[30898]: osdmap e194: 8 total, 8 up, 6 in 2024-04-25T05:39:08.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:08.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:08.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:08.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:08.480 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-25T05:39:08.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-25T05:39:08.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:08.582 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:39:08.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:39:08.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:08.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:08.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:08.941 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:08.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:09.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:09.067 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:09.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:09.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:09.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:08 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:39:09.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:08 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:39:09.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:08 smithi060 ceph-mon[30898]: pgmap v257: 158 pgs: 158 active+clean; 46 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:09.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:08 smithi060 ceph-mon[30898]: 7.4 scrub starts 2024-04-25T05:39:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:08 smithi060 ceph-mon[30898]: 7.4 scrub ok 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[27010]: pgmap v257: 158 pgs: 158 active+clean; 46 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[27010]: 7.4 scrub starts 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[27010]: 7.4 scrub ok 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:39:09.280 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:39:09.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[37492]: pgmap v257: 158 pgs: 158 active+clean; 46 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s 2024-04-25T05:39:09.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[37492]: 7.4 scrub starts 2024-04-25T05:39:09.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:08 smithi043 ceph-mon[37492]: 7.4 scrub ok 2024-04-25T05:39:09.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:09.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:39:09.319 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:09.368 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:09.369 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:09.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:09.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:09.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:09.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:09.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:39:09.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:09.696 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:09.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:09.825 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:09.828 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:09.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:09.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:10.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:10.076 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:10.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:10.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 6.1b scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: 6.1b scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[27010]: osdmap e195: 8 total, 8 up, 6 in 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 6.1b scrub starts 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: 6.1b scrub ok 2024-04-25T05:39:10.223 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:09 smithi043 ceph-mon[37492]: osdmap e195: 8 total, 8 up, 6 in 2024-04-25T05:39:10.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:39:10.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:39:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:39:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:39:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 6.1b scrub starts 2024-04-25T05:39:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: 6.1b scrub ok 2024-04-25T05:39:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:09 smithi060 ceph-mon[30898]: osdmap e195: 8 total, 8 up, 6 in 2024-04-25T05:39:10.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:10.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:10.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:10.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:10.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:10.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:10.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:10.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:10.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:10.916 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:39:10.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_0' removed 2024-04-25T05:39:10.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:10.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:11.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:11.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 6.7 deep-scrub starts 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 6.7 deep-scrub ok 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: pgmap v258: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 779 op/s 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 4.2 deep-scrub starts 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 4.2 deep-scrub ok 2024-04-25T05:39:11.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/679577180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 6.7 deep-scrub starts 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 6.7 deep-scrub ok 2024-04-25T05:39:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: pgmap v258: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 779 op/s 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 4.2 deep-scrub starts 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 4.2 deep-scrub ok 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/679577180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:39:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:39:11.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:11.199 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:39:11.202 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:39:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 6.7 deep-scrub starts 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 6.7 deep-scrub ok 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: pgmap v258: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 779 op/s 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 4.2 deep-scrub starts 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 4.2 deep-scrub ok 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/679577180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:39:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:39:11.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:11.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:11.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:11.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:11.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:11.671 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:11.671 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:11.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:11.699 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:11.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:11.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:11.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:11.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:12.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:12.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: osdmap e196: 8 total, 8 up, 6 in 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:39:12.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[27010]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: osdmap e196: 8 total, 8 up, 6 in 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:39:12.084 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:11 smithi043 ceph-mon[37492]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:12.111 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-25T05:39:10.898433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1302,"num_read_kb":1022,"num_write":628,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2119,"ondisk_log_size":2119,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1069560,"kb_used_data":9264,"kb_used_omap":9,"kb_used_meta":1060214,"kb_avail":561401352,"statfs":{"total":575970213888,"available":574874984448,"internally_reserved":0,"allocated":9486336,"data_stored":5855109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":1085659813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-79560728,"num_objects":-1215,"num_object_clones":0,"num_object_copies":-3645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1215,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-232519,"num_write_kb":-7479425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-53193,"num_bytes_recovered":-3485794352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.139298"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":262,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886806+0000","last_change":"2024-04-25T05:37:49.635619+0000","last_active":"2024-04-25T05:39:06.886806+0000","last_peered":"2024-04-25T05:39:06.886806+0000","last_clean":"2024-04-25T05:39:06.886806+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:39:06.886806+0000","last_undegraded":"2024-04-25T05:39:06.886806+0000","last_fullsized":"2024-04-25T05:39:06.886806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:49.635538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:49.635538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":267,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887745+0000","last_change":"2024-04-25T05:35:24.260778+0000","last_active":"2024-04-25T05:39:06.887745+0000","last_peered":"2024-04-25T05:39:06.887745+0000","last_clean":"2024-04-25T05:39:06.887745+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:39:06.887745+0000","last_undegraded":"2024-04-25T05:39:06.887745+0000","last_fullsized":"2024-04-25T05:39:06.887745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:24.260670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:24.260670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":88,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380723+0000","last_change":"2024-04-25T05:39:07.379627+0000","last_active":"2024-04-25T05:39:07.380723+0000","last_peered":"2024-04-25T05:39:07.380723+0000","last_clean":"2024-04-25T05:39:07.380723+0000","last_became_active":"2024-04-25T05:38:13.237423+0000","last_became_peered":"2024-04-25T05:38:13.237423+0000","last_unstale":"2024-04-25T05:39:07.380723+0000","last_undegraded":"2024-04-25T05:39:07.380723+0000","last_fullsized":"2024-04-25T05:39:07.380723+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.552329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":316,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886387+0000","last_change":"2024-04-25T05:36:54.265352+0000","last_active":"2024-04-25T05:39:06.886387+0000","last_peered":"2024-04-25T05:39:06.886387+0000","last_clean":"2024-04-25T05:39:06.886387+0000","last_became_active":"2024-04-25T05:36:54.243668+0000","last_became_peered":"2024-04-25T05:36:54.243668+0000","last_unstale":"2024-04-25T05:39:06.886387+0000","last_undegraded":"2024-04-25T05:39:06.886387+0000","last_fullsized":"2024-04-25T05:39:06.886387+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":261,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887923+0000","last_change":"2024-04-25T05:36:54.236185+0000","last_active":"2024-04-25T05:39:06.887923+0000","last_peered":"2024-04-25T05:39:06.887923+0000","last_clean":"2024-04-25T05:39:06.887923+0000","last_became_active":"2024-04-25T05:36:54.235695+0000","last_became_peered":"2024-04-25T05:36:54.235695+0000","last_unstale":"2024-04-25T05:39:06.887923+0000","last_undegraded":"2024-04-25T05:39:06.887923+0000","last_fullsized":"2024-04-25T05:39:06.887923+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":272,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886581+0000","last_change":"2024-04-25T05:37:47.617448+0000","last_active":"2024-04-25T05:39:06.886581+0000","last_peered":"2024-04-25T05:39:06.886581+0000","last_clean":"2024-04-25T05:39:06.886581+0000","last_became_active":"2024-04-25T05:36:54.229992+0000","last_became_peered":"2024-04-25T05:36:54.229992+0000","last_unstale":"2024-04-25T05:39:06.886581+0000","last_undegraded":"2024-04-25T05:39:06.886581+0000","last_fullsized":"2024-04-25T05:39:06.886581+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:47.617351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:47.617351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:16.683166+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889541+0000","last_change":"2024-04-25T05:36:54.230655+0000","last_active":"2024-04-25T05:39:06.889541+0000","last_peered":"2024-04-25T05:39:06.889541+0000","last_clean":"2024-04-25T05:39:06.889541+0000","last_became_active":"2024-04-25T05:36:54.230169+0000","last_became_peered":"2024-04-25T05:36:54.230169+0000","last_unstale":"2024-04-25T05:39:06.889541+0000","last_undegraded":"2024-04-25T05:39:06.889541+0000","last_fullsized":"2024-04-25T05:39:06.889541+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":366,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889562+0000","last_change":"2024-04-25T05:38:13.261747+0000","last_active":"2024-04-25T05:39:06.889562+0000","last_peered":"2024-04-25T05:39:06.889562+0000","last_clean":"2024-04-25T05:39:06.889562+0000","last_became_active":"2024-04-25T05:38:13.246798+0000","last_became_peered":"2024-04-25T05:38:13.246798+0000","last_unstale":"2024-04-25T05:39:06.889562+0000","last_undegraded":"2024-04-25T05:39:06.889562+0000","last_fullsized":"2024-04-25T05:39:06.889562+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":88,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886030+0000","last_change":"2024-04-25T05:38:59.924478+0000","last_active":"2024-04-25T05:39:06.886030+0000","last_peered":"2024-04-25T05:39:06.886030+0000","last_clean":"2024-04-25T05:39:06.886030+0000","last_became_active":"2024-04-25T05:38:13.304062+0000","last_became_peered":"2024-04-25T05:38:13.304062+0000","last_unstale":"2024-04-25T05:39:06.886030+0000","last_undegraded":"2024-04-25T05:39:06.886030+0000","last_fullsized":"2024-04-25T05:39:06.886030+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:02.578128+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":273,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379934+0000","last_change":"2024-04-25T05:38:13.239184+0000","last_active":"2024-04-25T05:39:07.379934+0000","last_peered":"2024-04-25T05:39:07.379934+0000","last_clean":"2024-04-25T05:39:07.379934+0000","last_became_active":"2024-04-25T05:38:13.237479+0000","last_became_peered":"2024-04-25T05:38:13.237479+0000","last_unstale":"2024-04-25T05:39:07.379934+0000","last_undegraded":"2024-04-25T05:39:07.379934+0000","last_fullsized":"2024-04-25T05:39:07.379934+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":289,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380279+0000","last_change":"2024-04-25T05:36:54.244217+0000","last_active":"2024-04-25T05:39:07.380279+0000","last_peered":"2024-04-25T05:39:07.380279+0000","last_clean":"2024-04-25T05:39:07.380279+0000","last_became_active":"2024-04-25T05:36:54.243924+0000","last_became_peered":"2024-04-25T05:36:54.243924+0000","last_unstale":"2024-04-25T05:39:07.380279+0000","last_undegraded":"2024-04-25T05:39:07.380279+0000","last_fullsized":"2024-04-25T05:39:07.380279+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":331,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886912+0000","last_change":"2024-04-25T05:38:13.254897+0000","last_active":"2024-04-25T05:39:06.886912+0000","last_peered":"2024-04-25T05:39:06.886912+0000","last_clean":"2024-04-25T05:39:06.886912+0000","last_became_active":"2024-04-25T05:38:13.239434+0000","last_became_peered":"2024-04-25T05:38:13.239434+0000","last_unstale":"2024-04-25T05:39:06.886912+0000","last_undegraded":"2024-04-25T05:39:06.886912+0000","last_fullsized":"2024-04-25T05:39:06.886912+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":271,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379369+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:39:07.379369+0000","last_peered":"2024-04-25T05:39:07.379369+0000","last_clean":"2024-04-25T05:39:07.379369+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:39:07.379369+0000","last_undegraded":"2024-04-25T05:39:07.379369+0000","last_fullsized":"2024-04-25T05:39:07.379369+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":273,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061759+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:39:05.061759+0000","last_peered":"2024-04-25T05:39:05.061759+0000","last_clean":"2024-04-25T05:39:05.061759+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:39:05.061759+0000","last_undegraded":"2024-04-25T05:39:05.061759+0000","last_fullsized":"2024-04-25T05:39:05.061759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":283,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380252+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:39:07.380252+0000","last_peered":"2024-04-25T05:39:07.380252+0000","last_clean":"2024-04-25T05:39:07.380252+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:39:07.380252+0000","last_undegraded":"2024-04-25T05:39:07.380252+0000","last_fullsized":"2024-04-25T05:39:07.380252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":306,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887381+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:39:06.887381+0000","last_peered":"2024-04-25T05:39:06.887381+0000","last_clean":"2024-04-25T05:39:06.887381+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:39:06.887381+0000","last_undegraded":"2024-04-25T05:39:06.887381+0000","last_fullsized":"2024-04-25T05:39:06.887381+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":268,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888105+0000","last_change":"2024-04-25T05:36:54.245109+0000","last_active":"2024-04-25T05:39:06.888105+0000","last_peered":"2024-04-25T05:39:06.888105+0000","last_clean":"2024-04-25T05:39:06.888105+0000","last_became_active":"2024-04-25T05:36:54.244292+0000","last_became_peered":"2024-04-25T05:36:54.244292+0000","last_unstale":"2024-04-25T05:39:06.888105+0000","last_undegraded":"2024-04-25T05:39:06.888105+0000","last_fullsized":"2024-04-25T05:39:06.888105+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888497+0000","last_change":"2024-04-25T05:38:13.285789+0000","last_active":"2024-04-25T05:39:06.888497+0000","last_peered":"2024-04-25T05:39:06.888497+0000","last_clean":"2024-04-25T05:39:06.888497+0000","last_became_active":"2024-04-25T05:38:13.285434+0000","last_became_peered":"2024-04-25T05:38:13.285434+0000","last_unstale":"2024-04-25T05:39:06.888497+0000","last_undegraded":"2024-04-25T05:39:06.888497+0000","last_fullsized":"2024-04-25T05:39:06.888497+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":80,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887314+0000","last_change":"2024-04-25T05:38:13.289898+0000","last_active":"2024-04-25T05:39:06.887314+0000","last_peered":"2024-04-25T05:39:06.887314+0000","last_clean":"2024-04-25T05:39:06.887314+0000","last_became_active":"2024-04-25T05:38:13.289539+0000","last_became_peered":"2024-04-25T05:38:13.289539+0000","last_unstale":"2024-04-25T05:39:06.887314+0000","last_undegraded":"2024-04-25T05:39:06.887314+0000","last_fullsized":"2024-04-25T05:39:06.887314+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":308,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.762008+0000","last_change":"2024-04-25T05:37:50.872876+0000","last_active":"2024-04-25T05:39:04.762008+0000","last_peered":"2024-04-25T05:39:04.762008+0000","last_clean":"2024-04-25T05:39:04.762008+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:39:04.762008+0000","last_undegraded":"2024-04-25T05:39:04.762008+0000","last_fullsized":"2024-04-25T05:39:04.762008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:37:50.872688+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":269,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889848+0000","last_change":"2024-04-25T05:39:06.146701+0000","last_active":"2024-04-25T05:39:06.889848+0000","last_peered":"2024-04-25T05:39:06.889848+0000","last_clean":"2024-04-25T05:39:06.889848+0000","last_became_active":"2024-04-25T05:36:54.237282+0000","last_became_peered":"2024-04-25T05:36:54.237282+0000","last_unstale":"2024-04-25T05:39:06.889848+0000","last_undegraded":"2024-04-25T05:39:06.889848+0000","last_fullsized":"2024-04-25T05:39:06.889848+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:06.486556+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":266,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887747+0000","last_change":"2024-04-25T05:38:54.633069+0000","last_active":"2024-04-25T05:39:06.887747+0000","last_peered":"2024-04-25T05:39:06.887747+0000","last_clean":"2024-04-25T05:39:06.887747+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:39:06.887747+0000","last_undegraded":"2024-04-25T05:39:06.887747+0000","last_fullsized":"2024-04-25T05:39:06.887747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.565126+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":274,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887051+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:39:06.887051+0000","last_peered":"2024-04-25T05:39:06.887051+0000","last_clean":"2024-04-25T05:39:06.887051+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:39:06.887051+0000","last_undegraded":"2024-04-25T05:39:06.887051+0000","last_fullsized":"2024-04-25T05:39:06.887051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":298,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889499+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:39:06.889499+0000","last_peered":"2024-04-25T05:39:06.889499+0000","last_clean":"2024-04-25T05:39:06.889499+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:39:06.889499+0000","last_undegraded":"2024-04-25T05:39:06.889499+0000","last_fullsized":"2024-04-25T05:39:06.889499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":270,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062461+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:39:05.062461+0000","last_peered":"2024-04-25T05:39:05.062461+0000","last_clean":"2024-04-25T05:39:05.062461+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:39:05.062461+0000","last_undegraded":"2024-04-25T05:39:05.062461+0000","last_fullsized":"2024-04-25T05:39:05.062461+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"130'9","reported_seq":200,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888088+0000","last_change":"2024-04-25T05:36:54.240381+0000","last_active":"2024-04-25T05:39:06.888088+0000","last_peered":"2024-04-25T05:39:06.888088+0000","last_clean":"2024-04-25T05:39:06.888088+0000","last_became_active":"2024-04-25T05:36:54.232031+0000","last_became_peered":"2024-04-25T05:36:54.232031+0000","last_unstale":"2024-04-25T05:39:06.888088+0000","last_undegraded":"2024-04-25T05:39:06.888088+0000","last_fullsized":"2024-04-25T05:39:06.888088+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":284,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887648+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:39:06.887648+0000","last_peered":"2024-04-25T05:39:06.887648+0000","last_clean":"2024-04-25T05:39:06.887648+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:39:06.887648+0000","last_undegraded":"2024-04-25T05:39:06.887648+0000","last_fullsized":"2024-04-25T05:39:06.887648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":152,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062952+0000","last_change":"2024-04-25T05:39:02.184665+0000","last_active":"2024-04-25T05:39:05.062952+0000","last_peered":"2024-04-25T05:39:05.062952+0000","last_clean":"2024-04-25T05:39:05.062952+0000","last_became_active":"2024-04-25T05:38:13.281967+0000","last_became_peered":"2024-04-25T05:38:13.281967+0000","last_unstale":"2024-04-25T05:39:05.062952+0000","last_undegraded":"2024-04-25T05:39:05.062952+0000","last_fullsized":"2024-04-25T05:39:05.062952+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.101025+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":263,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888096+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:39:06.888096+0000","last_peered":"2024-04-25T05:39:06.888096+0000","last_clean":"2024-04-25T05:39:06.888096+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:39:06.888096+0000","last_undegraded":"2024-04-25T05:39:06.888096+0000","last_fullsized":"2024-04-25T05:39:06.888096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":86,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062750+0000","last_change":"2024-04-25T05:39:04.108082+0000","last_active":"2024-04-25T05:39:05.062750+0000","last_peered":"2024-04-25T05:39:05.062750+0000","last_clean":"2024-04-25T05:39:05.062750+0000","last_became_active":"2024-04-25T05:38:13.282204+0000","last_became_peered":"2024-04-25T05:38:13.282204+0000","last_unstale":"2024-04-25T05:39:05.062750+0000","last_undegraded":"2024-04-25T05:39:05.062750+0000","last_fullsized":"2024-04-25T05:39:05.062750+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.765638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888171+0000","last_change":"2024-04-25T05:38:13.305988+0000","last_active":"2024-04-25T05:39:06.888171+0000","last_peered":"2024-04-25T05:39:06.888171+0000","last_clean":"2024-04-25T05:39:06.888171+0000","last_became_active":"2024-04-25T05:38:13.305466+0000","last_became_peered":"2024-04-25T05:38:13.305466+0000","last_unstale":"2024-04-25T05:39:06.888171+0000","last_undegraded":"2024-04-25T05:39:06.888171+0000","last_fullsized":"2024-04-25T05:39:06.888171+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":233,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380899+0000","last_change":"2024-04-25T05:38:13.323005+0000","last_active":"2024-04-25T05:39:07.380899+0000","last_peered":"2024-04-25T05:39:07.380899+0000","last_clean":"2024-04-25T05:39:07.380899+0000","last_became_active":"2024-04-25T05:38:13.297139+0000","last_became_peered":"2024-04-25T05:38:13.297139+0000","last_unstale":"2024-04-25T05:39:07.380899+0000","last_undegraded":"2024-04-25T05:39:07.380899+0000","last_fullsized":"2024-04-25T05:39:07.380899+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:37:35.189581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:35.189581+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":270,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061885+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:39:05.061885+0000","last_peered":"2024-04-25T05:39:05.061885+0000","last_clean":"2024-04-25T05:39:05.061885+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:39:05.061885+0000","last_undegraded":"2024-04-25T05:39:05.061885+0000","last_fullsized":"2024-04-25T05:39:05.061885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":147,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886521+0000","last_change":"2024-04-25T05:38:13.290337+0000","last_active":"2024-04-25T05:39:06.886521+0000","last_peered":"2024-04-25T05:39:06.886521+0000","last_clean":"2024-04-25T05:39:06.886521+0000","last_became_active":"2024-04-25T05:38:13.289541+0000","last_became_peered":"2024-04-25T05:38:13.289541+0000","last_unstale":"2024-04-25T05:39:06.886521+0000","last_undegraded":"2024-04-25T05:39:06.886521+0000","last_fullsized":"2024-04-25T05:39:06.886521+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":80,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380769+0000","last_change":"2024-04-25T05:38:13.296470+0000","last_active":"2024-04-25T05:39:07.380769+0000","last_peered":"2024-04-25T05:39:07.380769+0000","last_clean":"2024-04-25T05:39:07.380769+0000","last_became_active":"2024-04-25T05:38:13.295979+0000","last_became_peered":"2024-04-25T05:38:13.295979+0000","last_unstale":"2024-04-25T05:39:07.380769+0000","last_undegraded":"2024-04-25T05:39:07.380769+0000","last_fullsized":"2024-04-25T05:39:07.380769+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":289,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888310+0000","last_change":"2024-04-25T05:36:54.262135+0000","last_active":"2024-04-25T05:39:06.888310+0000","last_peered":"2024-04-25T05:39:06.888310+0000","last_clean":"2024-04-25T05:39:06.888310+0000","last_became_active":"2024-04-25T05:36:54.244002+0000","last_became_peered":"2024-04-25T05:36:54.244002+0000","last_unstale":"2024-04-25T05:39:06.888310+0000","last_undegraded":"2024-04-25T05:39:06.888310+0000","last_fullsized":"2024-04-25T05:39:06.888310+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":275,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888310+0000","last_change":"2024-04-25T05:39:03.160875+0000","last_active":"2024-04-25T05:39:06.888310+0000","last_peered":"2024-04-25T05:39:06.888310+0000","last_clean":"2024-04-25T05:39:06.888310+0000","last_became_active":"2024-04-25T05:38:13.245676+0000","last_became_peered":"2024-04-25T05:38:13.245676+0000","last_unstale":"2024-04-25T05:39:06.888310+0000","last_undegraded":"2024-04-25T05:39:06.888310+0000","last_fullsized":"2024-04-25T05:39:06.888310+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.160752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.528850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":285,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061680+0000","last_change":"2024-04-25T05:38:13.245873+0000","last_active":"2024-04-25T05:39:05.061680+0000","last_peered":"2024-04-25T05:39:05.061680+0000","last_clean":"2024-04-25T05:39:05.061680+0000","last_became_active":"2024-04-25T05:38:13.245293+0000","last_became_peered":"2024-04-25T05:38:13.245293+0000","last_unstale":"2024-04-25T05:39:05.061680+0000","last_undegraded":"2024-04-25T05:39:05.061680+0000","last_fullsized":"2024-04-25T05:39:05.061680+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":275,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380287+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:39:07.380287+0000","last_peered":"2024-04-25T05:39:07.380287+0000","last_clean":"2024-04-25T05:39:07.380287+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:39:07.380287+0000","last_undegraded":"2024-04-25T05:39:07.380287+0000","last_fullsized":"2024-04-25T05:39:07.380287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":322,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889525+0000","last_change":"2024-04-25T05:36:54.252887+0000","last_active":"2024-04-25T05:39:06.889525+0000","last_peered":"2024-04-25T05:39:06.889525+0000","last_clean":"2024-04-25T05:39:06.889525+0000","last_became_active":"2024-04-25T05:36:54.232522+0000","last_became_peered":"2024-04-25T05:36:54.232522+0000","last_unstale":"2024-04-25T05:39:06.889525+0000","last_undegraded":"2024-04-25T05:39:06.889525+0000","last_fullsized":"2024-04-25T05:39:06.889525+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":88,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379393+0000","last_change":"2024-04-25T05:38:54.429610+0000","last_active":"2024-04-25T05:39:07.379393+0000","last_peered":"2024-04-25T05:39:07.379393+0000","last_clean":"2024-04-25T05:39:07.379393+0000","last_became_active":"2024-04-25T05:38:13.305781+0000","last_became_peered":"2024-04-25T05:38:13.305781+0000","last_unstale":"2024-04-25T05:39:07.379393+0000","last_undegraded":"2024-04-25T05:39:07.379393+0000","last_fullsized":"2024-04-25T05:39:07.379393+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.390666+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":265,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887818+0000","last_change":"2024-04-25T05:38:13.287078+0000","last_active":"2024-04-25T05:39:06.887818+0000","last_peered":"2024-04-25T05:39:06.887818+0000","last_clean":"2024-04-25T05:39:06.887818+0000","last_became_active":"2024-04-25T05:38:13.286756+0000","last_became_peered":"2024-04-25T05:38:13.286756+0000","last_unstale":"2024-04-25T05:39:06.887818+0000","last_undegraded":"2024-04-25T05:39:06.887818+0000","last_fullsized":"2024-04-25T05:39:06.887818+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":281,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062351+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:39:05.062351+0000","last_peered":"2024-04-25T05:39:05.062351+0000","last_clean":"2024-04-25T05:39:05.062351+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:39:05.062351+0000","last_undegraded":"2024-04-25T05:39:05.062351+0000","last_fullsized":"2024-04-25T05:39:05.062351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":89,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886476+0000","last_change":"2024-04-25T05:38:13.316204+0000","last_active":"2024-04-25T05:39:06.886476+0000","last_peered":"2024-04-25T05:39:06.886476+0000","last_clean":"2024-04-25T05:39:06.886476+0000","last_became_active":"2024-04-25T05:38:13.296338+0000","last_became_peered":"2024-04-25T05:38:13.296338+0000","last_unstale":"2024-04-25T05:39:06.886476+0000","last_undegraded":"2024-04-25T05:39:06.886476+0000","last_fullsized":"2024-04-25T05:39:06.886476+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":269,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889244+0000","last_change":"2024-04-25T05:39:04.115851+0000","last_active":"2024-04-25T05:39:06.889244+0000","last_peered":"2024-04-25T05:39:06.889244+0000","last_clean":"2024-04-25T05:39:06.889244+0000","last_became_active":"2024-04-25T05:38:13.246584+0000","last_became_peered":"2024-04-25T05:38:13.246584+0000","last_unstale":"2024-04-25T05:39:06.889244+0000","last_undegraded":"2024-04-25T05:39:06.889244+0000","last_fullsized":"2024-04-25T05:39:06.889244+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.115756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.115756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.122079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":267,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886377+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:39:06.886377+0000","last_peered":"2024-04-25T05:39:06.886377+0000","last_clean":"2024-04-25T05:39:06.886377+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:39:06.886377+0000","last_undegraded":"2024-04-25T05:39:06.886377+0000","last_fullsized":"2024-04-25T05:39:06.886377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":298,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887355+0000","last_change":"2024-04-25T05:39:00.543672+0000","last_active":"2024-04-25T05:39:06.887355+0000","last_peered":"2024-04-25T05:39:06.887355+0000","last_clean":"2024-04-25T05:39:06.887355+0000","last_became_active":"2024-04-25T05:36:54.235617+0000","last_became_peered":"2024-04-25T05:36:54.235617+0000","last_unstale":"2024-04-25T05:39:06.887355+0000","last_undegraded":"2024-04-25T05:39:06.887355+0000","last_fullsized":"2024-04-25T05:39:06.887355+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:39:00.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.543557+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:10.367791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":315,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887521+0000","last_change":"2024-04-25T05:38:55.603487+0000","last_active":"2024-04-25T05:39:06.887521+0000","last_peered":"2024-04-25T05:39:06.887521+0000","last_clean":"2024-04-25T05:39:06.887521+0000","last_became_active":"2024-04-25T05:36:54.236292+0000","last_became_peered":"2024-04-25T05:36:54.236292+0000","last_unstale":"2024-04-25T05:39:06.887521+0000","last_undegraded":"2024-04-25T05:39:06.887521+0000","last_fullsized":"2024-04-25T05:39:06.887521+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:38:55.603358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.603358+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:00.207469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":78,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062468+0000","last_change":"2024-04-25T05:38:13.246984+0000","last_active":"2024-04-25T05:39:05.062468+0000","last_peered":"2024-04-25T05:39:05.062468+0000","last_clean":"2024-04-25T05:39:05.062468+0000","last_became_active":"2024-04-25T05:38:13.246544+0000","last_became_peered":"2024-04-25T05:38:13.246544+0000","last_unstale":"2024-04-25T05:39:05.062468+0000","last_undegraded":"2024-04-25T05:39:05.062468+0000","last_fullsized":"2024-04-25T05:39:05.062468+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"160'9","reported_seq":356,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380520+0000","last_change":"2024-04-25T05:38:55.388653+0000","last_active":"2024-04-25T05:39:07.380520+0000","last_peered":"2024-04-25T05:39:07.380520+0000","last_clean":"2024-04-25T05:39:07.380520+0000","last_became_active":"2024-04-25T05:38:13.295760+0000","last_became_peered":"2024-04-25T05:38:13.295760+0000","last_unstale":"2024-04-25T05:39:07.380520+0000","last_undegraded":"2024-04-25T05:39:07.380520+0000","last_fullsized":"2024-04-25T05:39:07.380520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'9","last_scrub_stamp":"2024-04-25T05:38:55.388487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.388487+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:57.015303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":281,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061705+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:39:05.061705+0000","last_peered":"2024-04-25T05:39:05.061705+0000","last_clean":"2024-04-25T05:39:05.061705+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:39:05.061705+0000","last_undegraded":"2024-04-25T05:39:05.061705+0000","last_fullsized":"2024-04-25T05:39:05.061705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":310,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888765+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:39:06.888765+0000","last_peered":"2024-04-25T05:39:06.888765+0000","last_clean":"2024-04-25T05:39:06.888765+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:39:06.888765+0000","last_undegraded":"2024-04-25T05:39:06.888765+0000","last_fullsized":"2024-04-25T05:39:06.888765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":262,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888198+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:39:06.888198+0000","last_peered":"2024-04-25T05:39:06.888198+0000","last_clean":"2024-04-25T05:39:06.888198+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:39:06.888198+0000","last_undegraded":"2024-04-25T05:39:06.888198+0000","last_fullsized":"2024-04-25T05:39:06.888198+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":329,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889907+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:39:06.889907+0000","last_peered":"2024-04-25T05:39:06.889907+0000","last_clean":"2024-04-25T05:39:06.889907+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:39:06.889907+0000","last_undegraded":"2024-04-25T05:39:06.889907+0000","last_fullsized":"2024-04-25T05:39:06.889907+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":78,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061604+0000","last_change":"2024-04-25T05:38:13.282321+0000","last_active":"2024-04-25T05:39:05.061604+0000","last_peered":"2024-04-25T05:39:05.061604+0000","last_clean":"2024-04-25T05:39:05.061604+0000","last_became_active":"2024-04-25T05:38:13.282059+0000","last_became_peered":"2024-04-25T05:38:13.282059+0000","last_unstale":"2024-04-25T05:39:05.061604+0000","last_undegraded":"2024-04-25T05:39:05.061604+0000","last_fullsized":"2024-04-25T05:39:05.061604+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":312,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380788+0000","last_change":"2024-04-25T05:39:03.365135+0000","last_active":"2024-04-25T05:39:07.380788+0000","last_peered":"2024-04-25T05:39:07.380788+0000","last_clean":"2024-04-25T05:39:07.380788+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:39:07.380788+0000","last_undegraded":"2024-04-25T05:39:07.380788+0000","last_fullsized":"2024-04-25T05:39:07.380788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:39:03.364988+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.364988+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.589680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":268,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.762316+0000","last_change":"2024-04-25T05:38:13.947132+0000","last_active":"2024-04-25T05:39:04.762316+0000","last_peered":"2024-04-25T05:39:04.762316+0000","last_clean":"2024-04-25T05:39:04.762316+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:39:04.762316+0000","last_undegraded":"2024-04-25T05:39:04.762316+0000","last_fullsized":"2024-04-25T05:39:04.762316+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:13.947044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:13.947044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:19.521993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":259,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889652+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:39:06.889652+0000","last_peered":"2024-04-25T05:39:06.889652+0000","last_clean":"2024-04-25T05:39:06.889652+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:39:06.889652+0000","last_undegraded":"2024-04-25T05:39:06.889652+0000","last_fullsized":"2024-04-25T05:39:06.889652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":293,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061709+0000","last_change":"2024-04-25T05:38:13.241002+0000","last_active":"2024-04-25T05:39:05.061709+0000","last_peered":"2024-04-25T05:39:05.061709+0000","last_clean":"2024-04-25T05:39:05.061709+0000","last_became_active":"2024-04-25T05:38:13.240401+0000","last_became_peered":"2024-04-25T05:38:13.240401+0000","last_unstale":"2024-04-25T05:39:05.061709+0000","last_undegraded":"2024-04-25T05:39:05.061709+0000","last_fullsized":"2024-04-25T05:39:05.061709+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":153,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889974+0000","last_change":"2024-04-25T05:38:13.255668+0000","last_active":"2024-04-25T05:39:06.889974+0000","last_peered":"2024-04-25T05:39:06.889974+0000","last_clean":"2024-04-25T05:39:06.889974+0000","last_became_active":"2024-04-25T05:38:13.245172+0000","last_became_peered":"2024-04-25T05:38:13.245172+0000","last_unstale":"2024-04-25T05:39:06.889974+0000","last_undegraded":"2024-04-25T05:39:06.889974+0000","last_fullsized":"2024-04-25T05:39:06.889974+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":275,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062265+0000","last_change":"2024-04-25T05:37:41.064307+0000","last_active":"2024-04-25T05:39:05.062265+0000","last_peered":"2024-04-25T05:39:05.062265+0000","last_clean":"2024-04-25T05:39:05.062265+0000","last_became_active":"2024-04-25T05:36:54.232503+0000","last_became_peered":"2024-04-25T05:36:54.232503+0000","last_unstale":"2024-04-25T05:39:05.062265+0000","last_undegraded":"2024-04-25T05:39:05.062265+0000","last_fullsized":"2024-04-25T05:39:05.062265+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":267,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380073+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:39:07.380073+0000","last_peered":"2024-04-25T05:39:07.380073+0000","last_clean":"2024-04-25T05:39:07.380073+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:39:07.380073+0000","last_undegraded":"2024-04-25T05:39:07.380073+0000","last_fullsized":"2024-04-25T05:39:07.380073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":91,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379253+0000","last_change":"2024-04-25T05:38:26.418295+0000","last_active":"2024-04-25T05:39:07.379253+0000","last_peered":"2024-04-25T05:39:07.379253+0000","last_clean":"2024-04-25T05:39:07.379253+0000","last_became_active":"2024-04-25T05:38:13.244520+0000","last_became_peered":"2024-04-25T05:38:13.244520+0000","last_unstale":"2024-04-25T05:39:07.379253+0000","last_undegraded":"2024-04-25T05:39:07.379253+0000","last_fullsized":"2024-04-25T05:39:07.379253+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:26.418226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:38:26.418226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:47.053592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":292,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889979+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:39:06.889979+0000","last_peered":"2024-04-25T05:39:06.889979+0000","last_clean":"2024-04-25T05:39:06.889979+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:39:06.889979+0000","last_undegraded":"2024-04-25T05:39:06.889979+0000","last_fullsized":"2024-04-25T05:39:06.889979+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":283,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379811+0000","last_change":"2024-04-25T05:38:13.285501+0000","last_active":"2024-04-25T05:39:07.379811+0000","last_peered":"2024-04-25T05:39:07.379811+0000","last_clean":"2024-04-25T05:39:07.379811+0000","last_became_active":"2024-04-25T05:38:13.284963+0000","last_became_peered":"2024-04-25T05:38:13.284963+0000","last_unstale":"2024-04-25T05:39:07.379811+0000","last_undegraded":"2024-04-25T05:39:07.379811+0000","last_fullsized":"2024-04-25T05:39:07.379811+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":305,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062679+0000","last_change":"2024-04-25T05:38:13.313381+0000","last_active":"2024-04-25T05:39:05.062679+0000","last_peered":"2024-04-25T05:39:05.062679+0000","last_clean":"2024-04-25T05:39:05.062679+0000","last_became_active":"2024-04-25T05:38:13.295320+0000","last_became_peered":"2024-04-25T05:38:13.295320+0000","last_unstale":"2024-04-25T05:39:05.062679+0000","last_undegraded":"2024-04-25T05:39:05.062679+0000","last_fullsized":"2024-04-25T05:39:05.062679+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":346,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379225+0000","last_change":"2024-04-25T05:36:54.267732+0000","last_active":"2024-04-25T05:39:07.379225+0000","last_peered":"2024-04-25T05:39:07.379225+0000","last_clean":"2024-04-25T05:39:07.379225+0000","last_became_active":"2024-04-25T05:36:54.244247+0000","last_became_peered":"2024-04-25T05:36:54.244247+0000","last_unstale":"2024-04-25T05:39:07.379225+0000","last_undegraded":"2024-04-25T05:39:07.379225+0000","last_fullsized":"2024-04-25T05:39:07.379225+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"130'10","reported_seq":354,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061961+0000","last_change":"2024-04-25T05:36:54.243037+0000","last_active":"2024-04-25T05:39:05.061961+0000","last_peered":"2024-04-25T05:39:05.061961+0000","last_clean":"2024-04-25T05:39:05.061961+0000","last_became_active":"2024-04-25T05:36:54.235542+0000","last_became_peered":"2024-04-25T05:36:54.235542+0000","last_unstale":"2024-04-25T05:39:05.061961+0000","last_undegraded":"2024-04-25T05:39:05.061961+0000","last_fullsized":"2024-04-25T05:39:05.061961+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":255,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888021+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:39:06.888021+0000","last_peered":"2024-04-25T05:39:06.888021+0000","last_clean":"2024-04-25T05:39:06.888021+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:39:06.888021+0000","last_undegraded":"2024-04-25T05:39:06.888021+0000","last_fullsized":"2024-04-25T05:39:06.888021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":281,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062668+0000","last_change":"2024-04-25T05:36:48.522033+0000","last_active":"2024-04-25T05:39:05.062668+0000","last_peered":"2024-04-25T05:39:05.062668+0000","last_clean":"2024-04-25T05:39:05.062668+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:39:05.062668+0000","last_undegraded":"2024-04-25T05:39:05.062668+0000","last_fullsized":"2024-04-25T05:39:05.062668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:48.521908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:48.521908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":317,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887105+0000","last_change":"2024-04-25T05:35:29.708012+0000","last_active":"2024-04-25T05:39:06.887105+0000","last_peered":"2024-04-25T05:39:06.887105+0000","last_clean":"2024-04-25T05:39:06.887105+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:39:06.887105+0000","last_undegraded":"2024-04-25T05:39:06.887105+0000","last_fullsized":"2024-04-25T05:39:06.887105+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:35:29.707870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:29.707870+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":278,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.760562+0000","last_change":"2024-04-25T05:38:13.302895+0000","last_active":"2024-04-25T05:39:04.760562+0000","last_peered":"2024-04-25T05:39:04.760562+0000","last_clean":"2024-04-25T05:39:04.760562+0000","last_became_active":"2024-04-25T05:38:13.302285+0000","last_became_peered":"2024-04-25T05:38:13.302285+0000","last_unstale":"2024-04-25T05:39:04.760562+0000","last_undegraded":"2024-04-25T05:39:04.760562+0000","last_fullsized":"2024-04-25T05:39:04.760562+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:47.489244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.489244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":261,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061235+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:39:05.061235+0000","last_peered":"2024-04-25T05:39:05.061235+0000","last_clean":"2024-04-25T05:39:05.061235+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:39:05.061235+0000","last_undegraded":"2024-04-25T05:39:05.061235+0000","last_fullsized":"2024-04-25T05:39:05.061235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":278,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.762363+0000","last_change":"2024-04-25T05:38:13.236944+0000","last_active":"2024-04-25T05:39:04.762363+0000","last_peered":"2024-04-25T05:39:04.762363+0000","last_clean":"2024-04-25T05:39:04.762363+0000","last_became_active":"2024-04-25T05:38:13.236574+0000","last_became_peered":"2024-04-25T05:38:13.236574+0000","last_unstale":"2024-04-25T05:39:04.762363+0000","last_undegraded":"2024-04-25T05:39:04.762363+0000","last_fullsized":"2024-04-25T05:39:04.762363+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:51.573898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.573898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":297,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062652+0000","last_change":"2024-04-25T05:36:54.240931+0000","last_active":"2024-04-25T05:39:05.062652+0000","last_peered":"2024-04-25T05:39:05.062652+0000","last_clean":"2024-04-25T05:39:05.062652+0000","last_became_active":"2024-04-25T05:36:54.232602+0000","last_became_peered":"2024-04-25T05:36:54.232602+0000","last_unstale":"2024-04-25T05:39:05.062652+0000","last_undegraded":"2024-04-25T05:39:05.062652+0000","last_fullsized":"2024-04-25T05:39:05.062652+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":225,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886271+0000","last_change":"2024-04-25T05:36:54.260249+0000","last_active":"2024-04-25T05:39:06.886271+0000","last_peered":"2024-04-25T05:39:06.886271+0000","last_clean":"2024-04-25T05:39:06.886271+0000","last_became_active":"2024-04-25T05:36:54.235633+0000","last_became_peered":"2024-04-25T05:36:54.235633+0000","last_unstale":"2024-04-25T05:39:06.886271+0000","last_undegraded":"2024-04-25T05:39:06.886271+0000","last_fullsized":"2024-04-25T05:39:06.886271+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.050549+0000","last_change":"2024-04-25T05:39:06.050576+0000","last_active":"2024-04-25T05:39:06.050549+0000","last_peered":"2024-04-25T05:39:06.050549+0000","last_clean":"2024-04-25T05:39:06.050549+0000","last_became_active":"2024-04-25T05:38:13.240067+0000","last_became_peered":"2024-04-25T05:38:13.240067+0000","last_unstale":"2024-04-25T05:39:06.050549+0000","last_undegraded":"2024-04-25T05:39:06.050549+0000","last_fullsized":"2024-04-25T05:39:06.050549+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.492020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":264,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.761760+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:39:04.761760+0000","last_peered":"2024-04-25T05:39:04.761760+0000","last_clean":"2024-04-25T05:39:04.761760+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:39:04.761760+0000","last_undegraded":"2024-04-25T05:39:04.761760+0000","last_fullsized":"2024-04-25T05:39:04.761760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":289,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380380+0000","last_change":"2024-04-25T05:37:30.237718+0000","last_active":"2024-04-25T05:39:07.380380+0000","last_peered":"2024-04-25T05:39:07.380380+0000","last_clean":"2024-04-25T05:39:07.380380+0000","last_became_active":"2024-04-25T05:36:54.234986+0000","last_became_peered":"2024-04-25T05:36:54.234986+0000","last_unstale":"2024-04-25T05:39:07.380380+0000","last_undegraded":"2024-04-25T05:39:07.380380+0000","last_fullsized":"2024-04-25T05:39:07.380380+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":139,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888111+0000","last_change":"2024-04-25T05:38:13.317524+0000","last_active":"2024-04-25T05:39:06.888111+0000","last_peered":"2024-04-25T05:39:06.888111+0000","last_clean":"2024-04-25T05:39:06.888111+0000","last_became_active":"2024-04-25T05:38:13.294321+0000","last_became_peered":"2024-04-25T05:38:13.294321+0000","last_unstale":"2024-04-25T05:39:06.888111+0000","last_undegraded":"2024-04-25T05:39:06.888111+0000","last_fullsized":"2024-04-25T05:39:06.888111+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379027+0000","last_change":"2024-04-25T05:38:58.389048+0000","last_active":"2024-04-25T05:39:07.379027+0000","last_peered":"2024-04-25T05:39:07.379027+0000","last_clean":"2024-04-25T05:39:07.379027+0000","last_became_active":"2024-04-25T05:38:34.759567+0000","last_became_peered":"2024-04-25T05:38:34.759567+0000","last_unstale":"2024-04-25T05:39:07.379027+0000","last_undegraded":"2024-04-25T05:39:07.379027+0000","last_fullsized":"2024-04-25T05:39:07.379027+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.388898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:06.835407+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":263,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.890006+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:39:06.890006+0000","last_peered":"2024-04-25T05:39:06.890006+0000","last_clean":"2024-04-25T05:39:06.890006+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:39:06.890006+0000","last_undegraded":"2024-04-25T05:39:06.890006+0000","last_fullsized":"2024-04-25T05:39:06.890006+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":305,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379588+0000","last_change":"2024-04-25T05:38:13.302140+0000","last_active":"2024-04-25T05:39:07.379588+0000","last_peered":"2024-04-25T05:39:07.379588+0000","last_clean":"2024-04-25T05:39:07.379588+0000","last_became_active":"2024-04-25T05:38:13.284779+0000","last_became_peered":"2024-04-25T05:38:13.284779+0000","last_unstale":"2024-04-25T05:39:07.379588+0000","last_undegraded":"2024-04-25T05:39:07.379588+0000","last_fullsized":"2024-04-25T05:39:07.379588+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:36:21.033577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.033577+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":286,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887162+0000","last_change":"2024-04-25T05:38:13.277150+0000","last_active":"2024-04-25T05:39:06.887162+0000","last_peered":"2024-04-25T05:39:06.887162+0000","last_clean":"2024-04-25T05:39:06.887162+0000","last_became_active":"2024-04-25T05:38:13.276807+0000","last_became_peered":"2024-04-25T05:38:13.276807+0000","last_unstale":"2024-04-25T05:39:06.887162+0000","last_undegraded":"2024-04-25T05:39:06.887162+0000","last_fullsized":"2024-04-25T05:39:06.887162+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:37:43.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":404,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886837+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:39:06.886837+0000","last_peered":"2024-04-25T05:39:06.886837+0000","last_clean":"2024-04-25T05:39:06.886837+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:39:06.886837+0000","last_undegraded":"2024-04-25T05:39:06.886837+0000","last_fullsized":"2024-04-25T05:39:06.886837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":80,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888405+0000","last_change":"2024-04-25T05:38:13.286961+0000","last_active":"2024-04-25T05:39:06.888405+0000","last_peered":"2024-04-25T05:39:06.888405+0000","last_clean":"2024-04-25T05:39:06.888405+0000","last_became_active":"2024-04-25T05:38:13.286575+0000","last_became_peered":"2024-04-25T05:38:13.286575+0000","last_unstale":"2024-04-25T05:39:06.888405+0000","last_undegraded":"2024-04-25T05:39:06.888405+0000","last_fullsized":"2024-04-25T05:39:06.888405+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":579,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061833+0000","last_change":"2024-04-25T05:39:00.133654+0000","last_active":"2024-04-25T05:39:05.061833+0000","last_peered":"2024-04-25T05:39:05.061833+0000","last_clean":"2024-04-25T05:39:05.061833+0000","last_became_active":"2024-04-25T05:38:34.758423+0000","last_became_peered":"2024-04-25T05:38:34.758423+0000","last_unstale":"2024-04-25T05:39:05.061833+0000","last_undegraded":"2024-04-25T05:39:05.061833+0000","last_fullsized":"2024-04-25T05:39:05.061833+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:00.133580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.133580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:12.718787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":275,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062792+0000","last_change":"2024-04-25T05:36:54.626908+0000","last_active":"2024-04-25T05:39:05.062792+0000","last_peered":"2024-04-25T05:39:05.062792+0000","last_clean":"2024-04-25T05:39:05.062792+0000","last_became_active":"2024-04-25T05:36:54.225394+0000","last_became_peered":"2024-04-25T05:36:54.225394+0000","last_unstale":"2024-04-25T05:39:05.062792+0000","last_undegraded":"2024-04-25T05:39:05.062792+0000","last_fullsized":"2024-04-25T05:39:05.062792+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":295,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886553+0000","last_change":"2024-04-25T05:38:13.307221+0000","last_active":"2024-04-25T05:39:06.886553+0000","last_peered":"2024-04-25T05:39:06.886553+0000","last_clean":"2024-04-25T05:39:06.886553+0000","last_became_active":"2024-04-25T05:38:13.306297+0000","last_became_peered":"2024-04-25T05:38:13.306297+0000","last_unstale":"2024-04-25T05:39:06.886553+0000","last_undegraded":"2024-04-25T05:39:06.886553+0000","last_fullsized":"2024-04-25T05:39:06.886553+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:52.054641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:52.054641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:17.305985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":294,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.760668+0000","last_change":"2024-04-25T05:38:13.284622+0000","last_active":"2024-04-25T05:39:04.760668+0000","last_peered":"2024-04-25T05:39:04.760668+0000","last_clean":"2024-04-25T05:39:04.760668+0000","last_became_active":"2024-04-25T05:38:13.284281+0000","last_became_peered":"2024-04-25T05:38:13.284281+0000","last_unstale":"2024-04-25T05:39:04.760668+0000","last_undegraded":"2024-04-25T05:39:04.760668+0000","last_fullsized":"2024-04-25T05:39:04.760668+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":336,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886688+0000","last_change":"2024-04-25T05:38:13.314279+0000","last_active":"2024-04-25T05:39:06.886688+0000","last_peered":"2024-04-25T05:39:06.886688+0000","last_clean":"2024-04-25T05:39:06.886688+0000","last_became_active":"2024-04-25T05:38:13.295547+0000","last_became_peered":"2024-04-25T05:38:13.295547+0000","last_unstale":"2024-04-25T05:39:06.886688+0000","last_undegraded":"2024-04-25T05:39:06.886688+0000","last_fullsized":"2024-04-25T05:39:06.886688+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":272,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.761747+0000","last_change":"2024-04-25T05:38:59.769234+0000","last_active":"2024-04-25T05:39:04.761747+0000","last_peered":"2024-04-25T05:39:04.761747+0000","last_clean":"2024-04-25T05:39:04.761747+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:39:04.761747+0000","last_undegraded":"2024-04-25T05:39:04.761747+0000","last_fullsized":"2024-04-25T05:39:04.761747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.769120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.769120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.633366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":263,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380611+0000","last_change":"2024-04-25T05:36:26.958315+0000","last_active":"2024-04-25T05:39:07.380611+0000","last_peered":"2024-04-25T05:39:07.380611+0000","last_clean":"2024-04-25T05:39:07.380611+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:39:07.380611+0000","last_undegraded":"2024-04-25T05:39:07.380611+0000","last_fullsized":"2024-04-25T05:39:07.380611+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.958197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.958197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":571,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061395+0000","last_change":"2024-04-25T05:38:59.104672+0000","last_active":"2024-04-25T05:39:05.061395+0000","last_peered":"2024-04-25T05:39:05.061395+0000","last_clean":"2024-04-25T05:39:05.061395+0000","last_became_active":"2024-04-25T05:38:29.240034+0000","last_became_peered":"2024-04-25T05:38:29.240034+0000","last_unstale":"2024-04-25T05:39:05.061395+0000","last_undegraded":"2024-04-25T05:39:05.061395+0000","last_fullsized":"2024-04-25T05:39:05.061395+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.104548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.104548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.075012+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888510+0000","last_change":"2024-04-25T05:36:54.233589+0000","last_active":"2024-04-25T05:39:06.888510+0000","last_peered":"2024-04-25T05:39:06.888510+0000","last_clean":"2024-04-25T05:39:06.888510+0000","last_became_active":"2024-04-25T05:36:54.232305+0000","last_became_peered":"2024-04-25T05:36:54.232305+0000","last_unstale":"2024-04-25T05:39:06.888510+0000","last_undegraded":"2024-04-25T05:39:06.888510+0000","last_fullsized":"2024-04-25T05:39:06.888510+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"160'18","reported_seq":262,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887788+0000","last_change":"2024-04-25T05:39:04.573814+0000","last_active":"2024-04-25T05:39:06.887788+0000","last_peered":"2024-04-25T05:39:06.887788+0000","last_clean":"2024-04-25T05:39:06.887788+0000","last_became_active":"2024-04-25T05:38:13.240414+0000","last_became_peered":"2024-04-25T05:38:13.240414+0000","last_unstale":"2024-04-25T05:39:06.887788+0000","last_undegraded":"2024-04-25T05:39:06.887788+0000","last_fullsized":"2024-04-25T05:39:06.887788+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'18","last_scrub_stamp":"2024-04-25T05:39:04.573693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.573693+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.209448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":273,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062402+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:39:05.062402+0000","last_peered":"2024-04-25T05:39:05.062402+0000","last_clean":"2024-04-25T05:39:05.062402+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:39:05.062402+0000","last_undegraded":"2024-04-25T05:39:05.062402+0000","last_fullsized":"2024-04-25T05:39:05.062402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":456,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887400+0000","last_change":"2024-04-25T05:38:58.010459+0000","last_active":"2024-04-25T05:39:06.887400+0000","last_peered":"2024-04-25T05:39:06.887400+0000","last_clean":"2024-04-25T05:39:06.887400+0000","last_became_active":"2024-04-25T05:38:34.751191+0000","last_became_peered":"2024-04-25T05:38:34.751191+0000","last_unstale":"2024-04-25T05:39:06.887400+0000","last_undegraded":"2024-04-25T05:39:06.887400+0000","last_fullsized":"2024-04-25T05:39:06.887400+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.010323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.010323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:11.636120+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":271,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886540+0000","last_change":"2024-04-25T05:39:00.922795+0000","last_active":"2024-04-25T05:39:06.886540+0000","last_peered":"2024-04-25T05:39:06.886540+0000","last_clean":"2024-04-25T05:39:06.886540+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:39:06.886540+0000","last_undegraded":"2024-04-25T05:39:06.886540+0000","last_fullsized":"2024-04-25T05:39:06.886540+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.922679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.922679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:18.355708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":80,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886800+0000","last_change":"2024-04-25T05:38:13.239349+0000","last_active":"2024-04-25T05:39:06.886800+0000","last_peered":"2024-04-25T05:39:06.886800+0000","last_clean":"2024-04-25T05:39:06.886800+0000","last_became_active":"2024-04-25T05:38:13.238969+0000","last_became_peered":"2024-04-25T05:38:13.238969+0000","last_unstale":"2024-04-25T05:39:06.886800+0000","last_undegraded":"2024-04-25T05:39:06.886800+0000","last_fullsized":"2024-04-25T05:39:06.886800+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":605,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.388508+0000","last_change":"2024-04-25T05:38:13.325017+0000","last_active":"2024-04-25T05:39:06.388508+0000","last_peered":"2024-04-25T05:39:06.388508+0000","last_clean":"2024-04-25T05:39:06.388508+0000","last_became_active":"2024-04-25T05:38:13.305327+0000","last_became_peered":"2024-04-25T05:38:13.305327+0000","last_unstale":"2024-04-25T05:39:06.388508+0000","last_undegraded":"2024-04-25T05:39:06.388508+0000","last_fullsized":"2024-04-25T05:39:06.388508+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":299,"num_read_kb":294,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":109,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380040+0000","last_change":"2024-04-25T05:37:56.295161+0000","last_active":"2024-04-25T05:39:07.380040+0000","last_peered":"2024-04-25T05:39:07.380040+0000","last_clean":"2024-04-25T05:39:07.380040+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:39:07.380040+0000","last_undegraded":"2024-04-25T05:39:07.380040+0000","last_fullsized":"2024-04-25T05:39:07.380040+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380990+0000","last_change":"2024-04-25T05:38:13.296713+0000","last_active":"2024-04-25T05:39:07.380990+0000","last_peered":"2024-04-25T05:39:07.380990+0000","last_clean":"2024-04-25T05:39:07.380990+0000","last_became_active":"2024-04-25T05:38:13.296013+0000","last_became_peered":"2024-04-25T05:38:13.296013+0000","last_unstale":"2024-04-25T05:39:07.380990+0000","last_undegraded":"2024-04-25T05:39:07.380990+0000","last_fullsized":"2024-04-25T05:39:07.380990+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888800+0000","last_change":"2024-04-25T05:38:57.081345+0000","last_active":"2024-04-25T05:39:06.888800+0000","last_peered":"2024-04-25T05:39:06.888800+0000","last_clean":"2024-04-25T05:39:06.888800+0000","last_became_active":"2024-04-25T05:38:34.758128+0000","last_became_peered":"2024-04-25T05:38:34.758128+0000","last_unstale":"2024-04-25T05:39:06.888800+0000","last_undegraded":"2024-04-25T05:39:06.888800+0000","last_fullsized":"2024-04-25T05:39:06.888800+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.081204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.081204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:16.313151+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":269,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887834+0000","last_change":"2024-04-25T05:36:54.234953+0000","last_active":"2024-04-25T05:39:06.887834+0000","last_peered":"2024-04-25T05:39:06.887834+0000","last_clean":"2024-04-25T05:39:06.887834+0000","last_became_active":"2024-04-25T05:36:54.234602+0000","last_became_peered":"2024-04-25T05:36:54.234602+0000","last_unstale":"2024-04-25T05:39:06.887834+0000","last_undegraded":"2024-04-25T05:39:06.887834+0000","last_fullsized":"2024-04-25T05:39:06.887834+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":283,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886753+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:39:06.886753+0000","last_peered":"2024-04-25T05:39:06.886753+0000","last_clean":"2024-04-25T05:39:06.886753+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:39:06.886753+0000","last_undegraded":"2024-04-25T05:39:06.886753+0000","last_fullsized":"2024-04-25T05:39:06.886753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":175,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380012+0000","last_change":"2024-04-25T05:38:22.405354+0000","last_active":"2024-04-25T05:39:07.380012+0000","last_peered":"2024-04-25T05:39:07.380012+0000","last_clean":"2024-04-25T05:39:07.380012+0000","last_became_active":"2024-04-25T05:38:13.296480+0000","last_became_peered":"2024-04-25T05:38:13.296480+0000","last_unstale":"2024-04-25T05:39:07.380012+0000","last_undegraded":"2024-04-25T05:39:07.380012+0000","last_fullsized":"2024-04-25T05:39:07.380012+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:38:22.404901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:22.404901+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:24.353079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":83,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379656+0000","last_change":"2024-04-25T05:38:13.239510+0000","last_active":"2024-04-25T05:39:07.379656+0000","last_peered":"2024-04-25T05:39:07.379656+0000","last_clean":"2024-04-25T05:39:07.379656+0000","last_became_active":"2024-04-25T05:38:13.237844+0000","last_became_peered":"2024-04-25T05:38:13.237844+0000","last_unstale":"2024-04-25T05:39:07.379656+0000","last_undegraded":"2024-04-25T05:39:07.379656+0000","last_fullsized":"2024-04-25T05:39:07.379656+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888573+0000","last_change":"2024-04-25T05:38:59.069446+0000","last_active":"2024-04-25T05:39:06.888573+0000","last_peered":"2024-04-25T05:39:06.888573+0000","last_clean":"2024-04-25T05:39:06.888573+0000","last_became_active":"2024-04-25T05:38:34.761903+0000","last_became_peered":"2024-04-25T05:38:34.761903+0000","last_unstale":"2024-04-25T05:39:06.888573+0000","last_undegraded":"2024-04-25T05:39:06.888573+0000","last_fullsized":"2024-04-25T05:39:06.888573+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.069332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.069332+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:19.625381+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":80,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888073+0000","last_change":"2024-04-25T05:38:13.244062+0000","last_active":"2024-04-25T05:39:06.888073+0000","last_peered":"2024-04-25T05:39:06.888073+0000","last_clean":"2024-04-25T05:39:06.888073+0000","last_became_active":"2024-04-25T05:38:13.243650+0000","last_became_peered":"2024-04-25T05:38:13.243650+0000","last_unstale":"2024-04-25T05:39:06.888073+0000","last_undegraded":"2024-04-25T05:39:06.888073+0000","last_fullsized":"2024-04-25T05:39:06.888073+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":275,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380574+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:39:07.380574+0000","last_peered":"2024-04-25T05:39:07.380574+0000","last_clean":"2024-04-25T05:39:07.380574+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:39:07.380574+0000","last_undegraded":"2024-04-25T05:39:07.380574+0000","last_fullsized":"2024-04-25T05:39:07.380574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'16","reported_seq":226,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.760881+0000","last_change":"2024-04-25T05:35:36.662905+0000","last_active":"2024-04-25T05:39:04.760881+0000","last_peered":"2024-04-25T05:39:04.760881+0000","last_clean":"2024-04-25T05:39:04.760881+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:39:04.760881+0000","last_undegraded":"2024-04-25T05:39:04.760881+0000","last_fullsized":"2024-04-25T05:39:04.760881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-04-25T05:35:36.662752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.662752+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"160'10","reported_seq":128,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380267+0000","last_change":"2024-04-25T05:38:24.406772+0000","last_active":"2024-04-25T05:39:07.380267+0000","last_peered":"2024-04-25T05:39:07.380267+0000","last_clean":"2024-04-25T05:39:07.380267+0000","last_became_active":"2024-04-25T05:38:13.295784+0000","last_became_peered":"2024-04-25T05:38:13.295784+0000","last_unstale":"2024-04-25T05:39:07.380267+0000","last_undegraded":"2024-04-25T05:39:07.380267+0000","last_fullsized":"2024-04-25T05:39:07.380267+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10","last_scrub_stamp":"2024-04-25T05:38:24.406643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:24.406643+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:35.459225+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888246+0000","last_change":"2024-04-25T05:38:58.078848+0000","last_active":"2024-04-25T05:39:06.888246+0000","last_peered":"2024-04-25T05:39:06.888246+0000","last_clean":"2024-04-25T05:39:06.888246+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:39:06.888246+0000","last_undegraded":"2024-04-25T05:39:06.888246+0000","last_fullsized":"2024-04-25T05:39:06.888246+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.078483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.078483+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.982642+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":275,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.062142+0000","last_change":"2024-04-25T05:36:59.627764+0000","last_active":"2024-04-25T05:39:05.062142+0000","last_peered":"2024-04-25T05:39:05.062142+0000","last_clean":"2024-04-25T05:39:05.062142+0000","last_became_active":"2024-04-25T05:36:54.234860+0000","last_became_peered":"2024-04-25T05:36:54.234860+0000","last_unstale":"2024-04-25T05:39:05.062142+0000","last_undegraded":"2024-04-25T05:39:05.062142+0000","last_fullsized":"2024-04-25T05:39:05.062142+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":161,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887328+0000","last_change":"2024-04-25T05:38:13.301084+0000","last_active":"2024-04-25T05:39:06.887328+0000","last_peered":"2024-04-25T05:39:06.887328+0000","last_clean":"2024-04-25T05:39:06.887328+0000","last_became_active":"2024-04-25T05:38:13.289662+0000","last_became_peered":"2024-04-25T05:38:13.289662+0000","last_unstale":"2024-04-25T05:39:06.887328+0000","last_undegraded":"2024-04-25T05:39:06.887328+0000","last_fullsized":"2024-04-25T05:39:06.887328+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":352,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886945+0000","last_change":"2024-04-25T05:38:13.255913+0000","last_active":"2024-04-25T05:39:06.886945+0000","last_peered":"2024-04-25T05:39:06.886945+0000","last_clean":"2024-04-25T05:39:06.886945+0000","last_became_active":"2024-04-25T05:38:13.244214+0000","last_became_peered":"2024-04-25T05:38:13.244214+0000","last_unstale":"2024-04-25T05:39:06.886945+0000","last_undegraded":"2024-04-25T05:39:06.886945+0000","last_fullsized":"2024-04-25T05:39:06.886945+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":337,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888645+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:39:06.888645+0000","last_peered":"2024-04-25T05:39:06.888645+0000","last_clean":"2024-04-25T05:39:06.888645+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:39:06.888645+0000","last_undegraded":"2024-04-25T05:39:06.888645+0000","last_fullsized":"2024-04-25T05:39:06.888645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887671+0000","last_change":"2024-04-25T05:38:57.548615+0000","last_active":"2024-04-25T05:39:06.887671+0000","last_peered":"2024-04-25T05:39:06.887671+0000","last_clean":"2024-04-25T05:39:06.887671+0000","last_became_active":"2024-04-25T05:38:34.758569+0000","last_became_peered":"2024-04-25T05:38:34.758569+0000","last_unstale":"2024-04-25T05:39:06.887671+0000","last_undegraded":"2024-04-25T05:39:06.887671+0000","last_fullsized":"2024-04-25T05:39:06.887671+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.548529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.548529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.774155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":262,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886768+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:39:06.886768+0000","last_peered":"2024-04-25T05:39:06.886768+0000","last_clean":"2024-04-25T05:39:06.886768+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:39:06.886768+0000","last_undegraded":"2024-04-25T05:39:06.886768+0000","last_fullsized":"2024-04-25T05:39:06.886768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.889393+0000","last_change":"2024-04-25T05:38:13.247469+0000","last_active":"2024-04-25T05:39:06.889393+0000","last_peered":"2024-04-25T05:39:06.889393+0000","last_clean":"2024-04-25T05:39:06.889393+0000","last_became_active":"2024-04-25T05:38:13.246750+0000","last_became_peered":"2024-04-25T05:38:13.246750+0000","last_unstale":"2024-04-25T05:39:06.889393+0000","last_undegraded":"2024-04-25T05:39:06.889393+0000","last_fullsized":"2024-04-25T05:39:06.889393+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":355,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888494+0000","last_change":"2024-04-25T05:38:13.326846+0000","last_active":"2024-04-25T05:39:06.888494+0000","last_peered":"2024-04-25T05:39:06.888494+0000","last_clean":"2024-04-25T05:39:06.888494+0000","last_became_active":"2024-04-25T05:38:13.305935+0000","last_became_peered":"2024-04-25T05:38:13.305935+0000","last_unstale":"2024-04-25T05:39:06.888494+0000","last_undegraded":"2024-04-25T05:39:06.888494+0000","last_fullsized":"2024-04-25T05:39:06.888494+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":266,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886666+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:39:06.886666+0000","last_peered":"2024-04-25T05:39:06.886666+0000","last_clean":"2024-04-25T05:39:06.886666+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:39:06.886666+0000","last_undegraded":"2024-04-25T05:39:06.886666+0000","last_fullsized":"2024-04-25T05:39:06.886666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.379405+0000","last_change":"2024-04-25T05:38:57.405872+0000","last_active":"2024-04-25T05:39:07.379405+0000","last_peered":"2024-04-25T05:39:07.379405+0000","last_clean":"2024-04-25T05:39:07.379405+0000","last_became_active":"2024-04-25T05:38:34.758522+0000","last_became_peered":"2024-04-25T05:38:34.758522+0000","last_unstale":"2024-04-25T05:39:07.379405+0000","last_undegraded":"2024-04-25T05:39:07.379405+0000","last_fullsized":"2024-04-25T05:39:07.379405+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:16.857424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":277,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380402+0000","last_change":"2024-04-25T05:38:13.284810+0000","last_active":"2024-04-25T05:39:07.380402+0000","last_peered":"2024-04-25T05:39:07.380402+0000","last_clean":"2024-04-25T05:39:07.380402+0000","last_became_active":"2024-04-25T05:38:13.284451+0000","last_became_peered":"2024-04-25T05:38:13.284451+0000","last_unstale":"2024-04-25T05:39:07.380402+0000","last_undegraded":"2024-04-25T05:39:07.380402+0000","last_fullsized":"2024-04-25T05:39:07.380402+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"160'9","reported_seq":347,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888175+0000","last_change":"2024-04-25T05:38:13.244991+0000","last_active":"2024-04-25T05:39:06.888175+0000","last_peered":"2024-04-25T05:39:06.888175+0000","last_clean":"2024-04-25T05:39:06.888175+0000","last_became_active":"2024-04-25T05:38:13.238743+0000","last_became_peered":"2024-04-25T05:38:13.238743+0000","last_unstale":"2024-04-25T05:39:06.888175+0000","last_undegraded":"2024-04-25T05:39:06.888175+0000","last_fullsized":"2024-04-25T05:39:06.888175+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:37:48.604653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:48.604653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":275,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888457+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:39:06.888457+0000","last_peered":"2024-04-25T05:39:06.888457+0000","last_clean":"2024-04-25T05:39:06.888457+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:39:06.888457+0000","last_undegraded":"2024-04-25T05:39:06.888457+0000","last_fullsized":"2024-04-25T05:39:06.888457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":296,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886786+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:39:06.886786+0000","last_peered":"2024-04-25T05:39:06.886786+0000","last_clean":"2024-04-25T05:39:06.886786+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:39:06.886786+0000","last_undegraded":"2024-04-25T05:39:06.886786+0000","last_fullsized":"2024-04-25T05:39:06.886786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":261,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888661+0000","last_change":"2024-04-25T05:36:54.232549+0000","last_active":"2024-04-25T05:39:06.888661+0000","last_peered":"2024-04-25T05:39:06.888661+0000","last_clean":"2024-04-25T05:39:06.888661+0000","last_became_active":"2024-04-25T05:36:54.231886+0000","last_became_peered":"2024-04-25T05:36:54.231886+0000","last_unstale":"2024-04-25T05:39:06.888661+0000","last_undegraded":"2024-04-25T05:39:06.888661+0000","last_fullsized":"2024-04-25T05:39:06.888661+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.380900+0000","last_change":"2024-04-25T05:37:08.197220+0000","last_active":"2024-04-25T05:39:07.380900+0000","last_peered":"2024-04-25T05:39:07.380900+0000","last_clean":"2024-04-25T05:39:07.380900+0000","last_became_active":"2024-04-25T05:36:54.244909+0000","last_became_peered":"2024-04-25T05:36:54.244909+0000","last_unstale":"2024-04-25T05:39:07.380900+0000","last_undegraded":"2024-04-25T05:39:07.380900+0000","last_fullsized":"2024-04-25T05:39:07.380900+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":274,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886731+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:39:06.886731+0000","last_peered":"2024-04-25T05:39:06.886731+0000","last_clean":"2024-04-25T05:39:06.886731+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:39:06.886731+0000","last_undegraded":"2024-04-25T05:39:06.886731+0000","last_fullsized":"2024-04-25T05:39:06.886731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":321,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887879+0000","last_change":"2024-04-25T05:36:54.269613+0000","last_active":"2024-04-25T05:39:06.887879+0000","last_peered":"2024-04-25T05:39:06.887879+0000","last_clean":"2024-04-25T05:39:06.887879+0000","last_became_active":"2024-04-25T05:36:54.244739+0000","last_became_peered":"2024-04-25T05:36:54.244739+0000","last_unstale":"2024-04-25T05:39:06.887879+0000","last_undegraded":"2024-04-25T05:39:06.887879+0000","last_fullsized":"2024-04-25T05:39:06.887879+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":450,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:07.378916+0000","last_change":"2024-04-25T05:38:59.374948+0000","last_active":"2024-04-25T05:39:07.378916+0000","last_peered":"2024-04-25T05:39:07.378916+0000","last_clean":"2024-04-25T05:39:07.378916+0000","last_became_active":"2024-04-25T05:38:34.759755+0000","last_became_peered":"2024-04-25T05:38:34.759755+0000","last_unstale":"2024-04-25T05:39:07.378916+0000","last_undegraded":"2024-04-25T05:39:07.378916+0000","last_fullsized":"2024-04-25T05:39:07.378916+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:00.299246+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":268,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.761538+0000","last_change":"2024-04-25T05:39:00.794514+0000","last_active":"2024-04-25T05:39:04.761538+0000","last_peered":"2024-04-25T05:39:04.761538+0000","last_clean":"2024-04-25T05:39:04.761538+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:39:04.761538+0000","last_undegraded":"2024-04-25T05:39:04.761538+0000","last_fullsized":"2024-04-25T05:39:04.761538+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.794388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.794388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.097831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886993+0000","last_change":"2024-04-25T05:38:13.289683+0000","last_active":"2024-04-25T05:39:06.886993+0000","last_peered":"2024-04-25T05:39:06.886993+0000","last_clean":"2024-04-25T05:39:06.886993+0000","last_became_active":"2024-04-25T05:38:13.289313+0000","last_became_peered":"2024-04-25T05:38:13.289313+0000","last_unstale":"2024-04-25T05:39:06.886993+0000","last_undegraded":"2024-04-25T05:39:06.886993+0000","last_fullsized":"2024-04-25T05:39:06.886993+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":142,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.886959+0000","last_change":"2024-04-25T05:36:54.226814+0000","last_active":"2024-04-25T05:39:06.886959+0000","last_peered":"2024-04-25T05:39:06.886959+0000","last_clean":"2024-04-25T05:39:06.886959+0000","last_became_active":"2024-04-25T05:36:54.226138+0000","last_became_peered":"2024-04-25T05:36:54.226138+0000","last_unstale":"2024-04-25T05:39:06.886959+0000","last_undegraded":"2024-04-25T05:39:06.886959+0000","last_fullsized":"2024-04-25T05:39:06.886959+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":298,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887905+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:39:06.887905+0000","last_peered":"2024-04-25T05:39:06.887905+0000","last_clean":"2024-04-25T05:39:06.887905+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:39:06.887905+0000","last_undegraded":"2024-04-25T05:39:06.887905+0000","last_fullsized":"2024-04-25T05:39:06.887905+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":579,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:05.061294+0000","last_change":"2024-04-25T05:39:01.087263+0000","last_active":"2024-04-25T05:39:05.061294+0000","last_peered":"2024-04-25T05:39:05.061294+0000","last_clean":"2024-04-25T05:39:05.061294+0000","last_became_active":"2024-04-25T05:38:34.761642+0000","last_became_peered":"2024-04-25T05:38:34.761642+0000","last_unstale":"2024-04-25T05:39:05.061294+0000","last_undegraded":"2024-04-25T05:39:05.061294+0000","last_fullsized":"2024-04-25T05:39:05.061294+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:01.087145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:01.087145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.030738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":262,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.888245+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:39:06.888245+0000","last_peered":"2024-04-25T05:39:06.888245+0000","last_clean":"2024-04-25T05:39:06.888245+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:39:06.888245+0000","last_undegraded":"2024-04-25T05:39:06.888245+0000","last_fullsized":"2024-04-25T05:39:06.888245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":338,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-04-25T05:39:04.761515+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:39:04.761515+0000","last_peered":"2024-04-25T05:39:04.761515+0000","last_clean":"2024-04-25T05:39:04.761515+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:39:04.761515+0000","last_undegraded":"2024-04-25T05:39:04.761515+0000","last_fullsized":"2024-04-25T05:39:04.761515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887369+0000","last_change":"2024-04-25T05:38:13.286297+0000","last_active":"2024-04-25T05:39:06.887369+0000","last_peered":"2024-04-25T05:39:06.887369+0000","last_clean":"2024-04-25T05:39:06.887369+0000","last_became_active":"2024-04-25T05:38:13.285812+0000","last_became_peered":"2024-04-25T05:38:13.285812+0000","last_unstale":"2024-04-25T05:39:06.887369+0000","last_undegraded":"2024-04-25T05:39:06.887369+0000","last_fullsized":"2024-04-25T05:39:06.887369+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":303,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-04-25T05:39:06.887621+0000","last_change":"2024-04-25T05:36:54.254870+0000","last_active":"2024-04-25T05:39:06.887621+0000","last_peered":"2024-04-25T05:39:06.887621+0000","last_clean":"2024-04-25T05:39:06.887621+0000","last_became_active":"2024-04-25T05:36:54.234440+0000","last_became_peered":"2024-04-25T05:36:54.234440+0000","last_unstale":"2024-04-25T05:39:06.887621+0000","last_undegraded":"2024-04-25T05:39:06.887621+0000","last_fullsized":"2024-04-25T05:39:06.887621+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":299,"num_read_kb":294,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":13834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":623,"ondisk_log_size":623,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070531,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163240,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93581912,"statfs":{"total":95995035648,"available":95827877888,"internally_reserved":0,"allocated":1605632,"data_stored":1005787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217120,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528032,"statfs":{"total":95995035648,"available":95772704768,"internally_reserved":0,"allocated":1331200,"data_stored":684415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203408,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93541744,"statfs":{"total":95995035648,"available":95786745856,"internally_reserved":0,"allocated":1511424,"data_stored":939376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":231568,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93513584,"statfs":{"total":95995035648,"available":95757910016,"internally_reserved":0,"allocated":1904640,"data_stored":1274795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986256,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130996,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614156,"statfs":{"total":95995035648,"available":95860895744,"internally_reserved":0,"allocated":1282048,"data_stored":683819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691916,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123228,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621924,"statfs":{"total":95995035648,"available":95868850176,"internally_reserved":0,"allocated":1851392,"data_stored":1266917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:39:12.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:39:12.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:12.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:12.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/988724067' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:39:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: osdmap e196: 8 total, 8 up, 6 in 2024-04-25T05:39:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:39:12.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:11 smithi060 ceph-mon[30898]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:12.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:12.342 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:12.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:12.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:12.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:12.592 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:12.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:12.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:12.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:12.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:12.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:12.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:12.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:13.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:13.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:13.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:12 smithi060 ceph-mon[30898]: pgmap v261: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:13.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:12 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:13.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:13.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:13.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:12 smithi043 ceph-mon[27010]: pgmap v261: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:13.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:12 smithi043 ceph-mon[37492]: pgmap v261: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:13.399 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-25T05:39:13.400 INFO:tasks.radosbench:Beginning radosbench... 2024-04-25T05:39:13.400 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-25T05:39:13.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-04-25T05:39:13.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:13.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:13.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:13.595 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:13.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:13.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:13.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:13.846 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:13.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:13.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:13.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:14.062 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:14.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:14.081 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:14.100 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:14.101 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:14.121 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:14.142 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:14.163 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:14.185 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:14.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:13 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:14.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:13 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:14.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:14.205 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:13 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:14.290 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:39:14.290 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:39:14.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:14.353 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:14.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:14.477 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:14.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:14.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:14.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:14.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:14.846 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:14.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:14.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:15.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:15.091 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:15.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:15.213 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:15.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:14 smithi060 ceph-mon[30898]: pgmap v262: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:39:15.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2331943530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:14 smithi060 ceph-mon[30898]: osdmap e197: 8 total, 8 up, 6 in 2024-04-25T05:39:15.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[27010]: pgmap v262: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:39:15.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2331943530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:15.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[27010]: osdmap e197: 8 total, 8 up, 6 in 2024-04-25T05:39:15.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[37492]: pgmap v262: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:39:15.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2331943530' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:15.286 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:14 smithi043 ceph-mon[37492]: osdmap e197: 8 total, 8 up, 6 in 2024-04-25T05:39:15.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:15.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:15.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:15.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:15.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:15.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:39:15.629 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:15.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:15.705 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:15.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:15.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:15.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:15.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:16.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:16.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:16.177 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:16.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:16.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:16.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:16.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:16.446 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:16.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:16.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:16.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:16.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:16.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:16.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:16.928 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:16.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:39:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:16.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:16.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:17.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:17.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:17.111 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:17.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:17.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:17.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:16 smithi043 ceph-mon[27010]: pgmap v264: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:17.246 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:16 smithi043 ceph-mon[37492]: pgmap v264: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:16 smithi060 ceph-mon[30898]: pgmap v264: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:39:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:39:17.267 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:39:17.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:17.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:17.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:17.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:17.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:17.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:17.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:17.703 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:17.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:17.928 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:17.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:18.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:18.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:39:18.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:18.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:18.228 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_1' created 2024-04-25T05:39:18.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:18.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:18.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[27010]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:18.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-25T05:39:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[37492]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:18.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-25T05:39:18.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:18.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:18.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:18.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:18.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:18 smithi060 ceph-mon[30898]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-25T05:39:18.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:18.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:18.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:18.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:18.724 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-04-25T05:39:15.761792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1307,"num_read_kb":1027,"num_write":628,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2125,"ondisk_log_size":2125,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":452,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1087828,"kb_used_data":9468,"kb_used_omap":9,"kb_used_meta":1078262,"kb_avail":561383084,"statfs":{"total":575970213888,"available":574856278016,"internally_reserved":0,"allocated":9695232,"data_stored":5958399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":1104140965},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-233734,"num_write_kb":-7479425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-53193,"num_bytes_recovered":-3485794352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001571"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":266,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900607+0000","last_change":"2024-04-25T05:37:49.635619+0000","last_active":"2024-04-25T05:39:10.900607+0000","last_peered":"2024-04-25T05:39:10.900607+0000","last_clean":"2024-04-25T05:39:10.900607+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:39:10.900607+0000","last_undegraded":"2024-04-25T05:39:10.900607+0000","last_fullsized":"2024-04-25T05:39:10.900607+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:49.635538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:49.635538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.042388+0000","last_change":"2024-04-25T05:39:08.797448+0000","last_active":"2024-04-25T05:39:11.042388+0000","last_peered":"2024-04-25T05:39:11.042388+0000","last_clean":"2024-04-25T05:39:11.042388+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:39:11.042388+0000","last_undegraded":"2024-04-25T05:39:11.042388+0000","last_fullsized":"2024-04-25T05:39:11.042388+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:08.797334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.797334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.918004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439480+0000","last_change":"2024-04-25T05:39:07.379627+0000","last_active":"2024-04-25T05:39:11.439480+0000","last_peered":"2024-04-25T05:39:11.439480+0000","last_clean":"2024-04-25T05:39:11.439480+0000","last_became_active":"2024-04-25T05:38:13.237423+0000","last_became_peered":"2024-04-25T05:38:13.237423+0000","last_unstale":"2024-04-25T05:39:11.439480+0000","last_undegraded":"2024-04-25T05:39:11.439480+0000","last_fullsized":"2024-04-25T05:39:11.439480+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.552329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'15","reported_seq":320,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.899856+0000","last_change":"2024-04-25T05:36:54.265352+0000","last_active":"2024-04-25T05:39:10.899856+0000","last_peered":"2024-04-25T05:39:10.899856+0000","last_clean":"2024-04-25T05:39:10.899856+0000","last_became_active":"2024-04-25T05:36:54.243668+0000","last_became_peered":"2024-04-25T05:36:54.243668+0000","last_unstale":"2024-04-25T05:39:10.899856+0000","last_undegraded":"2024-04-25T05:39:10.899856+0000","last_fullsized":"2024-04-25T05:39:10.899856+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:36:13.708802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:13.708802+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":273,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900286+0000","last_change":"2024-04-25T05:39:09.117305+0000","last_active":"2024-04-25T05:39:10.900286+0000","last_peered":"2024-04-25T05:39:10.900286+0000","last_clean":"2024-04-25T05:39:10.900286+0000","last_became_active":"2024-04-25T05:36:54.235695+0000","last_became_peered":"2024-04-25T05:36:54.235695+0000","last_unstale":"2024-04-25T05:39:10.900286+0000","last_undegraded":"2024-04-25T05:39:10.900286+0000","last_fullsized":"2024-04-25T05:39:10.900286+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:18.796008+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":276,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900235+0000","last_change":"2024-04-25T05:37:47.617448+0000","last_active":"2024-04-25T05:39:10.900235+0000","last_peered":"2024-04-25T05:39:10.900235+0000","last_clean":"2024-04-25T05:39:10.900235+0000","last_became_active":"2024-04-25T05:36:54.229992+0000","last_became_peered":"2024-04-25T05:36:54.229992+0000","last_unstale":"2024-04-25T05:39:10.900235+0000","last_undegraded":"2024-04-25T05:39:10.900235+0000","last_fullsized":"2024-04-25T05:39:10.900235+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:47.617351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:47.617351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:16.683166+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902873+0000","last_change":"2024-04-25T05:36:54.230655+0000","last_active":"2024-04-25T05:39:10.902873+0000","last_peered":"2024-04-25T05:39:10.902873+0000","last_clean":"2024-04-25T05:39:10.902873+0000","last_became_active":"2024-04-25T05:36:54.230169+0000","last_became_peered":"2024-04-25T05:36:54.230169+0000","last_unstale":"2024-04-25T05:39:10.902873+0000","last_undegraded":"2024-04-25T05:39:10.902873+0000","last_fullsized":"2024-04-25T05:39:10.902873+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":370,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902961+0000","last_change":"2024-04-25T05:38:13.261747+0000","last_active":"2024-04-25T05:39:10.902961+0000","last_peered":"2024-04-25T05:39:10.902961+0000","last_clean":"2024-04-25T05:39:10.902961+0000","last_became_active":"2024-04-25T05:38:13.246798+0000","last_became_peered":"2024-04-25T05:38:13.246798+0000","last_unstale":"2024-04-25T05:39:10.902961+0000","last_undegraded":"2024-04-25T05:39:10.902961+0000","last_fullsized":"2024-04-25T05:39:10.902961+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039025+0000","last_change":"2024-04-25T05:38:59.924478+0000","last_active":"2024-04-25T05:39:11.039025+0000","last_peered":"2024-04-25T05:39:11.039025+0000","last_clean":"2024-04-25T05:39:11.039025+0000","last_became_active":"2024-04-25T05:38:13.304062+0000","last_became_peered":"2024-04-25T05:38:13.304062+0000","last_unstale":"2024-04-25T05:39:11.039025+0000","last_undegraded":"2024-04-25T05:39:11.039025+0000","last_fullsized":"2024-04-25T05:39:11.039025+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:02.578128+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":277,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438386+0000","last_change":"2024-04-25T05:38:13.239184+0000","last_active":"2024-04-25T05:39:11.438386+0000","last_peered":"2024-04-25T05:39:11.438386+0000","last_clean":"2024-04-25T05:39:11.438386+0000","last_became_active":"2024-04-25T05:38:13.237479+0000","last_became_peered":"2024-04-25T05:38:13.237479+0000","last_unstale":"2024-04-25T05:39:11.438386+0000","last_undegraded":"2024-04-25T05:39:11.438386+0000","last_fullsized":"2024-04-25T05:39:11.438386+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439121+0000","last_change":"2024-04-25T05:36:54.244217+0000","last_active":"2024-04-25T05:39:11.439121+0000","last_peered":"2024-04-25T05:39:11.439121+0000","last_clean":"2024-04-25T05:39:11.439121+0000","last_became_active":"2024-04-25T05:36:54.243924+0000","last_became_peered":"2024-04-25T05:36:54.243924+0000","last_unstale":"2024-04-25T05:39:11.439121+0000","last_undegraded":"2024-04-25T05:39:11.439121+0000","last_fullsized":"2024-04-25T05:39:11.439121+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.998645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.998645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":335,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900887+0000","last_change":"2024-04-25T05:38:13.254897+0000","last_active":"2024-04-25T05:39:10.900887+0000","last_peered":"2024-04-25T05:39:10.900887+0000","last_clean":"2024-04-25T05:39:10.900887+0000","last_became_active":"2024-04-25T05:38:13.239434+0000","last_became_peered":"2024-04-25T05:38:13.239434+0000","last_unstale":"2024-04-25T05:39:10.900887+0000","last_undegraded":"2024-04-25T05:39:10.900887+0000","last_fullsized":"2024-04-25T05:39:10.900887+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":275,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438197+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:39:11.438197+0000","last_peered":"2024-04-25T05:39:11.438197+0000","last_clean":"2024-04-25T05:39:11.438197+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:39:11.438197+0000","last_undegraded":"2024-04-25T05:39:11.438197+0000","last_fullsized":"2024-04-25T05:39:11.438197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035204+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:39:11.035204+0000","last_peered":"2024-04-25T05:39:11.035204+0000","last_clean":"2024-04-25T05:39:11.035204+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:39:11.035204+0000","last_undegraded":"2024-04-25T05:39:11.035204+0000","last_fullsized":"2024-04-25T05:39:11.035204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439254+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:39:11.439254+0000","last_peered":"2024-04-25T05:39:11.439254+0000","last_clean":"2024-04-25T05:39:11.439254+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:39:11.439254+0000","last_undegraded":"2024-04-25T05:39:11.439254+0000","last_fullsized":"2024-04-25T05:39:11.439254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":310,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902399+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:39:10.902399+0000","last_peered":"2024-04-25T05:39:10.902399+0000","last_clean":"2024-04-25T05:39:10.902399+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:39:10.902399+0000","last_undegraded":"2024-04-25T05:39:10.902399+0000","last_fullsized":"2024-04-25T05:39:10.902399+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":272,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901497+0000","last_change":"2024-04-25T05:36:54.245109+0000","last_active":"2024-04-25T05:39:10.901497+0000","last_peered":"2024-04-25T05:39:10.901497+0000","last_clean":"2024-04-25T05:39:10.901497+0000","last_became_active":"2024-04-25T05:36:54.244292+0000","last_became_peered":"2024-04-25T05:36:54.244292+0000","last_unstale":"2024-04-25T05:39:10.901497+0000","last_undegraded":"2024-04-25T05:39:10.901497+0000","last_fullsized":"2024-04-25T05:39:10.901497+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.045344+0000","last_change":"2024-04-25T05:38:13.285789+0000","last_active":"2024-04-25T05:39:11.045344+0000","last_peered":"2024-04-25T05:39:11.045344+0000","last_clean":"2024-04-25T05:39:11.045344+0000","last_became_active":"2024-04-25T05:38:13.285434+0000","last_became_peered":"2024-04-25T05:38:13.285434+0000","last_unstale":"2024-04-25T05:39:11.045344+0000","last_undegraded":"2024-04-25T05:39:11.045344+0000","last_fullsized":"2024-04-25T05:39:11.045344+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902285+0000","last_change":"2024-04-25T05:38:13.289898+0000","last_active":"2024-04-25T05:39:10.902285+0000","last_peered":"2024-04-25T05:39:10.902285+0000","last_clean":"2024-04-25T05:39:10.902285+0000","last_became_active":"2024-04-25T05:38:13.289539+0000","last_became_peered":"2024-04-25T05:38:13.289539+0000","last_unstale":"2024-04-25T05:39:10.902285+0000","last_undegraded":"2024-04-25T05:39:10.902285+0000","last_fullsized":"2024-04-25T05:39:10.902285+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901732+0000","last_change":"2024-04-25T05:37:50.872876+0000","last_active":"2024-04-25T05:39:10.901732+0000","last_peered":"2024-04-25T05:39:10.901732+0000","last_clean":"2024-04-25T05:39:10.901732+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:39:10.901732+0000","last_undegraded":"2024-04-25T05:39:10.901732+0000","last_fullsized":"2024-04-25T05:39:10.901732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:37:50.872688+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":273,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.948773+0000","last_change":"2024-04-25T05:39:06.146701+0000","last_active":"2024-04-25T05:39:10.948773+0000","last_peered":"2024-04-25T05:39:10.948773+0000","last_clean":"2024-04-25T05:39:10.948773+0000","last_became_active":"2024-04-25T05:36:54.237282+0000","last_became_peered":"2024-04-25T05:36:54.237282+0000","last_unstale":"2024-04-25T05:39:10.948773+0000","last_undegraded":"2024-04-25T05:39:10.948773+0000","last_fullsized":"2024-04-25T05:39:10.948773+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:06.486556+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":270,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.905284+0000","last_change":"2024-04-25T05:38:54.633069+0000","last_active":"2024-04-25T05:39:10.905284+0000","last_peered":"2024-04-25T05:39:10.905284+0000","last_clean":"2024-04-25T05:39:10.905284+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:39:10.905284+0000","last_undegraded":"2024-04-25T05:39:10.905284+0000","last_fullsized":"2024-04-25T05:39:10.905284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.565126+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":278,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901023+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:39:10.901023+0000","last_peered":"2024-04-25T05:39:10.901023+0000","last_clean":"2024-04-25T05:39:10.901023+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:39:10.901023+0000","last_undegraded":"2024-04-25T05:39:10.901023+0000","last_fullsized":"2024-04-25T05:39:10.901023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902638+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:39:10.902638+0000","last_peered":"2024-04-25T05:39:10.902638+0000","last_clean":"2024-04-25T05:39:10.902638+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:39:10.902638+0000","last_undegraded":"2024-04-25T05:39:10.902638+0000","last_fullsized":"2024-04-25T05:39:10.902638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":276,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036190+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:39:11.036190+0000","last_peered":"2024-04-25T05:39:11.036190+0000","last_clean":"2024-04-25T05:39:11.036190+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:39:11.036190+0000","last_undegraded":"2024-04-25T05:39:11.036190+0000","last_fullsized":"2024-04-25T05:39:11.036190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"130'9","reported_seq":205,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.903001+0000","last_change":"2024-04-25T05:36:54.240381+0000","last_active":"2024-04-25T05:39:10.903001+0000","last_peered":"2024-04-25T05:39:10.903001+0000","last_clean":"2024-04-25T05:39:10.903001+0000","last_became_active":"2024-04-25T05:36:54.232031+0000","last_became_peered":"2024-04-25T05:36:54.232031+0000","last_unstale":"2024-04-25T05:39:10.903001+0000","last_undegraded":"2024-04-25T05:39:10.903001+0000","last_fullsized":"2024-04-25T05:39:10.903001+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:22.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:22.989318+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.042310+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:39:11.042310+0000","last_peered":"2024-04-25T05:39:11.042310+0000","last_clean":"2024-04-25T05:39:11.042310+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:39:11.042310+0000","last_undegraded":"2024-04-25T05:39:11.042310+0000","last_fullsized":"2024-04-25T05:39:11.042310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":158,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.037712+0000","last_change":"2024-04-25T05:39:02.184665+0000","last_active":"2024-04-25T05:39:11.037712+0000","last_peered":"2024-04-25T05:39:11.037712+0000","last_clean":"2024-04-25T05:39:11.037712+0000","last_became_active":"2024-04-25T05:38:13.281967+0000","last_became_peered":"2024-04-25T05:38:13.281967+0000","last_unstale":"2024-04-25T05:39:11.037712+0000","last_undegraded":"2024-04-25T05:39:11.037712+0000","last_fullsized":"2024-04-25T05:39:11.037712+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.101025+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":267,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.045090+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:39:11.045090+0000","last_peered":"2024-04-25T05:39:11.045090+0000","last_clean":"2024-04-25T05:39:11.045090+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:39:11.045090+0000","last_undegraded":"2024-04-25T05:39:11.045090+0000","last_fullsized":"2024-04-25T05:39:11.045090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036580+0000","last_change":"2024-04-25T05:39:04.108082+0000","last_active":"2024-04-25T05:39:11.036580+0000","last_peered":"2024-04-25T05:39:11.036580+0000","last_clean":"2024-04-25T05:39:11.036580+0000","last_became_active":"2024-04-25T05:38:13.282204+0000","last_became_peered":"2024-04-25T05:38:13.282204+0000","last_unstale":"2024-04-25T05:39:11.036580+0000","last_undegraded":"2024-04-25T05:39:11.036580+0000","last_fullsized":"2024-04-25T05:39:11.036580+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.765638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900446+0000","last_change":"2024-04-25T05:38:13.305988+0000","last_active":"2024-04-25T05:39:10.900446+0000","last_peered":"2024-04-25T05:39:10.900446+0000","last_clean":"2024-04-25T05:39:10.900446+0000","last_became_active":"2024-04-25T05:38:13.305466+0000","last_became_peered":"2024-04-25T05:38:13.305466+0000","last_unstale":"2024-04-25T05:39:10.900446+0000","last_undegraded":"2024-04-25T05:39:10.900446+0000","last_fullsized":"2024-04-25T05:39:10.900446+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":237,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439751+0000","last_change":"2024-04-25T05:38:13.323005+0000","last_active":"2024-04-25T05:39:11.439751+0000","last_peered":"2024-04-25T05:39:11.439751+0000","last_clean":"2024-04-25T05:39:11.439751+0000","last_became_active":"2024-04-25T05:38:13.297139+0000","last_became_peered":"2024-04-25T05:38:13.297139+0000","last_unstale":"2024-04-25T05:39:11.439751+0000","last_undegraded":"2024-04-25T05:39:11.439751+0000","last_fullsized":"2024-04-25T05:39:11.439751+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:37:35.189581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:35.189581+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":276,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035315+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:39:11.035315+0000","last_peered":"2024-04-25T05:39:11.035315+0000","last_clean":"2024-04-25T05:39:11.035315+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:39:11.035315+0000","last_undegraded":"2024-04-25T05:39:11.035315+0000","last_fullsized":"2024-04-25T05:39:11.035315+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":151,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900526+0000","last_change":"2024-04-25T05:38:13.290337+0000","last_active":"2024-04-25T05:39:10.900526+0000","last_peered":"2024-04-25T05:39:10.900526+0000","last_clean":"2024-04-25T05:39:10.900526+0000","last_became_active":"2024-04-25T05:38:13.289541+0000","last_became_peered":"2024-04-25T05:38:13.289541+0000","last_unstale":"2024-04-25T05:39:10.900526+0000","last_undegraded":"2024-04-25T05:39:10.900526+0000","last_fullsized":"2024-04-25T05:39:10.900526+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:28.647857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:28.647857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439583+0000","last_change":"2024-04-25T05:38:13.296470+0000","last_active":"2024-04-25T05:39:11.439583+0000","last_peered":"2024-04-25T05:39:11.439583+0000","last_clean":"2024-04-25T05:39:11.439583+0000","last_became_active":"2024-04-25T05:38:13.295979+0000","last_became_peered":"2024-04-25T05:38:13.295979+0000","last_unstale":"2024-04-25T05:39:11.439583+0000","last_undegraded":"2024-04-25T05:39:11.439583+0000","last_fullsized":"2024-04-25T05:39:11.439583+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.217216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.217216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900536+0000","last_change":"2024-04-25T05:36:54.262135+0000","last_active":"2024-04-25T05:39:10.900536+0000","last_peered":"2024-04-25T05:39:10.900536+0000","last_clean":"2024-04-25T05:39:10.900536+0000","last_became_active":"2024-04-25T05:36:54.244002+0000","last_became_peered":"2024-04-25T05:36:54.244002+0000","last_unstale":"2024-04-25T05:39:10.900536+0000","last_undegraded":"2024-04-25T05:39:10.900536+0000","last_fullsized":"2024-04-25T05:39:10.900536+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.906932+0000","last_change":"2024-04-25T05:39:03.160875+0000","last_active":"2024-04-25T05:39:10.906932+0000","last_peered":"2024-04-25T05:39:10.906932+0000","last_clean":"2024-04-25T05:39:10.906932+0000","last_became_active":"2024-04-25T05:38:13.245676+0000","last_became_peered":"2024-04-25T05:38:13.245676+0000","last_unstale":"2024-04-25T05:39:10.906932+0000","last_undegraded":"2024-04-25T05:39:10.906932+0000","last_fullsized":"2024-04-25T05:39:10.906932+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.160752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.528850+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035477+0000","last_change":"2024-04-25T05:38:13.245873+0000","last_active":"2024-04-25T05:39:11.035477+0000","last_peered":"2024-04-25T05:39:11.035477+0000","last_clean":"2024-04-25T05:39:11.035477+0000","last_became_active":"2024-04-25T05:38:13.245293+0000","last_became_peered":"2024-04-25T05:38:13.245293+0000","last_unstale":"2024-04-25T05:39:11.035477+0000","last_undegraded":"2024-04-25T05:39:11.035477+0000","last_fullsized":"2024-04-25T05:39:11.035477+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439340+0000","last_change":"2024-04-25T05:36:16.072963+0000","last_active":"2024-04-25T05:39:11.439340+0000","last_peered":"2024-04-25T05:39:11.439340+0000","last_clean":"2024-04-25T05:39:11.439340+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:39:11.439340+0000","last_undegraded":"2024-04-25T05:39:11.439340+0000","last_fullsized":"2024-04-25T05:39:11.439340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.072833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.072833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":326,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902934+0000","last_change":"2024-04-25T05:36:54.252887+0000","last_active":"2024-04-25T05:39:10.902934+0000","last_peered":"2024-04-25T05:39:10.902934+0000","last_clean":"2024-04-25T05:39:10.902934+0000","last_became_active":"2024-04-25T05:36:54.232522+0000","last_became_peered":"2024-04-25T05:36:54.232522+0000","last_unstale":"2024-04-25T05:39:10.902934+0000","last_undegraded":"2024-04-25T05:39:10.902934+0000","last_fullsized":"2024-04-25T05:39:10.902934+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438229+0000","last_change":"2024-04-25T05:38:54.429610+0000","last_active":"2024-04-25T05:39:11.438229+0000","last_peered":"2024-04-25T05:39:11.438229+0000","last_clean":"2024-04-25T05:39:11.438229+0000","last_became_active":"2024-04-25T05:38:13.305781+0000","last_became_peered":"2024-04-25T05:38:13.305781+0000","last_unstale":"2024-04-25T05:39:11.438229+0000","last_undegraded":"2024-04-25T05:39:11.438229+0000","last_fullsized":"2024-04-25T05:39:11.438229+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.390666+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":269,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900196+0000","last_change":"2024-04-25T05:38:13.287078+0000","last_active":"2024-04-25T05:39:10.900196+0000","last_peered":"2024-04-25T05:39:10.900196+0000","last_clean":"2024-04-25T05:39:10.900196+0000","last_became_active":"2024-04-25T05:38:13.286756+0000","last_became_peered":"2024-04-25T05:38:13.286756+0000","last_unstale":"2024-04-25T05:39:10.900196+0000","last_undegraded":"2024-04-25T05:39:10.900196+0000","last_fullsized":"2024-04-25T05:39:10.900196+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036904+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:39:11.036904+0000","last_peered":"2024-04-25T05:39:11.036904+0000","last_clean":"2024-04-25T05:39:11.036904+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:39:11.036904+0000","last_undegraded":"2024-04-25T05:39:11.036904+0000","last_fullsized":"2024-04-25T05:39:11.036904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":93,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039468+0000","last_change":"2024-04-25T05:38:13.316204+0000","last_active":"2024-04-25T05:39:11.039468+0000","last_peered":"2024-04-25T05:39:11.039468+0000","last_clean":"2024-04-25T05:39:11.039468+0000","last_became_active":"2024-04-25T05:38:13.296338+0000","last_became_peered":"2024-04-25T05:38:13.296338+0000","last_unstale":"2024-04-25T05:39:11.039468+0000","last_undegraded":"2024-04-25T05:39:11.039468+0000","last_fullsized":"2024-04-25T05:39:11.039468+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":273,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902634+0000","last_change":"2024-04-25T05:39:04.115851+0000","last_active":"2024-04-25T05:39:10.902634+0000","last_peered":"2024-04-25T05:39:10.902634+0000","last_clean":"2024-04-25T05:39:10.902634+0000","last_became_active":"2024-04-25T05:38:13.246584+0000","last_became_peered":"2024-04-25T05:38:13.246584+0000","last_unstale":"2024-04-25T05:39:10.902634+0000","last_undegraded":"2024-04-25T05:39:10.902634+0000","last_fullsized":"2024-04-25T05:39:10.902634+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.115756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.115756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.122079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":271,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039467+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:39:11.039467+0000","last_peered":"2024-04-25T05:39:11.039467+0000","last_clean":"2024-04-25T05:39:11.039467+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:39:11.039467+0000","last_undegraded":"2024-04-25T05:39:11.039467+0000","last_fullsized":"2024-04-25T05:39:11.039467+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902010+0000","last_change":"2024-04-25T05:39:00.543672+0000","last_active":"2024-04-25T05:39:10.902010+0000","last_peered":"2024-04-25T05:39:10.902010+0000","last_clean":"2024-04-25T05:39:10.902010+0000","last_became_active":"2024-04-25T05:36:54.235617+0000","last_became_peered":"2024-04-25T05:36:54.235617+0000","last_unstale":"2024-04-25T05:39:10.902010+0000","last_undegraded":"2024-04-25T05:39:10.902010+0000","last_fullsized":"2024-04-25T05:39:10.902010+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:39:00.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.543557+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:10.367791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":319,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902564+0000","last_change":"2024-04-25T05:38:55.603487+0000","last_active":"2024-04-25T05:39:10.902564+0000","last_peered":"2024-04-25T05:39:10.902564+0000","last_clean":"2024-04-25T05:39:10.902564+0000","last_became_active":"2024-04-25T05:36:54.236292+0000","last_became_peered":"2024-04-25T05:36:54.236292+0000","last_unstale":"2024-04-25T05:39:10.902564+0000","last_undegraded":"2024-04-25T05:39:10.902564+0000","last_fullsized":"2024-04-25T05:39:10.902564+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:38:55.603358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.603358+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:00.207469+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035835+0000","last_change":"2024-04-25T05:38:13.246984+0000","last_active":"2024-04-25T05:39:11.035835+0000","last_peered":"2024-04-25T05:39:11.035835+0000","last_clean":"2024-04-25T05:39:11.035835+0000","last_became_active":"2024-04-25T05:38:13.246544+0000","last_became_peered":"2024-04-25T05:38:13.246544+0000","last_unstale":"2024-04-25T05:39:11.035835+0000","last_undegraded":"2024-04-25T05:39:11.035835+0000","last_fullsized":"2024-04-25T05:39:11.035835+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:30.108688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.108688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"160'9","reported_seq":361,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439179+0000","last_change":"2024-04-25T05:38:55.388653+0000","last_active":"2024-04-25T05:39:11.439179+0000","last_peered":"2024-04-25T05:39:11.439179+0000","last_clean":"2024-04-25T05:39:11.439179+0000","last_became_active":"2024-04-25T05:38:13.295760+0000","last_became_peered":"2024-04-25T05:38:13.295760+0000","last_unstale":"2024-04-25T05:39:11.439179+0000","last_undegraded":"2024-04-25T05:39:11.439179+0000","last_fullsized":"2024-04-25T05:39:11.439179+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'9","last_scrub_stamp":"2024-04-25T05:38:55.388487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.388487+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:57.015303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035300+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:39:11.035300+0000","last_peered":"2024-04-25T05:39:11.035300+0000","last_clean":"2024-04-25T05:39:11.035300+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:39:11.035300+0000","last_undegraded":"2024-04-25T05:39:11.035300+0000","last_fullsized":"2024-04-25T05:39:11.035300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.909868+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:39:10.909868+0000","last_peered":"2024-04-25T05:39:10.909868+0000","last_clean":"2024-04-25T05:39:10.909868+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:39:10.909868+0000","last_undegraded":"2024-04-25T05:39:10.909868+0000","last_fullsized":"2024-04-25T05:39:10.909868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":266,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902688+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:39:10.902688+0000","last_peered":"2024-04-25T05:39:10.902688+0000","last_clean":"2024-04-25T05:39:10.902688+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:39:10.902688+0000","last_undegraded":"2024-04-25T05:39:10.902688+0000","last_fullsized":"2024-04-25T05:39:10.902688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":334,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.911132+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:39:10.911132+0000","last_peered":"2024-04-25T05:39:10.911132+0000","last_clean":"2024-04-25T05:39:10.911132+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:39:10.911132+0000","last_undegraded":"2024-04-25T05:39:10.911132+0000","last_fullsized":"2024-04-25T05:39:10.911132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035099+0000","last_change":"2024-04-25T05:38:13.282321+0000","last_active":"2024-04-25T05:39:11.035099+0000","last_peered":"2024-04-25T05:39:11.035099+0000","last_clean":"2024-04-25T05:39:11.035099+0000","last_became_active":"2024-04-25T05:38:13.282059+0000","last_became_peered":"2024-04-25T05:38:13.282059+0000","last_unstale":"2024-04-25T05:39:11.035099+0000","last_undegraded":"2024-04-25T05:39:11.035099+0000","last_fullsized":"2024-04-25T05:39:11.035099+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.117874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":316,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439344+0000","last_change":"2024-04-25T05:39:03.365135+0000","last_active":"2024-04-25T05:39:11.439344+0000","last_peered":"2024-04-25T05:39:11.439344+0000","last_clean":"2024-04-25T05:39:11.439344+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:39:11.439344+0000","last_undegraded":"2024-04-25T05:39:11.439344+0000","last_fullsized":"2024-04-25T05:39:11.439344+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:39:03.364988+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.364988+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.589680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":274,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902663+0000","last_change":"2024-04-25T05:38:13.947132+0000","last_active":"2024-04-25T05:39:10.902663+0000","last_peered":"2024-04-25T05:39:10.902663+0000","last_clean":"2024-04-25T05:39:10.902663+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:39:10.902663+0000","last_undegraded":"2024-04-25T05:39:10.902663+0000","last_fullsized":"2024-04-25T05:39:10.902663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:13.947044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:13.947044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:19.521993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":263,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.931995+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:39:10.931995+0000","last_peered":"2024-04-25T05:39:10.931995+0000","last_clean":"2024-04-25T05:39:10.931995+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:39:10.931995+0000","last_undegraded":"2024-04-25T05:39:10.931995+0000","last_fullsized":"2024-04-25T05:39:10.931995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035566+0000","last_change":"2024-04-25T05:38:13.241002+0000","last_active":"2024-04-25T05:39:11.035566+0000","last_peered":"2024-04-25T05:39:11.035566+0000","last_clean":"2024-04-25T05:39:11.035566+0000","last_became_active":"2024-04-25T05:38:13.240401+0000","last_became_peered":"2024-04-25T05:38:13.240401+0000","last_unstale":"2024-04-25T05:39:11.035566+0000","last_undegraded":"2024-04-25T05:39:11.035566+0000","last_fullsized":"2024-04-25T05:39:11.035566+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":157,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.904016+0000","last_change":"2024-04-25T05:38:13.255668+0000","last_active":"2024-04-25T05:39:10.904016+0000","last_peered":"2024-04-25T05:39:10.904016+0000","last_clean":"2024-04-25T05:39:10.904016+0000","last_became_active":"2024-04-25T05:38:13.245172+0000","last_became_peered":"2024-04-25T05:38:13.245172+0000","last_unstale":"2024-04-25T05:39:10.904016+0000","last_undegraded":"2024-04-25T05:39:10.904016+0000","last_fullsized":"2024-04-25T05:39:10.904016+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036808+0000","last_change":"2024-04-25T05:37:41.064307+0000","last_active":"2024-04-25T05:39:11.036808+0000","last_peered":"2024-04-25T05:39:11.036808+0000","last_clean":"2024-04-25T05:39:11.036808+0000","last_became_active":"2024-04-25T05:36:54.232503+0000","last_became_peered":"2024-04-25T05:36:54.232503+0000","last_unstale":"2024-04-25T05:39:11.036808+0000","last_undegraded":"2024-04-25T05:39:11.036808+0000","last_fullsized":"2024-04-25T05:39:11.036808+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":271,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439229+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:39:11.439229+0000","last_peered":"2024-04-25T05:39:11.439229+0000","last_clean":"2024-04-25T05:39:11.439229+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:39:11.439229+0000","last_undegraded":"2024-04-25T05:39:11.439229+0000","last_fullsized":"2024-04-25T05:39:11.439229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":95,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438122+0000","last_change":"2024-04-25T05:38:26.418295+0000","last_active":"2024-04-25T05:39:11.438122+0000","last_peered":"2024-04-25T05:39:11.438122+0000","last_clean":"2024-04-25T05:39:11.438122+0000","last_became_active":"2024-04-25T05:38:13.244520+0000","last_became_peered":"2024-04-25T05:38:13.244520+0000","last_unstale":"2024-04-25T05:39:11.438122+0000","last_undegraded":"2024-04-25T05:39:11.438122+0000","last_fullsized":"2024-04-25T05:39:11.438122+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:26.418226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:38:26.418226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:47.053592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.948917+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:39:10.948917+0000","last_peered":"2024-04-25T05:39:10.948917+0000","last_clean":"2024-04-25T05:39:10.948917+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:39:10.948917+0000","last_undegraded":"2024-04-25T05:39:10.948917+0000","last_fullsized":"2024-04-25T05:39:10.948917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.440543+0000","last_change":"2024-04-25T05:38:13.285501+0000","last_active":"2024-04-25T05:39:11.440543+0000","last_peered":"2024-04-25T05:39:11.440543+0000","last_clean":"2024-04-25T05:39:11.440543+0000","last_became_active":"2024-04-25T05:38:13.284963+0000","last_became_peered":"2024-04-25T05:38:13.284963+0000","last_unstale":"2024-04-25T05:39:11.440543+0000","last_undegraded":"2024-04-25T05:39:11.440543+0000","last_fullsized":"2024-04-25T05:39:11.440543+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036583+0000","last_change":"2024-04-25T05:38:13.313381+0000","last_active":"2024-04-25T05:39:11.036583+0000","last_peered":"2024-04-25T05:39:11.036583+0000","last_clean":"2024-04-25T05:39:11.036583+0000","last_became_active":"2024-04-25T05:38:13.295320+0000","last_became_peered":"2024-04-25T05:38:13.295320+0000","last_unstale":"2024-04-25T05:39:11.036583+0000","last_undegraded":"2024-04-25T05:39:11.036583+0000","last_fullsized":"2024-04-25T05:39:11.036583+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":350,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438119+0000","last_change":"2024-04-25T05:36:54.267732+0000","last_active":"2024-04-25T05:39:11.438119+0000","last_peered":"2024-04-25T05:39:11.438119+0000","last_clean":"2024-04-25T05:39:11.438119+0000","last_became_active":"2024-04-25T05:36:54.244247+0000","last_became_peered":"2024-04-25T05:36:54.244247+0000","last_unstale":"2024-04-25T05:39:11.438119+0000","last_undegraded":"2024-04-25T05:39:11.438119+0000","last_fullsized":"2024-04-25T05:39:11.438119+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:36:17.062126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.062126+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"130'10","reported_seq":364,"reported_epoch":196,"state":"active+clean+scrubbing","last_fresh":"2024-04-25T05:39:11.036131+0000","last_change":"2024-04-25T05:39:11.032969+0000","last_active":"2024-04-25T05:39:11.036131+0000","last_peered":"2024-04-25T05:39:11.036131+0000","last_clean":"2024-04-25T05:39:11.036131+0000","last_became_active":"2024-04-25T05:36:54.235542+0000","last_became_peered":"2024-04-25T05:36:54.235542+0000","last_unstale":"2024-04-25T05:39:11.036131+0000","last_undegraded":"2024-04-25T05:39:11.036131+0000","last_fullsized":"2024-04-25T05:39:11.036131+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'9","last_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.348920+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":259,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900374+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:39:10.900374+0000","last_peered":"2024-04-25T05:39:10.900374+0000","last_clean":"2024-04-25T05:39:10.900374+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:39:10.900374+0000","last_undegraded":"2024-04-25T05:39:10.900374+0000","last_fullsized":"2024-04-25T05:39:10.900374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036463+0000","last_change":"2024-04-25T05:36:48.522033+0000","last_active":"2024-04-25T05:39:11.036463+0000","last_peered":"2024-04-25T05:39:11.036463+0000","last_clean":"2024-04-25T05:39:11.036463+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:39:11.036463+0000","last_undegraded":"2024-04-25T05:39:11.036463+0000","last_fullsized":"2024-04-25T05:39:11.036463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:48.521908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:48.521908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":329,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901303+0000","last_change":"2024-04-25T05:39:10.593759+0000","last_active":"2024-04-25T05:39:10.901303+0000","last_peered":"2024-04-25T05:39:10.901303+0000","last_clean":"2024-04-25T05:39:10.901303+0000","last_became_active":"2024-04-25T05:34:00.693495+0000","last_became_peered":"2024-04-25T05:34:00.693495+0000","last_unstale":"2024-04-25T05:39:10.901303+0000","last_undegraded":"2024-04-25T05:39:10.901303+0000","last_fullsized":"2024-04-25T05:39:10.901303+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:39:10.593601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:10.593601+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:29.311795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901657+0000","last_change":"2024-04-25T05:38:13.302895+0000","last_active":"2024-04-25T05:39:10.901657+0000","last_peered":"2024-04-25T05:39:10.901657+0000","last_clean":"2024-04-25T05:39:10.901657+0000","last_became_active":"2024-04-25T05:38:13.302285+0000","last_became_peered":"2024-04-25T05:38:13.302285+0000","last_unstale":"2024-04-25T05:39:10.901657+0000","last_undegraded":"2024-04-25T05:39:10.901657+0000","last_fullsized":"2024-04-25T05:39:10.901657+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:47.489244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.489244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":267,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.034914+0000","last_change":"2024-04-25T05:36:25.396141+0000","last_active":"2024-04-25T05:39:11.034914+0000","last_peered":"2024-04-25T05:39:11.034914+0000","last_clean":"2024-04-25T05:39:11.034914+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:39:11.034914+0000","last_undegraded":"2024-04-25T05:39:11.034914+0000","last_fullsized":"2024-04-25T05:39:11.034914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:25.396064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:25.396064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.907318+0000","last_change":"2024-04-25T05:38:13.236944+0000","last_active":"2024-04-25T05:39:10.907318+0000","last_peered":"2024-04-25T05:39:10.907318+0000","last_clean":"2024-04-25T05:39:10.907318+0000","last_became_active":"2024-04-25T05:38:13.236574+0000","last_became_peered":"2024-04-25T05:38:13.236574+0000","last_unstale":"2024-04-25T05:39:10.907318+0000","last_undegraded":"2024-04-25T05:39:10.907318+0000","last_fullsized":"2024-04-25T05:39:10.907318+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:51.573898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:51.573898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":303,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.037102+0000","last_change":"2024-04-25T05:36:54.240931+0000","last_active":"2024-04-25T05:39:11.037102+0000","last_peered":"2024-04-25T05:39:11.037102+0000","last_clean":"2024-04-25T05:39:11.037102+0000","last_became_active":"2024-04-25T05:36:54.232602+0000","last_became_peered":"2024-04-25T05:36:54.232602+0000","last_unstale":"2024-04-25T05:39:11.037102+0000","last_undegraded":"2024-04-25T05:39:11.037102+0000","last_fullsized":"2024-04-25T05:39:11.037102+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":229,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039348+0000","last_change":"2024-04-25T05:36:54.260249+0000","last_active":"2024-04-25T05:39:11.039348+0000","last_peered":"2024-04-25T05:39:11.039348+0000","last_clean":"2024-04-25T05:39:11.039348+0000","last_became_active":"2024-04-25T05:36:54.235633+0000","last_became_peered":"2024-04-25T05:36:54.235633+0000","last_unstale":"2024-04-25T05:39:11.039348+0000","last_undegraded":"2024-04-25T05:39:11.039348+0000","last_fullsized":"2024-04-25T05:39:11.039348+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:35:32.693324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:32.693324+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":92,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035195+0000","last_change":"2024-04-25T05:39:06.050576+0000","last_active":"2024-04-25T05:39:11.035195+0000","last_peered":"2024-04-25T05:39:11.035195+0000","last_clean":"2024-04-25T05:39:11.035195+0000","last_became_active":"2024-04-25T05:38:13.240067+0000","last_became_peered":"2024-04-25T05:38:13.240067+0000","last_unstale":"2024-04-25T05:39:11.035195+0000","last_undegraded":"2024-04-25T05:39:11.035195+0000","last_fullsized":"2024-04-25T05:39:11.035195+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.492020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":270,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.905204+0000","last_change":"2024-04-25T05:35:38.723838+0000","last_active":"2024-04-25T05:39:10.905204+0000","last_peered":"2024-04-25T05:39:10.905204+0000","last_clean":"2024-04-25T05:39:10.905204+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:39:10.905204+0000","last_undegraded":"2024-04-25T05:39:10.905204+0000","last_fullsized":"2024-04-25T05:39:10.905204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:38.723143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.723143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439367+0000","last_change":"2024-04-25T05:37:30.237718+0000","last_active":"2024-04-25T05:39:11.439367+0000","last_peered":"2024-04-25T05:39:11.439367+0000","last_clean":"2024-04-25T05:39:11.439367+0000","last_became_active":"2024-04-25T05:36:54.234986+0000","last_became_peered":"2024-04-25T05:36:54.234986+0000","last_unstale":"2024-04-25T05:39:11.439367+0000","last_undegraded":"2024-04-25T05:39:11.439367+0000","last_fullsized":"2024-04-25T05:39:11.439367+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'15","reported_seq":143,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039624+0000","last_change":"2024-04-25T05:38:13.317524+0000","last_active":"2024-04-25T05:39:11.039624+0000","last_peered":"2024-04-25T05:39:11.039624+0000","last_clean":"2024-04-25T05:39:11.039624+0000","last_became_active":"2024-04-25T05:38:13.294321+0000","last_became_peered":"2024-04-25T05:38:13.294321+0000","last_unstale":"2024-04-25T05:39:11.039624+0000","last_undegraded":"2024-04-25T05:39:11.039624+0000","last_fullsized":"2024-04-25T05:39:11.039624+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:40.352414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:40.352414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.437929+0000","last_change":"2024-04-25T05:38:58.389048+0000","last_active":"2024-04-25T05:39:11.437929+0000","last_peered":"2024-04-25T05:39:11.437929+0000","last_clean":"2024-04-25T05:39:11.437929+0000","last_became_active":"2024-04-25T05:38:34.759567+0000","last_became_peered":"2024-04-25T05:38:34.759567+0000","last_unstale":"2024-04-25T05:39:11.437929+0000","last_undegraded":"2024-04-25T05:39:11.437929+0000","last_fullsized":"2024-04-25T05:39:11.437929+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.388898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:06.835407+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":267,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.904105+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:39:10.904105+0000","last_peered":"2024-04-25T05:39:10.904105+0000","last_clean":"2024-04-25T05:39:10.904105+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:39:10.904105+0000","last_undegraded":"2024-04-25T05:39:10.904105+0000","last_fullsized":"2024-04-25T05:39:10.904105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":317,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438782+0000","last_change":"2024-04-25T05:39:08.351706+0000","last_active":"2024-04-25T05:39:11.438782+0000","last_peered":"2024-04-25T05:39:11.438782+0000","last_clean":"2024-04-25T05:39:11.438782+0000","last_became_active":"2024-04-25T05:38:13.284779+0000","last_became_peered":"2024-04-25T05:38:13.284779+0000","last_unstale":"2024-04-25T05:39:11.438782+0000","last_undegraded":"2024-04-25T05:39:11.438782+0000","last_fullsized":"2024-04-25T05:39:11.438782+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:21.066221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901939+0000","last_change":"2024-04-25T05:38:13.277150+0000","last_active":"2024-04-25T05:39:10.901939+0000","last_peered":"2024-04-25T05:39:10.901939+0000","last_clean":"2024-04-25T05:39:10.901939+0000","last_became_active":"2024-04-25T05:38:13.276807+0000","last_became_peered":"2024-04-25T05:38:13.276807+0000","last_unstale":"2024-04-25T05:39:10.901939+0000","last_undegraded":"2024-04-25T05:39:10.901939+0000","last_fullsized":"2024-04-25T05:39:10.901939+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:37:43.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":409,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.040653+0000","last_change":"2024-04-25T05:35:44.336529+0000","last_active":"2024-04-25T05:39:11.040653+0000","last_peered":"2024-04-25T05:39:11.040653+0000","last_clean":"2024-04-25T05:39:11.040653+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:39:11.040653+0000","last_undegraded":"2024-04-25T05:39:11.040653+0000","last_fullsized":"2024-04-25T05:39:11.040653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.336118+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.045430+0000","last_change":"2024-04-25T05:38:13.286961+0000","last_active":"2024-04-25T05:39:11.045430+0000","last_peered":"2024-04-25T05:39:11.045430+0000","last_clean":"2024-04-25T05:39:11.045430+0000","last_became_active":"2024-04-25T05:38:13.286575+0000","last_became_peered":"2024-04-25T05:38:13.286575+0000","last_unstale":"2024-04-25T05:39:11.045430+0000","last_undegraded":"2024-04-25T05:39:11.045430+0000","last_fullsized":"2024-04-25T05:39:11.045430+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":585,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035217+0000","last_change":"2024-04-25T05:39:00.133654+0000","last_active":"2024-04-25T05:39:11.035217+0000","last_peered":"2024-04-25T05:39:11.035217+0000","last_clean":"2024-04-25T05:39:11.035217+0000","last_became_active":"2024-04-25T05:38:34.758423+0000","last_became_peered":"2024-04-25T05:38:34.758423+0000","last_unstale":"2024-04-25T05:39:11.035217+0000","last_undegraded":"2024-04-25T05:39:11.035217+0000","last_fullsized":"2024-04-25T05:39:11.035217+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:00.133580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.133580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:12.718787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.037287+0000","last_change":"2024-04-25T05:36:54.626908+0000","last_active":"2024-04-25T05:39:11.037287+0000","last_peered":"2024-04-25T05:39:11.037287+0000","last_clean":"2024-04-25T05:39:11.037287+0000","last_became_active":"2024-04-25T05:36:54.225394+0000","last_became_peered":"2024-04-25T05:36:54.225394+0000","last_unstale":"2024-04-25T05:39:11.037287+0000","last_undegraded":"2024-04-25T05:39:11.037287+0000","last_fullsized":"2024-04-25T05:39:11.037287+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039628+0000","last_change":"2024-04-25T05:38:13.307221+0000","last_active":"2024-04-25T05:39:11.039628+0000","last_peered":"2024-04-25T05:39:11.039628+0000","last_clean":"2024-04-25T05:39:11.039628+0000","last_became_active":"2024-04-25T05:38:13.306297+0000","last_became_peered":"2024-04-25T05:38:13.306297+0000","last_unstale":"2024-04-25T05:39:11.039628+0000","last_undegraded":"2024-04-25T05:39:11.039628+0000","last_fullsized":"2024-04-25T05:39:11.039628+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:52.054641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:52.054641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:17.305985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900240+0000","last_change":"2024-04-25T05:38:13.284622+0000","last_active":"2024-04-25T05:39:10.900240+0000","last_peered":"2024-04-25T05:39:10.900240+0000","last_clean":"2024-04-25T05:39:10.900240+0000","last_became_active":"2024-04-25T05:38:13.284281+0000","last_became_peered":"2024-04-25T05:38:13.284281+0000","last_unstale":"2024-04-25T05:39:10.900240+0000","last_undegraded":"2024-04-25T05:39:10.900240+0000","last_fullsized":"2024-04-25T05:39:10.900240+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.674049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.674049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":340,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.041585+0000","last_change":"2024-04-25T05:38:13.314279+0000","last_active":"2024-04-25T05:39:11.041585+0000","last_peered":"2024-04-25T05:39:11.041585+0000","last_clean":"2024-04-25T05:39:11.041585+0000","last_became_active":"2024-04-25T05:38:13.295547+0000","last_became_peered":"2024-04-25T05:38:13.295547+0000","last_unstale":"2024-04-25T05:39:11.041585+0000","last_undegraded":"2024-04-25T05:39:11.041585+0000","last_fullsized":"2024-04-25T05:39:11.041585+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.905047+0000","last_change":"2024-04-25T05:38:59.769234+0000","last_active":"2024-04-25T05:39:10.905047+0000","last_peered":"2024-04-25T05:39:10.905047+0000","last_clean":"2024-04-25T05:39:10.905047+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:39:10.905047+0000","last_undegraded":"2024-04-25T05:39:10.905047+0000","last_fullsized":"2024-04-25T05:39:10.905047+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.769120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.769120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.633366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":275,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439128+0000","last_change":"2024-04-25T05:39:09.353431+0000","last_active":"2024-04-25T05:39:11.439128+0000","last_peered":"2024-04-25T05:39:11.439128+0000","last_clean":"2024-04-25T05:39:11.439128+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:39:11.439128+0000","last_undegraded":"2024-04-25T05:39:11.439128+0000","last_fullsized":"2024-04-25T05:39:11.439128+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.353310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:10.766600+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":577,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035162+0000","last_change":"2024-04-25T05:38:59.104672+0000","last_active":"2024-04-25T05:39:11.035162+0000","last_peered":"2024-04-25T05:39:11.035162+0000","last_clean":"2024-04-25T05:39:11.035162+0000","last_became_active":"2024-04-25T05:38:29.240034+0000","last_became_peered":"2024-04-25T05:38:29.240034+0000","last_unstale":"2024-04-25T05:39:11.035162+0000","last_undegraded":"2024-04-25T05:39:11.035162+0000","last_fullsized":"2024-04-25T05:39:11.035162+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.104548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.104548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.075012+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.907103+0000","last_change":"2024-04-25T05:36:54.233589+0000","last_active":"2024-04-25T05:39:10.907103+0000","last_peered":"2024-04-25T05:39:10.907103+0000","last_clean":"2024-04-25T05:39:10.907103+0000","last_became_active":"2024-04-25T05:36:54.232305+0000","last_became_peered":"2024-04-25T05:36:54.232305+0000","last_unstale":"2024-04-25T05:39:10.907103+0000","last_undegraded":"2024-04-25T05:39:10.907103+0000","last_fullsized":"2024-04-25T05:39:10.907103+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"160'18","reported_seq":267,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901600+0000","last_change":"2024-04-25T05:39:04.573814+0000","last_active":"2024-04-25T05:39:10.901600+0000","last_peered":"2024-04-25T05:39:10.901600+0000","last_clean":"2024-04-25T05:39:10.901600+0000","last_became_active":"2024-04-25T05:38:13.240414+0000","last_became_peered":"2024-04-25T05:38:13.240414+0000","last_unstale":"2024-04-25T05:39:10.901600+0000","last_undegraded":"2024-04-25T05:39:10.901600+0000","last_fullsized":"2024-04-25T05:39:10.901600+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'18","last_scrub_stamp":"2024-04-25T05:39:04.573693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.573693+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.209448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.036304+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:39:11.036304+0000","last_peered":"2024-04-25T05:39:11.036304+0000","last_clean":"2024-04-25T05:39:11.036304+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:39:11.036304+0000","last_undegraded":"2024-04-25T05:39:11.036304+0000","last_fullsized":"2024-04-25T05:39:11.036304+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":460,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039201+0000","last_change":"2024-04-25T05:38:58.010459+0000","last_active":"2024-04-25T05:39:11.039201+0000","last_peered":"2024-04-25T05:39:11.039201+0000","last_clean":"2024-04-25T05:39:11.039201+0000","last_became_active":"2024-04-25T05:38:34.751191+0000","last_became_peered":"2024-04-25T05:38:34.751191+0000","last_unstale":"2024-04-25T05:39:11.039201+0000","last_undegraded":"2024-04-25T05:39:11.039201+0000","last_fullsized":"2024-04-25T05:39:11.039201+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.010323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.010323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:11.636120+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":275,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.040407+0000","last_change":"2024-04-25T05:39:00.922795+0000","last_active":"2024-04-25T05:39:11.040407+0000","last_peered":"2024-04-25T05:39:11.040407+0000","last_clean":"2024-04-25T05:39:11.040407+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:39:11.040407+0000","last_undegraded":"2024-04-25T05:39:11.040407+0000","last_fullsized":"2024-04-25T05:39:11.040407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.922679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.922679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:18.355708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900900+0000","last_change":"2024-04-25T05:38:13.239349+0000","last_active":"2024-04-25T05:39:10.900900+0000","last_peered":"2024-04-25T05:39:10.900900+0000","last_clean":"2024-04-25T05:39:10.900900+0000","last_became_active":"2024-04-25T05:38:13.238969+0000","last_became_peered":"2024-04-25T05:38:13.238969+0000","last_unstale":"2024-04-25T05:39:10.900900+0000","last_undegraded":"2024-04-25T05:39:10.900900+0000","last_fullsized":"2024-04-25T05:39:10.900900+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":616,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.393641+0000","last_change":"2024-04-25T05:38:13.325017+0000","last_active":"2024-04-25T05:39:11.393641+0000","last_peered":"2024-04-25T05:39:11.393641+0000","last_clean":"2024-04-25T05:39:11.393641+0000","last_became_active":"2024-04-25T05:38:13.305327+0000","last_became_peered":"2024-04-25T05:38:13.305327+0000","last_unstale":"2024-04-25T05:39:11.393641+0000","last_undegraded":"2024-04-25T05:39:11.393641+0000","last_fullsized":"2024-04-25T05:39:11.393641+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":113,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438994+0000","last_change":"2024-04-25T05:37:56.295161+0000","last_active":"2024-04-25T05:39:11.438994+0000","last_peered":"2024-04-25T05:39:11.438994+0000","last_clean":"2024-04-25T05:39:11.438994+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:39:11.438994+0000","last_undegraded":"2024-04-25T05:39:11.438994+0000","last_fullsized":"2024-04-25T05:39:11.438994+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439945+0000","last_change":"2024-04-25T05:38:13.296713+0000","last_active":"2024-04-25T05:39:11.439945+0000","last_peered":"2024-04-25T05:39:11.439945+0000","last_clean":"2024-04-25T05:39:11.439945+0000","last_became_active":"2024-04-25T05:38:13.296013+0000","last_became_peered":"2024-04-25T05:38:13.296013+0000","last_unstale":"2024-04-25T05:39:11.439945+0000","last_undegraded":"2024-04-25T05:39:11.439945+0000","last_fullsized":"2024-04-25T05:39:11.439945+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902611+0000","last_change":"2024-04-25T05:38:57.081345+0000","last_active":"2024-04-25T05:39:10.902611+0000","last_peered":"2024-04-25T05:39:10.902611+0000","last_clean":"2024-04-25T05:39:10.902611+0000","last_became_active":"2024-04-25T05:38:34.758128+0000","last_became_peered":"2024-04-25T05:38:34.758128+0000","last_unstale":"2024-04-25T05:39:10.902611+0000","last_undegraded":"2024-04-25T05:39:10.902611+0000","last_fullsized":"2024-04-25T05:39:10.902611+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.081204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.081204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:16.313151+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":273,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.042458+0000","last_change":"2024-04-25T05:36:54.234953+0000","last_active":"2024-04-25T05:39:11.042458+0000","last_peered":"2024-04-25T05:39:11.042458+0000","last_clean":"2024-04-25T05:39:11.042458+0000","last_became_active":"2024-04-25T05:36:54.234602+0000","last_became_peered":"2024-04-25T05:36:54.234602+0000","last_unstale":"2024-04-25T05:39:11.042458+0000","last_undegraded":"2024-04-25T05:39:11.042458+0000","last_fullsized":"2024-04-25T05:39:11.042458+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.237176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.237176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.040578+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:39:11.040578+0000","last_peered":"2024-04-25T05:39:11.040578+0000","last_clean":"2024-04-25T05:39:11.040578+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:39:11.040578+0000","last_undegraded":"2024-04-25T05:39:11.040578+0000","last_fullsized":"2024-04-25T05:39:11.040578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":179,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438413+0000","last_change":"2024-04-25T05:38:22.405354+0000","last_active":"2024-04-25T05:39:11.438413+0000","last_peered":"2024-04-25T05:39:11.438413+0000","last_clean":"2024-04-25T05:39:11.438413+0000","last_became_active":"2024-04-25T05:38:13.296480+0000","last_became_peered":"2024-04-25T05:38:13.296480+0000","last_unstale":"2024-04-25T05:39:11.438413+0000","last_undegraded":"2024-04-25T05:39:11.438413+0000","last_fullsized":"2024-04-25T05:39:11.438413+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:38:22.404901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:22.404901+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:24.353079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":87,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438948+0000","last_change":"2024-04-25T05:38:13.239510+0000","last_active":"2024-04-25T05:39:11.438948+0000","last_peered":"2024-04-25T05:39:11.438948+0000","last_clean":"2024-04-25T05:39:11.438948+0000","last_became_active":"2024-04-25T05:38:13.237844+0000","last_became_peered":"2024-04-25T05:38:13.237844+0000","last_unstale":"2024-04-25T05:39:11.438948+0000","last_undegraded":"2024-04-25T05:39:11.438948+0000","last_fullsized":"2024-04-25T05:39:11.438948+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.964139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.899919+0000","last_change":"2024-04-25T05:38:59.069446+0000","last_active":"2024-04-25T05:39:10.899919+0000","last_peered":"2024-04-25T05:39:10.899919+0000","last_clean":"2024-04-25T05:39:10.899919+0000","last_became_active":"2024-04-25T05:38:34.761903+0000","last_became_peered":"2024-04-25T05:38:34.761903+0000","last_unstale":"2024-04-25T05:39:10.899919+0000","last_undegraded":"2024-04-25T05:39:10.899919+0000","last_fullsized":"2024-04-25T05:39:10.899919+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.069332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.069332+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:19.625381+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":84,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900351+0000","last_change":"2024-04-25T05:38:13.244062+0000","last_active":"2024-04-25T05:39:10.900351+0000","last_peered":"2024-04-25T05:39:10.900351+0000","last_clean":"2024-04-25T05:39:10.900351+0000","last_became_active":"2024-04-25T05:38:13.243650+0000","last_became_peered":"2024-04-25T05:38:13.243650+0000","last_unstale":"2024-04-25T05:39:10.900351+0000","last_undegraded":"2024-04-25T05:39:10.900351+0000","last_fullsized":"2024-04-25T05:39:10.900351+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439238+0000","last_change":"2024-04-25T05:36:14.133939+0000","last_active":"2024-04-25T05:39:11.439238+0000","last_peered":"2024-04-25T05:39:11.439238+0000","last_clean":"2024-04-25T05:39:11.439238+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:39:11.439238+0000","last_undegraded":"2024-04-25T05:39:11.439238+0000","last_fullsized":"2024-04-25T05:39:11.439238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:14.133801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:14.133801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":246,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.904143+0000","last_change":"2024-04-25T05:39:09.887432+0000","last_active":"2024-04-25T05:39:10.904143+0000","last_peered":"2024-04-25T05:39:10.904143+0000","last_clean":"2024-04-25T05:39:10.904143+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:39:10.904143+0000","last_undegraded":"2024-04-25T05:39:10.904143+0000","last_fullsized":"2024-04-25T05:39:10.904143+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.887340+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.423904+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"160'10","reported_seq":133,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438576+0000","last_change":"2024-04-25T05:38:24.406772+0000","last_active":"2024-04-25T05:39:11.438576+0000","last_peered":"2024-04-25T05:39:11.438576+0000","last_clean":"2024-04-25T05:39:11.438576+0000","last_became_active":"2024-04-25T05:38:13.295784+0000","last_became_peered":"2024-04-25T05:38:13.295784+0000","last_unstale":"2024-04-25T05:39:11.438576+0000","last_undegraded":"2024-04-25T05:39:11.438576+0000","last_fullsized":"2024-04-25T05:39:11.438576+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10","last_scrub_stamp":"2024-04-25T05:38:24.406643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:24.406643+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:35.459225+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.907328+0000","last_change":"2024-04-25T05:38:58.078848+0000","last_active":"2024-04-25T05:39:10.907328+0000","last_peered":"2024-04-25T05:39:10.907328+0000","last_clean":"2024-04-25T05:39:10.907328+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:39:10.907328+0000","last_undegraded":"2024-04-25T05:39:10.907328+0000","last_fullsized":"2024-04-25T05:39:10.907328+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.078483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.078483+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.982642+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.035600+0000","last_change":"2024-04-25T05:36:59.627764+0000","last_active":"2024-04-25T05:39:11.035600+0000","last_peered":"2024-04-25T05:39:11.035600+0000","last_clean":"2024-04-25T05:39:11.035600+0000","last_became_active":"2024-04-25T05:36:54.234860+0000","last_became_peered":"2024-04-25T05:36:54.234860+0000","last_unstale":"2024-04-25T05:39:11.035600+0000","last_undegraded":"2024-04-25T05:39:11.035600+0000","last_fullsized":"2024-04-25T05:39:11.035600+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":165,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.039959+0000","last_change":"2024-04-25T05:38:13.301084+0000","last_active":"2024-04-25T05:39:11.039959+0000","last_peered":"2024-04-25T05:39:11.039959+0000","last_clean":"2024-04-25T05:39:11.039959+0000","last_became_active":"2024-04-25T05:38:13.289662+0000","last_became_peered":"2024-04-25T05:38:13.289662+0000","last_unstale":"2024-04-25T05:39:11.039959+0000","last_undegraded":"2024-04-25T05:39:11.039959+0000","last_fullsized":"2024-04-25T05:39:11.039959+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.294192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":356,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901105+0000","last_change":"2024-04-25T05:38:13.255913+0000","last_active":"2024-04-25T05:39:10.901105+0000","last_peered":"2024-04-25T05:39:10.901105+0000","last_clean":"2024-04-25T05:39:10.901105+0000","last_became_active":"2024-04-25T05:38:13.244214+0000","last_became_peered":"2024-04-25T05:38:13.244214+0000","last_unstale":"2024-04-25T05:39:10.901105+0000","last_undegraded":"2024-04-25T05:39:10.901105+0000","last_fullsized":"2024-04-25T05:39:10.901105+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":342,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901079+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:39:10.901079+0000","last_peered":"2024-04-25T05:39:10.901079+0000","last_clean":"2024-04-25T05:39:10.901079+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:39:10.901079+0000","last_undegraded":"2024-04-25T05:39:10.901079+0000","last_fullsized":"2024-04-25T05:39:10.901079+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900864+0000","last_change":"2024-04-25T05:38:57.548615+0000","last_active":"2024-04-25T05:39:10.900864+0000","last_peered":"2024-04-25T05:39:10.900864+0000","last_clean":"2024-04-25T05:39:10.900864+0000","last_became_active":"2024-04-25T05:38:34.758569+0000","last_became_peered":"2024-04-25T05:38:34.758569+0000","last_unstale":"2024-04-25T05:39:10.900864+0000","last_undegraded":"2024-04-25T05:39:10.900864+0000","last_fullsized":"2024-04-25T05:39:10.900864+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.548529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.548529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.774155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":266,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901335+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:39:10.901335+0000","last_peered":"2024-04-25T05:39:10.901335+0000","last_clean":"2024-04-25T05:39:10.901335+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:39:10.901335+0000","last_undegraded":"2024-04-25T05:39:10.901335+0000","last_fullsized":"2024-04-25T05:39:10.901335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.931689+0000","last_change":"2024-04-25T05:38:13.247469+0000","last_active":"2024-04-25T05:39:10.931689+0000","last_peered":"2024-04-25T05:39:10.931689+0000","last_clean":"2024-04-25T05:39:10.931689+0000","last_became_active":"2024-04-25T05:38:13.246750+0000","last_became_peered":"2024-04-25T05:38:13.246750+0000","last_unstale":"2024-04-25T05:39:10.931689+0000","last_undegraded":"2024-04-25T05:39:10.931689+0000","last_fullsized":"2024-04-25T05:39:10.931689+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":359,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900941+0000","last_change":"2024-04-25T05:38:13.326846+0000","last_active":"2024-04-25T05:39:10.900941+0000","last_peered":"2024-04-25T05:39:10.900941+0000","last_clean":"2024-04-25T05:39:10.900941+0000","last_became_active":"2024-04-25T05:38:13.305935+0000","last_became_peered":"2024-04-25T05:38:13.305935+0000","last_unstale":"2024-04-25T05:39:10.900941+0000","last_undegraded":"2024-04-25T05:39:10.900941+0000","last_fullsized":"2024-04-25T05:39:10.900941+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":270,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900705+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:39:10.900705+0000","last_peered":"2024-04-25T05:39:10.900705+0000","last_clean":"2024-04-25T05:39:10.900705+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:39:10.900705+0000","last_undegraded":"2024-04-25T05:39:10.900705+0000","last_fullsized":"2024-04-25T05:39:10.900705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.438225+0000","last_change":"2024-04-25T05:38:57.405872+0000","last_active":"2024-04-25T05:39:11.438225+0000","last_peered":"2024-04-25T05:39:11.438225+0000","last_clean":"2024-04-25T05:39:11.438225+0000","last_became_active":"2024-04-25T05:38:34.758522+0000","last_became_peered":"2024-04-25T05:38:34.758522+0000","last_unstale":"2024-04-25T05:39:11.438225+0000","last_undegraded":"2024-04-25T05:39:11.438225+0000","last_fullsized":"2024-04-25T05:39:11.438225+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:16.857424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439007+0000","last_change":"2024-04-25T05:38:13.284810+0000","last_active":"2024-04-25T05:39:11.439007+0000","last_peered":"2024-04-25T05:39:11.439007+0000","last_clean":"2024-04-25T05:39:11.439007+0000","last_became_active":"2024-04-25T05:38:13.284451+0000","last_became_peered":"2024-04-25T05:38:13.284451+0000","last_unstale":"2024-04-25T05:39:11.439007+0000","last_undegraded":"2024-04-25T05:39:11.439007+0000","last_fullsized":"2024-04-25T05:39:11.439007+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"160'9","reported_seq":352,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901586+0000","last_change":"2024-04-25T05:38:13.244991+0000","last_active":"2024-04-25T05:39:10.901586+0000","last_peered":"2024-04-25T05:39:10.901586+0000","last_clean":"2024-04-25T05:39:10.901586+0000","last_became_active":"2024-04-25T05:38:13.238743+0000","last_became_peered":"2024-04-25T05:38:13.238743+0000","last_unstale":"2024-04-25T05:39:10.901586+0000","last_undegraded":"2024-04-25T05:39:10.901586+0000","last_fullsized":"2024-04-25T05:39:10.901586+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:37:48.604653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:48.604653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900852+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:39:10.900852+0000","last_peered":"2024-04-25T05:39:10.900852+0000","last_clean":"2024-04-25T05:39:10.900852+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:39:10.900852+0000","last_undegraded":"2024-04-25T05:39:10.900852+0000","last_fullsized":"2024-04-25T05:39:10.900852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.041675+0000","last_change":"2024-04-25T05:35:26.270158+0000","last_active":"2024-04-25T05:39:11.041675+0000","last_peered":"2024-04-25T05:39:11.041675+0000","last_clean":"2024-04-25T05:39:11.041675+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:39:11.041675+0000","last_undegraded":"2024-04-25T05:39:11.041675+0000","last_fullsized":"2024-04-25T05:39:11.041675+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-04-25T05:35:26.269986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:26.269986+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":265,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.909768+0000","last_change":"2024-04-25T05:36:54.232549+0000","last_active":"2024-04-25T05:39:10.909768+0000","last_peered":"2024-04-25T05:39:10.909768+0000","last_clean":"2024-04-25T05:39:10.909768+0000","last_became_active":"2024-04-25T05:36:54.231886+0000","last_became_peered":"2024-04-25T05:36:54.231886+0000","last_unstale":"2024-04-25T05:39:10.909768+0000","last_undegraded":"2024-04-25T05:39:10.909768+0000","last_fullsized":"2024-04-25T05:39:10.909768+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.439853+0000","last_change":"2024-04-25T05:37:08.197220+0000","last_active":"2024-04-25T05:39:11.439853+0000","last_peered":"2024-04-25T05:39:11.439853+0000","last_clean":"2024-04-25T05:39:11.439853+0000","last_became_active":"2024-04-25T05:36:54.244909+0000","last_became_peered":"2024-04-25T05:36:54.244909+0000","last_unstale":"2024-04-25T05:39:11.439853+0000","last_undegraded":"2024-04-25T05:39:11.439853+0000","last_fullsized":"2024-04-25T05:39:11.439853+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":278,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900804+0000","last_change":"2024-04-25T05:35:35.729936+0000","last_active":"2024-04-25T05:39:10.900804+0000","last_peered":"2024-04-25T05:39:10.900804+0000","last_clean":"2024-04-25T05:39:10.900804+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:39:10.900804+0000","last_undegraded":"2024-04-25T05:39:10.900804+0000","last_fullsized":"2024-04-25T05:39:10.900804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:35.729825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:35.729825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":325,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901248+0000","last_change":"2024-04-25T05:36:54.269613+0000","last_active":"2024-04-25T05:39:10.901248+0000","last_peered":"2024-04-25T05:39:10.901248+0000","last_clean":"2024-04-25T05:39:10.901248+0000","last_became_active":"2024-04-25T05:36:54.244739+0000","last_became_peered":"2024-04-25T05:36:54.244739+0000","last_unstale":"2024-04-25T05:39:10.901248+0000","last_undegraded":"2024-04-25T05:39:10.901248+0000","last_fullsized":"2024-04-25T05:39:10.901248+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":454,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.437879+0000","last_change":"2024-04-25T05:38:59.374948+0000","last_active":"2024-04-25T05:39:11.437879+0000","last_peered":"2024-04-25T05:39:11.437879+0000","last_clean":"2024-04-25T05:39:11.437879+0000","last_became_active":"2024-04-25T05:38:34.759755+0000","last_became_peered":"2024-04-25T05:38:34.759755+0000","last_unstale":"2024-04-25T05:39:11.437879+0000","last_undegraded":"2024-04-25T05:39:11.437879+0000","last_fullsized":"2024-04-25T05:39:11.437879+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:00.299246+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":274,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900779+0000","last_change":"2024-04-25T05:39:00.794514+0000","last_active":"2024-04-25T05:39:10.900779+0000","last_peered":"2024-04-25T05:39:10.900779+0000","last_clean":"2024-04-25T05:39:10.900779+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:39:10.900779+0000","last_undegraded":"2024-04-25T05:39:10.900779+0000","last_fullsized":"2024-04-25T05:39:10.900779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.794388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.794388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.097831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.040784+0000","last_change":"2024-04-25T05:38:13.289683+0000","last_active":"2024-04-25T05:39:11.040784+0000","last_peered":"2024-04-25T05:39:11.040784+0000","last_clean":"2024-04-25T05:39:11.040784+0000","last_became_active":"2024-04-25T05:38:13.289313+0000","last_became_peered":"2024-04-25T05:38:13.289313+0000","last_unstale":"2024-04-25T05:39:11.040784+0000","last_undegraded":"2024-04-25T05:39:11.040784+0000","last_fullsized":"2024-04-25T05:39:11.040784+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":146,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900753+0000","last_change":"2024-04-25T05:36:54.226814+0000","last_active":"2024-04-25T05:39:10.900753+0000","last_peered":"2024-04-25T05:39:10.900753+0000","last_clean":"2024-04-25T05:39:10.900753+0000","last_became_active":"2024-04-25T05:36:54.226138+0000","last_became_peered":"2024-04-25T05:36:54.226138+0000","last_unstale":"2024-04-25T05:39:10.900753+0000","last_undegraded":"2024-04-25T05:39:10.900753+0000","last_fullsized":"2024-04-25T05:39:10.900753+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:31.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:36:31.976749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901851+0000","last_change":"2024-04-25T05:35:41.640543+0000","last_active":"2024-04-25T05:39:10.901851+0000","last_peered":"2024-04-25T05:39:10.901851+0000","last_clean":"2024-04-25T05:39:10.901851+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:39:10.901851+0000","last_undegraded":"2024-04-25T05:39:10.901851+0000","last_fullsized":"2024-04-25T05:39:10.901851+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:41.640442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.640442+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":585,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:11.034907+0000","last_change":"2024-04-25T05:39:01.087263+0000","last_active":"2024-04-25T05:39:11.034907+0000","last_peered":"2024-04-25T05:39:11.034907+0000","last_clean":"2024-04-25T05:39:11.034907+0000","last_became_active":"2024-04-25T05:38:34.761642+0000","last_became_peered":"2024-04-25T05:38:34.761642+0000","last_unstale":"2024-04-25T05:39:11.034907+0000","last_undegraded":"2024-04-25T05:39:11.034907+0000","last_fullsized":"2024-04-25T05:39:11.034907+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:01.087145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:01.087145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.030738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":266,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.901739+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:39:10.901739+0000","last_peered":"2024-04-25T05:39:10.901739+0000","last_clean":"2024-04-25T05:39:10.901739+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:39:10.901739+0000","last_undegraded":"2024-04-25T05:39:10.901739+0000","last_fullsized":"2024-04-25T05:39:10.901739+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":345,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.907687+0000","last_change":"2024-04-25T05:35:37.678317+0000","last_active":"2024-04-25T05:39:10.907687+0000","last_peered":"2024-04-25T05:39:10.907687+0000","last_clean":"2024-04-25T05:39:10.907687+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:39:10.907687+0000","last_undegraded":"2024-04-25T05:39:10.907687+0000","last_fullsized":"2024-04-25T05:39:10.907687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.678003+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":285,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.900104+0000","last_change":"2024-04-25T05:38:13.286297+0000","last_active":"2024-04-25T05:39:10.900104+0000","last_peered":"2024-04-25T05:39:10.900104+0000","last_clean":"2024-04-25T05:39:10.900104+0000","last_became_active":"2024-04-25T05:38:13.285812+0000","last_became_peered":"2024-04-25T05:38:13.285812+0000","last_unstale":"2024-04-25T05:39:10.900104+0000","last_undegraded":"2024-04-25T05:39:10.900104+0000","last_fullsized":"2024-04-25T05:39:10.900104+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":307,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-04-25T05:39:10.902909+0000","last_change":"2024-04-25T05:36:54.254870+0000","last_active":"2024-04-25T05:39:10.902909+0000","last_peered":"2024-04-25T05:39:10.902909+0000","last_clean":"2024-04-25T05:39:10.902909+0000","last_became_active":"2024-04-25T05:36:54.234440+0000","last_became_peered":"2024-04-25T05:36:54.234440+0000","last_unstale":"2024-04-25T05:39:10.902909+0000","last_undegraded":"2024-04-25T05:39:10.902909+0000","last_fullsized":"2024-04-25T05:39:10.902909+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":304,"num_read_kb":299,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":2838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":13834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070532,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163288,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93581864,"statfs":{"total":95995035648,"available":95827828736,"internally_reserved":0,"allocated":1646592,"data_stored":1026449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217152,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528000,"statfs":{"total":95995035648,"available":95772672000,"internally_reserved":0,"allocated":1359872,"data_stored":698183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203452,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93541700,"statfs":{"total":95995035648,"available":95786700800,"internally_reserved":0,"allocated":1552384,"data_stored":960038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249648,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93495504,"statfs":{"total":95995035648,"available":95739396096,"internally_reserved":0,"allocated":1933312,"data_stored":1288563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986257,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131020,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614132,"statfs":{"total":95995035648,"available":95860871168,"internally_reserved":0,"allocated":1310720,"data_stored":697587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691917,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123268,"kb_used_data":1848,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621884,"statfs":{"total":95995035648,"available":95868809216,"internally_reserved":0,"allocated":1892352,"data_stored":1287579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:39:18.730 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:39:18.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:39:18.730 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-25T05:39:18.730 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T05:39:18.730 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T05:39:18.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-04-25T05:39:18.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:18.815 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:18.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:18.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:18.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:18.942 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:39:18.942 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:39:18.996 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:39:19.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:19.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:19.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:19.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:19.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:19 smithi060 ceph-mon[30898]: pgmap v265: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:39:19.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-25T05:39:19.374 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:19 smithi060 ceph-mon[30898]: osdmap e198: 8 total, 8 up, 6 in 2024-04-25T05:39:19.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:19.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:19.503 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:39:19.503 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T05:39:19.503 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:39:19.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:19.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[27010]: pgmap v265: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:39:19.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-25T05:39:19.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[27010]: osdmap e198: 8 total, 8 up, 6 in 2024-04-25T05:39:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[37492]: pgmap v265: 142 pgs: 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:39:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3667317999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-25T05:39:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-mon[37492]: osdmap e198: 8 total, 8 up, 6 in 2024-04-25T05:39:19.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:19.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:19.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:19.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:19.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:19.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:19.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:20.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:20.033 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:20.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:20.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:20.169 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:39:20.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:20.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:20.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[27010]: osdmap e199: 8 total, 8 up, 6 in 2024-04-25T05:39:20.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:20.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1509966692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[37492]: osdmap e199: 8 total, 8 up, 6 in 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1509966692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:20.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:20.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:20 smithi060 ceph-mon[30898]: osdmap e199: 8 total, 8 up, 6 in 2024-04-25T05:39:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:20 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:20 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1509966692' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:20.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:20.537 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:20.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:20.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:20.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-25T05:39:20.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:20.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:20.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:20.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:20.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:20.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:21.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:21.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:21.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:21.154 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:21.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:21.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[27010]: pgmap v268: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[27010]: osdmap e200: 8 total, 8 up, 6 in 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[37492]: pgmap v268: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:39:21.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:21 smithi043 ceph-mon[37492]: osdmap e200: 8 total, 8 up, 6 in 2024-04-25T05:39:21.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:21 smithi060 ceph-mon[30898]: pgmap v268: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:21 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:39:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:21 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:21 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:21 smithi060 ceph-mon[30898]: osdmap e200: 8 total, 8 up, 6 in 2024-04-25T05:39:21.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:21.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:21.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:39:21.608 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:21.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:21.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:21.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:21.770 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:21.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:21.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:21.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:21.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:22.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:22.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:22.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:22.271 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:22.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:22.445 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:22.477 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:22.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:22.496 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:22.504 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:22.521 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:22.526 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:22.546 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:22.567 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:22.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:22.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:22.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:22.631 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:22.633 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:22.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:22.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:22.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:22.771 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:22.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:22.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:22.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:22.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:23.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:23.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:23.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:23.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:39:23.230 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:23.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:23.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:23.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:23 smithi043 ceph-mon[27010]: pgmap v270: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:23.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/227490564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:23.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:23 smithi043 ceph-mon[37492]: pgmap v270: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:23.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/227490564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:23.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:23.397 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:23.433 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:39:23.433 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:39:23.433 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:23.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:23 smithi060 ceph-mon[30898]: pgmap v270: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:23.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/227490564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:23.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:23.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:23.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:23.648 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:23.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:23.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:23.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:23.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:24.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:24.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:24.037 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:24.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:24.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:24.274 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:24.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:24.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:24.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:24.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:24.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:24.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:24.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.649 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:24.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:24.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:24.769 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-25T05:39:24.777 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:24.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:24.903 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:24.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:25.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:25.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:39:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:25.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:25.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:25.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:25.163 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7] out_osds: [6, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:39:25.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:39:25.164 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-25T05:39:25.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd in 6 2024-04-25T05:39:25.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:25.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:25.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:25.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[27010]: pgmap v271: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[27010]: osdmap e201: 8 total, 8 up, 6 in 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[37492]: pgmap v271: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:39:25.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:25 smithi043 ceph-mon[37492]: osdmap e201: 8 total, 8 up, 6 in 2024-04-25T05:39:25.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:25 smithi060 ceph-mon[30898]: pgmap v271: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:39:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:39:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3002991617' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:39:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:25 smithi060 ceph-mon[30898]: osdmap e201: 8 total, 8 up, 6 in 2024-04-25T05:39:25.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:25.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:25.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:25.654 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:25.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:25.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:25.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:25.905 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:25.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:26.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:26.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:26.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:26.232 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:39:26.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:26.283 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:26.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:26.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2557021168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.500 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:26.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:26.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:26.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:39:26.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2557021168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2557021168' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:39:26.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:26.654 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:26.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:26.782 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:26.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:26.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:26.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:27.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:27.025 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:27.025 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:39:27.025 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:39:27.025 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:39:27.025 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:27.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:27.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:27.158 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:27.214 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-25T05:39:27.217 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-25T05:39:27.217 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-25T05:39:27.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:27.269 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-04-25T05:39:27.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-25T05:39:27.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_104001 2024-04-25T05:39:27.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:39:27.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-04-25T05:39:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:27.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:27.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:27.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:27 smithi060 ceph-mon[30898]: pgmap v273: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 782 B/s rd, 0 op/s 2024-04-25T05:39:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:27 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:39:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:27 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:39:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:27 smithi060 ceph-mon[30898]: osdmap e202: 8 total, 8 up, 6 in 2024-04-25T05:39:27.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:27.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[27010]: pgmap v273: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 782 B/s rd, 0 op/s 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[27010]: osdmap e202: 8 total, 8 up, 6 in 2024-04-25T05:39:27.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[37492]: pgmap v273: 158 pgs: 158 active+clean; 583 KiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 782 B/s rd, 0 op/s 2024-04-25T05:39:27.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:39:27.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:27.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:39:27.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:27 smithi043 ceph-mon[37492]: osdmap e202: 8 total, 8 up, 6 in 2024-04-25T05:39:27.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:27.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:27.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:27.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:27.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:27.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:28.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:28.117 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:28.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:28.258 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:28.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 3551 3535 220.925 220.938 0.00315315 0.00450646 2024-04-25T05:39:28.278 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:28 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:39:28.278 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:28 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:39:28.279 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:28 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:39:28.279 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:28 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:39:28.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:28.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:28.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:28 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:39:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:28 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:39:28.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:28.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:28.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:28.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:28.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:28.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:28.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:28.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:29.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:29.081 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:29.109 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:29.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:39:29.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:29.211 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:29.269 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.6. 2024-04-25T05:39:29.270 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 6428 6412 200.332 179.812 0.0130764 0.0049397 2024-04-25T05:39:29.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:29.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: pgmap v275: 158 pgs: 158 active+clean; 46 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 MiB/s wr, 97 op/s 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:39:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:39:29.114+0000 7f173ec68700 -1 mon.a@0(leader).osd e202 definitely_dead 0 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: pgmap v275: 158 pgs: 158 active+clean; 46 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 MiB/s wr, 97 op/s 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:39:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-25T05:39:29.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:29.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:29.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:29.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:29.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:39:29.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:29.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:29.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: pgmap v275: 158 pgs: 158 active+clean; 46 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 6.0 MiB/s wr, 97 op/s 2024-04-25T05:39:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:39:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:39:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-25T05:39:29.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:29.844 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:29.888 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:29.931 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:29.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:29.951 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:29.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:29.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:29.971 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:29.992 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:30.013 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:30.034 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:30.054 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:30.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:30.075 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:30.077 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:30.077 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:30.091 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:30.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:30.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:30.326 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 6460 6444 134.227 2 0.00187976 0.00497441 2024-04-25T05:39:30.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:30.449 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:30.474 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-25T05:39:30.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-25T05:39:30.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[37492]: osdmap e203: 8 total, 8 up, 7 in 2024-04-25T05:39:30.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[37492]: osdmap e204: 8 total, 8 up, 7 in 2024-04-25T05:39:30.516 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1693403245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:30.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-25T05:39:30.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[27010]: osdmap e203: 8 total, 8 up, 7 in 2024-04-25T05:39:30.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[27010]: osdmap e204: 8 total, 8 up, 7 in 2024-04-25T05:39:30.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1693403245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:30.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:30.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:30.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1542794856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-25T05:39:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:30 smithi060 ceph-mon[30898]: osdmap e203: 8 total, 8 up, 7 in 2024-04-25T05:39:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:30 smithi060 ceph-mon[30898]: osdmap e204: 8 total, 8 up, 7 in 2024-04-25T05:39:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1693403245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:30.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:30.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:30.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:30.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:30.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:31.030 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:31.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.095 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:31.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:31.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:31.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:31.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 7569 7553 117.998 69.3125 0.00580955 0.00846142 2024-04-25T05:39:31.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:31.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:31.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:31.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: pgmap v277: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 24 MiB/s wr, 379 op/s 2024-04-25T05:39:31.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:39:31.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:39:31.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:39:31.506 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[27010]: osdmap e205: 8 total, 8 up, 7 in 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: pgmap v277: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 24 MiB/s wr, 379 op/s 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:39:31.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:31 smithi043 ceph-mon[37492]: osdmap e205: 8 total, 8 up, 7 in 2024-04-25T05:39:31.507 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:31.593 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:31.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:31.637 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:31.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:31.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: pgmap v277: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 24 MiB/s wr, 379 op/s 2024-04-25T05:39:31.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:39:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:39:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:39:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:39:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:31 smithi060 ceph-mon[30898]: osdmap e205: 8 total, 8 up, 7 in 2024-04-25T05:39:31.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:31.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:31.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:31.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:32.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:32.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:32.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:32.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:32.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 9559 9543 119.271 124.375 0.00765566 0.0083703 2024-04-25T05:39:32.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: osdmap e206: 8 total, 8 up, 7 in 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: osdmap e206: 8 total, 8 up, 7 in 2024-04-25T05:39:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:32.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:32.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:32.557 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:39:32.558 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:39:32.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:32.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:32.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:39:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:39:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: osdmap e206: 8 total, 8 up, 7 in 2024-04-25T05:39:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:39:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:32.756 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:32.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:32.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:32.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:32.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:33.027 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:33.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:33.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:33.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:33.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:33.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 11456 11440 119.151 118.562 0.00358404 0.0083845 2024-04-25T05:39:33.297 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:33.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:33.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:39:33.427 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:33.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:33.559 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:33.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:39:33.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: pgmap v280: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 927 B/s rd, 26 MiB/s wr, 412 op/s 2024-04-25T05:39:33.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-04-25T05:39:33.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:39:33.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: osdmap e207: 8 total, 8 up, 7 in 2024-04-25T05:39:33.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3461299648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.614 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: pgmap v280: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 927 B/s rd, 26 MiB/s wr, 412 op/s 2024-04-25T05:39:33.614 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-04-25T05:39:33.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:39:33.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: osdmap e207: 8 total, 8 up, 7 in 2024-04-25T05:39:33.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3461299648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:33.736 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:33.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: pgmap v280: 158 pgs: 158 active+clean; 143 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 927 B/s rd, 26 MiB/s wr, 412 op/s 2024-04-25T05:39:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-04-25T05:39:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:39:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: osdmap e207: 8 total, 8 up, 7 in 2024-04-25T05:39:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3461299648' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:39:33.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:33.860 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:33.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:33.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:33.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:34.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:34.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:34.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:34.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 12609 12593 112.424 72.0625 0.00192338 0.00831424 2024-04-25T05:39:34.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:34.357 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:34.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:34.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:34.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[27010]: osdmap e208: 8 total, 8 up, 7 in 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:39:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:34 smithi043 ceph-mon[37492]: osdmap e208: 8 total, 8 up, 7 in 2024-04-25T05:39:34.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:34.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:34 smithi060 ceph-mon[30898]: osdmap e208: 8 total, 8 up, 7 in 2024-04-25T05:39:34.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:34.861 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:34.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:34.984 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:34.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:35.031 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:35.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:35.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:35.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:35.229 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:35.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 12823 12807 100.042 13.375 0.00158134 0.0094465 2024-04-25T05:39:35.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:35.354 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:35.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:35.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:39:35.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:39:35.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: pgmap v283: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:35.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: Health check failed: Degraded data redundancy: 1/33063 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: 4.d scrub starts 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: 4.d scrub ok 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[27010]: osdmap e209: 8 total, 8 up, 7 in 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: pgmap v283: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: Health check failed: Degraded data redundancy: 1/33063 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: 4.d scrub starts 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: 4.d scrub ok 2024-04-25T05:39:35.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:35 smithi043 ceph-mon[37492]: osdmap e209: 8 total, 8 up, 7 in 2024-04-25T05:39:35.475 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:39:35.475 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:39:35.475 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T05:39:35.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:39:35.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:35.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:35.590 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:39:35.590 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:39:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:35.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:35.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: pgmap v283: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: Health check failed: Degraded data redundancy: 1/33063 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: 4.d scrub starts 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: 4.d scrub ok 2024-04-25T05:39:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:35 smithi060 ceph-mon[30898]: osdmap e209: 8 total, 8 up, 7 in 2024-04-25T05:39:35.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:35.865 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:35.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:35.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:36.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:36.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:36.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:36.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 14258 14242 98.8912 89.6875 0.00260239 0.0101009 2024-04-25T05:39:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.274 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:36.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:36.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:36.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:36.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:36.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 3.6 scrub starts 2024-04-25T05:39:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 3.6 scrub ok 2024-04-25T05:39:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: osdmap e210: 8 total, 8 up, 7 in 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 3.6 scrub starts 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 3.6 scrub ok 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: osdmap e210: 8 total, 8 up, 7 in 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:39:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:36 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:39:36.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:36.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:36.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 3.6 scrub starts 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 3.6 scrub ok 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: osdmap e210: 8 total, 8 up, 7 in 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:39:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:36 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:39:36.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:36.818 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:36.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:36.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:36.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:36.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:39:37.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:37.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:37.212 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:37.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 16692 16676 104.213 152.125 0.00378154 0.00958834 2024-04-25T05:39:37.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:37.346 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:37.427 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:37.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:39:37.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:37.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:37.519 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:37.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[37492]: pgmap v286: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:37.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:39:37.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:39:37.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[27010]: pgmap v286: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:37.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:39:37.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:37 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:39:37.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:37.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:37.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:37 smithi060 ceph-mon[30898]: pgmap v286: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 peering, 1 active+recovering+undersized+degraded+remapped, 148 active+clean; 675 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 1/33063 objects degraded (0.003%); 932/33063 objects misplaced (2.819%); 18 MiB/s, 295 objects/s recovering 2024-04-25T05:39:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:37 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:39:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:37 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:39:37.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:37.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:37.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:37.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:38.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.062 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:38.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:38.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:38.271 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 19009 18993 107.903 144.812 0.00568959 0.00925909 2024-04-25T05:39:38.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:38.336 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:38.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:38.470 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:39:38.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:39:38.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:39:38.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:39:38.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:39:38.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:39:38.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:38 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:39:38.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:38.594 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:38.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:38.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:38.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:39:38.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:39:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:39:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:39:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:39:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:38 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:39:38.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:38.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:38.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:38.900 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:38.939 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:38.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:38.992 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:39.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:39.046 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:39.047 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:39.092 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:39.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:39.143 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:39.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:39.185 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:39.232 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:39.239 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:39.240 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:39.241 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:39.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:39.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 15 21285 21270 110.766 142.312 0.00514245 0.00902136 2024-04-25T05:39:39.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:39.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:39.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: pgmap v288: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 151 active+clean; 793 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 1/38721 objects degraded (0.003%); 932/38721 objects misplaced (2.407%); 14 MiB/s, 229 objects/s recovering 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:39:39.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/931372615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: pgmap v288: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 151 active+clean; 793 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 1/38721 objects degraded (0.003%); 932/38721 objects misplaced (2.407%); 14 MiB/s, 229 objects/s recovering 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:39:39.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/931372615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 210, 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:39:35.767967+0000", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:39:29.246791+0000", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:39:39.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 8, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:39:39.575 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.576 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:39:39.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "174", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "171", 2024-04-25T05:39:39.578 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.579 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.580 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:39:39.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.1700000762939453, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.1700000762939453, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:39:39.582 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:39:39.584 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.585 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.586 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.587 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:39:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:39:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.590 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.591 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.592 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.593 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.595 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.2400000095367432, 2024-04-25T05:39:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.2400000095367432, 2024-04-25T05:39:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.0900000333786011, 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.0900000333786011, 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:39:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.620 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.621 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.622 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.624 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.625 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9900000095367432, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9900000095367432, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.626 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 8, 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_1", 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:39:18.082095+0000", 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:39:39.627 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:39:39.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "201", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:39:39.629 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:39:39.630 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:39:39.631 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:39:39.632 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9900000095367432, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9900000095367432, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 1, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:39:39.633 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 157, 2024-04-25T05:39:39.634 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 209, 2024-04-25T05:39:39.638 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:39:39.641 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.642 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 207, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:39:39.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:39:39.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 207, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.647 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.648 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:39:39.650 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.652 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.654 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 207, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:39:39.657 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:39:39.660 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.661 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 206, 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:39:39.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:39:39.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:39:39.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.665 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 209, 2024-04-25T05:39:39.666 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:39:39.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 206, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:39:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:39:39.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:38:13.807176+0000", 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:01.339561+0000", 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.675 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:28.355579+0000", 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:38:22.862263+0000", 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:01.871868+0000", 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:39:39.677 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:38:54.226354+0000", 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:39:39.678 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:19.229076+0000", 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:39:39.679 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:38:56.231278+0000", 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.2", 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.9", 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.12", 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.682 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:39:39.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [], 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:39:39.684 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:39:39.685 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:39:39.686 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:39:39.687 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:39.687 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:39.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: pgmap v288: 158 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 151 active+clean; 793 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 1/38721 objects degraded (0.003%); 932/38721 objects misplaced (2.407%); 14 MiB/s, 229 objects/s recovering 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:39:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/931372615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:39.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:39.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:39.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:39.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:40.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:40.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:40.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 23534 23518 113.052 140.5 0.00460055 0.00883969 2024-04-25T05:39:40.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:40.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:40.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:40.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:40.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: 6.c scrub starts 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: 6.c scrub ok 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/893563101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:39:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: Health check update: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[27010]: osdmap e211: 8 total, 8 up, 7 in 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: 6.c scrub starts 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: 6.c scrub ok 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/893563101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: Health check update: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:40.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:40 smithi043 ceph-mon[37492]: osdmap e211: 8 total, 8 up, 7 in 2024-04-25T05:39:40.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:40.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:40.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: 6.c scrub starts 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: 6.c scrub ok 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/893563101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: Health check update: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-04-25T05:39:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:40 smithi060 ceph-mon[30898]: osdmap e211: 8 total, 8 up, 7 in 2024-04-25T05:39:40.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:40.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:40.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:40.951 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:40.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:41.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:41.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:41.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:41.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:41.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 15 25490 25475 113.713 122.312 0.00910235 0.00878803 2024-04-25T05:39:41.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:41.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.488 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:41.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:41.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:41.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 4.f deep-scrub starts 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 4.f deep-scrub ok 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: pgmap v289: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 9.7 MiB/s, 157 objects/s recovering 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded) 2024-04-25T05:39:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 4.f deep-scrub starts 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 4.f deep-scrub ok 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: pgmap v289: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 9.7 MiB/s, 157 objects/s recovering 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded) 2024-04-25T05:39:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:39:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:39:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:39:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:39:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:41 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:39:41.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:41.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 4.f deep-scrub starts 2024-04-25T05:39:41.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 4.f deep-scrub ok 2024-04-25T05:39:41.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: pgmap v289: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s; 9.7 MiB/s, 157 objects/s recovering 2024-04-25T05:39:41.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:39:41.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/38721 objects degraded (0.003%), 1 pg degraded) 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:39:41.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:41 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:39:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:41.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:41.889 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:41.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:41.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:42.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:42.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:42.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:42.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:42.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 15 27845 27830 115.943 147.188 0.00566518 0.00861914 2024-04-25T05:39:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:42.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:42.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:42.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 4.19 scrub starts 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 4.19 scrub ok 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:39:42.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 4.19 scrub starts 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 4.19 scrub ok 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:39:42.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:42 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:39:42.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:42.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:42.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:39:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:39:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 4.19 scrub starts 2024-04-25T05:39:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 4.19 scrub ok 2024-04-25T05:39:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:39:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:39:42.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:42.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:42.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:42.992 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:42.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:43.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:43.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:43.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:43.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:43.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:43.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 30043 30027 117.278 137.312 0.00841334 0.0085218 2024-04-25T05:39:43.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: 4.b deep-scrub starts 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: 4.b deep-scrub ok 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: pgmap v291: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 731 B/s rd, 66 MiB/s wr, 1.05k op/s; 8.3 MiB/s, 134 objects/s recovering 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: 4.18 deep-scrub starts 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: 4.18 deep-scrub ok 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: 4.b deep-scrub starts 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: 4.b deep-scrub ok 2024-04-25T05:39:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: pgmap v291: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 731 B/s rd, 66 MiB/s wr, 1.05k op/s; 8.3 MiB/s, 134 objects/s recovering 2024-04-25T05:39:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: 4.18 deep-scrub starts 2024-04-25T05:39:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: 4.18 deep-scrub ok 2024-04-25T05:39:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:43.404 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:43.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:43.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:43.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:43.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: 4.b deep-scrub starts 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: 4.b deep-scrub ok 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: pgmap v291: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 731 B/s rd, 66 MiB/s wr, 1.05k op/s; 8.3 MiB/s, 134 objects/s recovering 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: 4.18 deep-scrub starts 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: 4.18 deep-scrub ok 2024-04-25T05:39:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:43.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:43.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:43.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:43.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:43.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:43.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:44.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:44.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:44.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:44.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 31950 31934 117.39 119.188 0.0023965 0.00848588 2024-04-25T05:39:44.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:44.344 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:44.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:44.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:44.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:44.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 4.4 scrub starts 2024-04-25T05:39:44.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 4.4 scrub ok 2024-04-25T05:39:44.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:39:44.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[27010]: pgmap v292: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 7.3 MiB/s, 118 objects/s recovering 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 4.4 scrub starts 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 4.4 scrub ok 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:44.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:44 smithi043 ceph-mon[37492]: pgmap v292: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 7.3 MiB/s, 118 objects/s recovering 2024-04-25T05:39:44.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:44.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 4.4 scrub starts 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 4.4 scrub ok 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:39:44.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:44 smithi060 ceph-mon[30898]: pgmap v292: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 7.3 MiB/s, 118 objects/s recovering 2024-04-25T05:39:44.671 INFO:tasks.thrashosds.thrasher:j is {'epoch': 210, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:39:35.767967+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:39:29.246791+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '171', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.2400000095367432, 'score_stable': 1.2400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.090000033378601, 'raw_score_stable': 1.090000033378601, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-25T05:39:18.082095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 157, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:38:13.807176+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:01.339561+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:28.355579+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:38:22.862263+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:01.871868+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:38:54.226354+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:19.229076+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:38:56.231278+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:39:44.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:39:44.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:44.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:39:44.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:44.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:44.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:45.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:45.062 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:45.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:45.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.249 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:45.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:45.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 15 33872 33857 117.545 120.188 0.0101828 0.00850158 2024-04-25T05:39:45.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:45.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:45.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:45.520 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:45.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: osdmap e212: 8 total, 8 up, 7 in 2024-04-25T05:39:45.639 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[27010]: osdmap e213: 8 total, 8 up, 7 in 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: osdmap e212: 8 total, 8 up, 7 in 2024-04-25T05:39:45.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:45 smithi043 ceph-mon[37492]: osdmap e213: 8 total, 8 up, 7 in 2024-04-25T05:39:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:45.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:45.737 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:45.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:39:45.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/738097911' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: osdmap e212: 8 total, 8 up, 7 in 2024-04-25T05:39:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:45 smithi060 ceph-mon[30898]: osdmap e213: 8 total, 8 up, 7 in 2024-04-25T05:39:45.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:45.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:45.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:45.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:45.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:46.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:46.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:46.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:46.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:46.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 35979 35963 118.285 131.625 0.00345487 0.00844914 2024-04-25T05:39:46.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.338 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:46.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:46.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:46.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:46.517 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:46.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:46.615 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:46.615 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:39:46.615 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:39:46.615 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:39:46.615 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:46 smithi043 ceph-mon[27010]: pgmap v295: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-04-25T05:39:46.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:46.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:46.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:46 smithi043 ceph-mon[37492]: pgmap v295: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-04-25T05:39:46.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:46.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:46 smithi060 ceph-mon[30898]: pgmap v295: 158 pgs: 158 active+clean; 1.8 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-04-25T05:39:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:46.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:46.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:46.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:46.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:46.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:47.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:47.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:47.164 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:47.208 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:47.209 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:47.210 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:47.210 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:47.210 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:47.210 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:47.210 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:47.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:47.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:39:47.273868+0000 min lat: 0.00119975 max lat: 1.57603 avg lat: 0.00844637 2024-04-25T05:39:47.272 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:39:47.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 37878 37862 118.304 118.688 0.00464962 0.00844637 2024-04-25T05:39:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:47.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:47.432 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:47.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:47.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2587331138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:47.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:47.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2587331138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:47.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:47.564 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:47.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:47.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:39:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2587331138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:39:47.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:47.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:47.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:47.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:47.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:48.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:48.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.250 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:48.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:48.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 39920 39904 118.748 127.625 0.00930367 0.00841584 2024-04-25T05:39:48.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:48.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:48.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:48.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:48.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:48 smithi043 ceph-mon[27010]: pgmap v296: 158 pgs: 158 active+clean; 1.9 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-25T05:39:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:48 smithi043 ceph-mon[37492]: pgmap v296: 158 pgs: 158 active+clean; 1.9 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-25T05:39:48.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:48.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:48 smithi060 ceph-mon[30898]: pgmap v296: 158 pgs: 158 active+clean; 1.9 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-25T05:39:48.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:48.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:48.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:48.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:48.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:49.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:49.056 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:49.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:49.192 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:49.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 15 41784 41769 118.648 116.562 0.0071253 0.00842279 2024-04-25T05:39:49.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:49.327 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:49.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:49.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:49.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:49.593 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:49.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:49.725 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:49.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:49.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:49.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:49.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:50.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:50.012 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:39:50.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:50.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:50.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:50.272 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:50.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 43377 43361 117.815 99.5 0.00817316 0.00848241 2024-04-25T05:39:50.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:50.407 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:50.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:50.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:50.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:50.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:50.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:50.792 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:39:50.796 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:39:50.830 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:50.868 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[27010]: osdmap e214: 8 total, 8 up, 7 in 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[27010]: pgmap v298: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[27010]: osdmap e215: 8 total, 8 up, 7 in 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[37492]: osdmap e214: 8 total, 8 up, 7 in 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[37492]: pgmap v298: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:39:50.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:50 smithi043 ceph-mon[37492]: osdmap e215: 8 total, 8 up, 7 in 2024-04-25T05:39:50.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:39:50.931 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:50.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:50.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:50.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:50 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:50.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:50 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:50 smithi060 ceph-mon[30898]: osdmap e214: 8 total, 8 up, 7 in 2024-04-25T05:39:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:50 smithi060 ceph-mon[30898]: pgmap v298: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:39:50.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:50 smithi060 ceph-mon[30898]: osdmap e215: 8 total, 8 up, 7 in 2024-04-25T05:39:51.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:51.121 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:51.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:51.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:51.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 45168 45152 117.57 111.938 0.004726 0.00850105 2024-04-25T05:39:51.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:39:51.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:51.399 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:51.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.532 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:51.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:51.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:51.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:51.760 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-25T05:39:49.782305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2569673243,"num_objects":39431,"num_object_clones":0,"num_object_copies":118293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39431,"num_whiteouts":0,"num_read":1342,"num_read_kb":1062,"num_write":39829,"num_write_kb":2508909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":134880358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41347,"ondisk_log_size":41347,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8796664,"kb_used_data":7370624,"kb_used_omap":10,"kb_used_meta":1425973,"kb_avail":647419400,"statfs":{"total":671965249536,"available":662957465600,"internally_reserved":0,"allocated":7547518976,"data_stored":7542632478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1460196459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":648675328,"num_objects":9898,"num_object_clones":0,"num_object_copies":29694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9898,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9898,"num_write_kb":633472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.441425"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788096+0000","last_change":"2024-04-25T05:37:49.635619+0000","last_active":"2024-04-25T05:39:44.788096+0000","last_peered":"2024-04-25T05:39:44.788096+0000","last_clean":"2024-04-25T05:39:44.788096+0000","last_became_active":"2024-04-25T05:34:04.720099+0000","last_became_peered":"2024-04-25T05:34:04.720099+0000","last_unstale":"2024-04-25T05:39:44.788096+0000","last_undegraded":"2024-04-25T05:39:44.788096+0000","last_fullsized":"2024-04-25T05:39:44.788096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:49.635538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:49.635538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786099+0000","last_change":"2024-04-25T05:39:08.797448+0000","last_active":"2024-04-25T05:39:44.786099+0000","last_peered":"2024-04-25T05:39:44.786099+0000","last_clean":"2024-04-25T05:39:44.786099+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:39:44.786099+0000","last_undegraded":"2024-04-25T05:39:44.786099+0000","last_fullsized":"2024-04-25T05:39:44.786099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:08.797334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.797334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.918004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":132,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782677+0000","last_change":"2024-04-25T05:39:29.795039+0000","last_active":"2024-04-25T05:39:44.782677+0000","last_peered":"2024-04-25T05:39:44.782677+0000","last_clean":"2024-04-25T05:39:44.782677+0000","last_became_active":"2024-04-25T05:39:29.792781+0000","last_became_peered":"2024-04-25T05:39:29.792781+0000","last_unstale":"2024-04-25T05:39:44.782677+0000","last_undegraded":"2024-04-25T05:39:44.782677+0000","last_fullsized":"2024-04-25T05:39:44.782677+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:07.672941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":385,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786541+0000","last_change":"2024-04-25T05:39:42.695596+0000","last_active":"2024-04-25T05:39:44.786541+0000","last_peered":"2024-04-25T05:39:44.786541+0000","last_clean":"2024-04-25T05:39:44.786541+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:39:44.786541+0000","last_undegraded":"2024-04-25T05:39:44.786541+0000","last_fullsized":"2024-04-25T05:39:44.786541+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.695422+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:55.591355+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793183+0000","last_change":"2024-04-25T05:39:29.799097+0000","last_active":"2024-04-25T05:39:44.793183+0000","last_peered":"2024-04-25T05:39:44.793183+0000","last_clean":"2024-04-25T05:39:44.793183+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:39:44.793183+0000","last_undegraded":"2024-04-25T05:39:44.793183+0000","last_fullsized":"2024-04-25T05:39:44.793183+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.118690+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786669+0000","last_change":"2024-04-25T05:39:29.787297+0000","last_active":"2024-04-25T05:39:44.786669+0000","last_peered":"2024-04-25T05:39:44.786669+0000","last_clean":"2024-04-25T05:39:44.786669+0000","last_became_active":"2024-04-25T05:39:29.786329+0000","last_became_peered":"2024-04-25T05:39:29.786329+0000","last_unstale":"2024-04-25T05:39:44.786669+0000","last_undegraded":"2024-04-25T05:39:44.786669+0000","last_fullsized":"2024-04-25T05:39:44.786669+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:47.617351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:47.617351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.792071+0000","last_change":"2024-04-25T05:39:29.797215+0000","last_active":"2024-04-25T05:39:44.792071+0000","last_peered":"2024-04-25T05:39:44.792071+0000","last_clean":"2024-04-25T05:39:44.792071+0000","last_became_active":"2024-04-25T05:39:29.796847+0000","last_became_peered":"2024-04-25T05:39:29.796847+0000","last_unstale":"2024-04-25T05:39:44.792071+0000","last_undegraded":"2024-04-25T05:39:44.792071+0000","last_fullsized":"2024-04-25T05:39:44.792071+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":412,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.791561+0000","last_change":"2024-04-25T05:39:41.220106+0000","last_active":"2024-04-25T05:39:44.791561+0000","last_peered":"2024-04-25T05:39:44.791561+0000","last_clean":"2024-04-25T05:39:44.791561+0000","last_became_active":"2024-04-25T05:38:13.246798+0000","last_became_peered":"2024-04-25T05:38:13.246798+0000","last_unstale":"2024-04-25T05:39:44.791561+0000","last_undegraded":"2024-04-25T05:39:44.791561+0000","last_fullsized":"2024-04-25T05:39:44.791561+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:39:41.219950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.219950+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.988927+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":126,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.789733+0000","last_change":"2024-04-25T05:38:59.924478+0000","last_active":"2024-04-25T05:39:44.789733+0000","last_peered":"2024-04-25T05:39:44.789733+0000","last_clean":"2024-04-25T05:39:44.789733+0000","last_became_active":"2024-04-25T05:38:13.304062+0000","last_became_peered":"2024-04-25T05:38:13.304062+0000","last_unstale":"2024-04-25T05:39:44.789733+0000","last_undegraded":"2024-04-25T05:39:44.789733+0000","last_fullsized":"2024-04-25T05:39:44.789733+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:02.578128+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":311,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783704+0000","last_change":"2024-04-25T05:38:13.239184+0000","last_active":"2024-04-25T05:39:44.783704+0000","last_peered":"2024-04-25T05:39:44.783704+0000","last_clean":"2024-04-25T05:39:44.783704+0000","last_became_active":"2024-04-25T05:38:13.237479+0000","last_became_peered":"2024-04-25T05:38:13.237479+0000","last_unstale":"2024-04-25T05:39:44.783704+0000","last_undegraded":"2024-04-25T05:39:44.783704+0000","last_fullsized":"2024-04-25T05:39:44.783704+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.981841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.981841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":341,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783282+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:39:44.783282+0000","last_peered":"2024-04-25T05:39:44.783282+0000","last_clean":"2024-04-25T05:39:44.783282+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:39:44.783282+0000","last_undegraded":"2024-04-25T05:39:44.783282+0000","last_fullsized":"2024-04-25T05:39:44.783282+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:42.090484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":369,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788263+0000","last_change":"2024-04-25T05:38:13.254897+0000","last_active":"2024-04-25T05:39:44.788263+0000","last_peered":"2024-04-25T05:39:44.788263+0000","last_clean":"2024-04-25T05:39:44.788263+0000","last_became_active":"2024-04-25T05:38:13.239434+0000","last_became_peered":"2024-04-25T05:38:13.239434+0000","last_unstale":"2024-04-25T05:39:44.788263+0000","last_undegraded":"2024-04-25T05:39:44.788263+0000","last_fullsized":"2024-04-25T05:39:44.788263+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":309,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781965+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:39:44.781965+0000","last_peered":"2024-04-25T05:39:44.781965+0000","last_clean":"2024-04-25T05:39:44.781965+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:39:44.781965+0000","last_undegraded":"2024-04-25T05:39:44.781965+0000","last_fullsized":"2024-04-25T05:39:44.781965+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779290+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:39:44.779290+0000","last_peered":"2024-04-25T05:39:44.779290+0000","last_clean":"2024-04-25T05:39:44.779290+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:39:44.779290+0000","last_undegraded":"2024-04-25T05:39:44.779290+0000","last_fullsized":"2024-04-25T05:39:44.779290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783125+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:39:44.783125+0000","last_peered":"2024-04-25T05:39:44.783125+0000","last_clean":"2024-04-25T05:39:44.783125+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:39:44.783125+0000","last_undegraded":"2024-04-25T05:39:44.783125+0000","last_fullsized":"2024-04-25T05:39:44.783125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":344,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787660+0000","last_change":"2024-04-25T05:36:20.720747+0000","last_active":"2024-04-25T05:39:44.787660+0000","last_peered":"2024-04-25T05:39:44.787660+0000","last_clean":"2024-04-25T05:39:44.787660+0000","last_became_active":"2024-04-25T05:35:32.891852+0000","last_became_peered":"2024-04-25T05:35:32.891852+0000","last_unstale":"2024-04-25T05:39:44.787660+0000","last_undegraded":"2024-04-25T05:39:44.787660+0000","last_fullsized":"2024-04-25T05:39:44.787660+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":312,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787711+0000","last_change":"2024-04-25T05:39:29.788548+0000","last_active":"2024-04-25T05:39:44.787711+0000","last_peered":"2024-04-25T05:39:44.787711+0000","last_clean":"2024-04-25T05:39:44.787711+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:39:44.787711+0000","last_undegraded":"2024-04-25T05:39:44.787711+0000","last_fullsized":"2024-04-25T05:39:44.787711+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.699868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.699868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786477+0000","last_change":"2024-04-25T05:38:13.285789+0000","last_active":"2024-04-25T05:39:44.786477+0000","last_peered":"2024-04-25T05:39:44.786477+0000","last_clean":"2024-04-25T05:39:44.786477+0000","last_became_active":"2024-04-25T05:38:13.285434+0000","last_became_peered":"2024-04-25T05:38:13.285434+0000","last_unstale":"2024-04-25T05:39:44.786477+0000","last_undegraded":"2024-04-25T05:39:44.786477+0000","last_fullsized":"2024-04-25T05:39:44.786477+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":118,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787572+0000","last_change":"2024-04-25T05:38:13.289898+0000","last_active":"2024-04-25T05:39:44.787572+0000","last_peered":"2024-04-25T05:39:44.787572+0000","last_clean":"2024-04-25T05:39:44.787572+0000","last_became_active":"2024-04-25T05:38:13.289539+0000","last_became_peered":"2024-04-25T05:38:13.289539+0000","last_unstale":"2024-04-25T05:39:44.787572+0000","last_undegraded":"2024-04-25T05:39:44.787572+0000","last_fullsized":"2024-04-25T05:39:44.787572+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.198247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.198247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":356,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783298+0000","last_change":"2024-04-25T05:39:42.757071+0000","last_active":"2024-04-25T05:39:44.783298+0000","last_peered":"2024-04-25T05:39:44.783298+0000","last_clean":"2024-04-25T05:39:44.783298+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:39:44.783298+0000","last_undegraded":"2024-04-25T05:39:44.783298+0000","last_fullsized":"2024-04-25T05:39:44.783298+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:39:42.756928+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.756928+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.676365+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.795277+0000","last_change":"2024-04-25T05:39:29.799140+0000","last_active":"2024-04-25T05:39:44.795277+0000","last_peered":"2024-04-25T05:39:44.795277+0000","last_clean":"2024-04-25T05:39:44.795277+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:39:44.795277+0000","last_undegraded":"2024-04-25T05:39:44.795277+0000","last_fullsized":"2024-04-25T05:39:44.795277+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.963090+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787434+0000","last_change":"2024-04-25T05:38:54.633069+0000","last_active":"2024-04-25T05:39:44.787434+0000","last_peered":"2024-04-25T05:39:44.787434+0000","last_clean":"2024-04-25T05:39:44.787434+0000","last_became_active":"2024-04-25T05:34:02.706283+0000","last_became_peered":"2024-04-25T05:34:02.706283+0000","last_unstale":"2024-04-25T05:39:44.787434+0000","last_undegraded":"2024-04-25T05:39:44.787434+0000","last_fullsized":"2024-04-25T05:39:44.787434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.565126+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":312,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788291+0000","last_change":"2024-04-25T05:36:26.665143+0000","last_active":"2024-04-25T05:39:44.788291+0000","last_peered":"2024-04-25T05:39:44.788291+0000","last_clean":"2024-04-25T05:39:44.788291+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:39:44.788291+0000","last_undegraded":"2024-04-25T05:39:44.788291+0000","last_fullsized":"2024-04-25T05:39:44.788291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.664875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.664875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.791674+0000","last_change":"2024-04-25T05:34:00.685073+0000","last_active":"2024-04-25T05:39:44.791674+0000","last_peered":"2024-04-25T05:39:44.791674+0000","last_clean":"2024-04-25T05:39:44.791674+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:39:44.791674+0000","last_undegraded":"2024-04-25T05:39:44.791674+0000","last_fullsized":"2024-04-25T05:39:44.791674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:33:59.667161+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":310,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781530+0000","last_change":"2024-04-25T05:36:53.608332+0000","last_active":"2024-04-25T05:39:44.781530+0000","last_peered":"2024-04-25T05:39:44.781530+0000","last_clean":"2024-04-25T05:39:44.781530+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:39:44.781530+0000","last_undegraded":"2024-04-25T05:39:44.781530+0000","last_fullsized":"2024-04-25T05:39:44.781530+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:36:53.608208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:53.608208+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":44,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782717+0000","last_change":"2024-04-25T05:39:38.521412+0000","last_active":"2024-04-25T05:39:44.782717+0000","last_peered":"2024-04-25T05:39:44.782717+0000","last_clean":"2024-04-25T05:39:44.782717+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:39:44.782717+0000","last_undegraded":"2024-04-25T05:39:44.782717+0000","last_fullsized":"2024-04-25T05:39:44.782717+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:39:38.521265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.521265+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.573956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786006+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:39:44.786006+0000","last_peered":"2024-04-25T05:39:44.786006+0000","last_clean":"2024-04-25T05:39:44.786006+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:39:44.786006+0000","last_undegraded":"2024-04-25T05:39:44.786006+0000","last_fullsized":"2024-04-25T05:39:44.786006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":192,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779687+0000","last_change":"2024-04-25T05:39:02.184665+0000","last_active":"2024-04-25T05:39:44.779687+0000","last_peered":"2024-04-25T05:39:44.779687+0000","last_clean":"2024-04-25T05:39:44.779687+0000","last_became_active":"2024-04-25T05:38:13.281967+0000","last_became_peered":"2024-04-25T05:38:13.281967+0000","last_unstale":"2024-04-25T05:39:44.779687+0000","last_undegraded":"2024-04-25T05:39:44.779687+0000","last_fullsized":"2024-04-25T05:39:44.779687+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.101025+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":301,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786172+0000","last_change":"2024-04-25T05:35:37.289544+0000","last_active":"2024-04-25T05:39:44.786172+0000","last_peered":"2024-04-25T05:39:44.786172+0000","last_clean":"2024-04-25T05:39:44.786172+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:39:44.786172+0000","last_undegraded":"2024-04-25T05:39:44.786172+0000","last_fullsized":"2024-04-25T05:39:44.786172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:37.289331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:37.289331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":126,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779543+0000","last_change":"2024-04-25T05:39:04.108082+0000","last_active":"2024-04-25T05:39:44.779543+0000","last_peered":"2024-04-25T05:39:44.779543+0000","last_clean":"2024-04-25T05:39:44.779543+0000","last_became_active":"2024-04-25T05:38:13.282204+0000","last_became_peered":"2024-04-25T05:38:13.282204+0000","last_unstale":"2024-04-25T05:39:44.779543+0000","last_undegraded":"2024-04-25T05:39:44.779543+0000","last_fullsized":"2024-04-25T05:39:44.779543+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.765638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788138+0000","last_change":"2024-04-25T05:38:13.305988+0000","last_active":"2024-04-25T05:39:44.788138+0000","last_peered":"2024-04-25T05:39:44.788138+0000","last_clean":"2024-04-25T05:39:44.788138+0000","last_became_active":"2024-04-25T05:38:13.305466+0000","last_became_peered":"2024-04-25T05:38:13.305466+0000","last_unstale":"2024-04-25T05:39:44.788138+0000","last_undegraded":"2024-04-25T05:39:44.788138+0000","last_fullsized":"2024-04-25T05:39:44.788138+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":51,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782932+0000","last_change":"2024-04-25T05:39:40.474408+0000","last_active":"2024-04-25T05:39:44.782932+0000","last_peered":"2024-04-25T05:39:44.782932+0000","last_clean":"2024-04-25T05:39:44.782932+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:39:44.782932+0000","last_undegraded":"2024-04-25T05:39:44.782932+0000","last_fullsized":"2024-04-25T05:39:44.782932+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:40.474140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.474140+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:51.243595+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":310,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779374+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:39:44.779374+0000","last_peered":"2024-04-25T05:39:44.779374+0000","last_clean":"2024-04-25T05:39:44.779374+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:39:44.779374+0000","last_undegraded":"2024-04-25T05:39:44.779374+0000","last_fullsized":"2024-04-25T05:39:44.779374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":29,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782332+0000","last_change":"2024-04-25T05:39:29.802693+0000","last_active":"2024-04-25T05:39:44.782332+0000","last_peered":"2024-04-25T05:39:44.782332+0000","last_clean":"2024-04-25T05:39:44.782332+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:39:44.782332+0000","last_undegraded":"2024-04-25T05:39:44.782332+0000","last_fullsized":"2024-04-25T05:39:44.782332+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:27.553199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:27.553199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.099188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":132,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783358+0000","last_change":"2024-04-25T05:39:37.412459+0000","last_active":"2024-04-25T05:39:44.783358+0000","last_peered":"2024-04-25T05:39:44.783358+0000","last_clean":"2024-04-25T05:39:44.783358+0000","last_became_active":"2024-04-25T05:39:29.794349+0000","last_became_peered":"2024-04-25T05:39:29.794349+0000","last_unstale":"2024-04-25T05:39:44.783358+0000","last_undegraded":"2024-04-25T05:39:44.783358+0000","last_fullsized":"2024-04-25T05:39:44.783358+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:50.967870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788228+0000","last_change":"2024-04-25T05:39:29.824926+0000","last_active":"2024-04-25T05:39:44.788228+0000","last_peered":"2024-04-25T05:39:44.788228+0000","last_clean":"2024-04-25T05:39:44.788228+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:39:44.788228+0000","last_undegraded":"2024-04-25T05:39:44.788228+0000","last_fullsized":"2024-04-25T05:39:44.788228+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793358+0000","last_change":"2024-04-25T05:39:29.799258+0000","last_active":"2024-04-25T05:39:44.793358+0000","last_peered":"2024-04-25T05:39:44.793358+0000","last_clean":"2024-04-25T05:39:44.793358+0000","last_became_active":"2024-04-25T05:39:29.798189+0000","last_became_peered":"2024-04-25T05:39:29.798189+0000","last_unstale":"2024-04-25T05:39:44.793358+0000","last_undegraded":"2024-04-25T05:39:44.793358+0000","last_fullsized":"2024-04-25T05:39:44.793358+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.160752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:17.103753+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.780308+0000","last_change":"2024-04-25T05:39:29.797510+0000","last_active":"2024-04-25T05:39:44.780308+0000","last_peered":"2024-04-25T05:39:44.780308+0000","last_clean":"2024-04-25T05:39:44.780308+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:39:44.780308+0000","last_undegraded":"2024-04-25T05:39:44.780308+0000","last_fullsized":"2024-04-25T05:39:44.780308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782430+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:39:44.782430+0000","last_peered":"2024-04-25T05:39:44.782430+0000","last_clean":"2024-04-25T05:39:44.782430+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:39:44.782430+0000","last_undegraded":"2024-04-25T05:39:44.782430+0000","last_fullsized":"2024-04-25T05:39:44.782430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.554804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":380,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.791454+0000","last_change":"2024-04-25T05:39:29.832661+0000","last_active":"2024-04-25T05:39:44.791454+0000","last_peered":"2024-04-25T05:39:44.791454+0000","last_clean":"2024-04-25T05:39:44.791454+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:39:44.791454+0000","last_undegraded":"2024-04-25T05:39:44.791454+0000","last_fullsized":"2024-04-25T05:39:44.791454+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":132,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783321+0000","last_change":"2024-04-25T05:39:29.797305+0000","last_active":"2024-04-25T05:39:44.783321+0000","last_peered":"2024-04-25T05:39:44.783321+0000","last_clean":"2024-04-25T05:39:44.783321+0000","last_became_active":"2024-04-25T05:39:29.794256+0000","last_became_peered":"2024-04-25T05:39:29.794256+0000","last_unstale":"2024-04-25T05:39:44.783321+0000","last_undegraded":"2024-04-25T05:39:44.783321+0000","last_fullsized":"2024-04-25T05:39:44.783321+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:20.270747+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":303,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793100+0000","last_change":"2024-04-25T05:38:13.287078+0000","last_active":"2024-04-25T05:39:44.793100+0000","last_peered":"2024-04-25T05:39:44.793100+0000","last_clean":"2024-04-25T05:39:44.793100+0000","last_became_active":"2024-04-25T05:38:13.286756+0000","last_became_peered":"2024-04-25T05:38:13.286756+0000","last_unstale":"2024-04-25T05:39:44.793100+0000","last_undegraded":"2024-04-25T05:39:44.793100+0000","last_fullsized":"2024-04-25T05:39:44.793100+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781543+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:39:44.781543+0000","last_peered":"2024-04-25T05:39:44.781543+0000","last_clean":"2024-04-25T05:39:44.781543+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:39:44.781543+0000","last_undegraded":"2024-04-25T05:39:44.781543+0000","last_fullsized":"2024-04-25T05:39:44.781543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":214,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781634+0000","last_change":"2024-04-25T05:39:29.809885+0000","last_active":"2024-04-25T05:39:44.781634+0000","last_peered":"2024-04-25T05:39:44.781634+0000","last_clean":"2024-04-25T05:39:44.781634+0000","last_became_active":"2024-04-25T05:39:29.792929+0000","last_became_peered":"2024-04-25T05:39:29.792929+0000","last_unstale":"2024-04-25T05:39:44.781634+0000","last_undegraded":"2024-04-25T05:39:44.781634+0000","last_fullsized":"2024-04-25T05:39:44.781634+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":307,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.791628+0000","last_change":"2024-04-25T05:39:04.115851+0000","last_active":"2024-04-25T05:39:44.791628+0000","last_peered":"2024-04-25T05:39:44.791628+0000","last_clean":"2024-04-25T05:39:44.791628+0000","last_became_active":"2024-04-25T05:38:13.246584+0000","last_became_peered":"2024-04-25T05:38:13.246584+0000","last_unstale":"2024-04-25T05:39:44.791628+0000","last_undegraded":"2024-04-25T05:39:44.791628+0000","last_fullsized":"2024-04-25T05:39:44.791628+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.115756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.115756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:04.122079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":305,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787621+0000","last_change":"2024-04-25T05:35:41.325134+0000","last_active":"2024-04-25T05:39:44.787621+0000","last_peered":"2024-04-25T05:39:44.787621+0000","last_clean":"2024-04-25T05:39:44.787621+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:39:44.787621+0000","last_undegraded":"2024-04-25T05:39:44.787621+0000","last_fullsized":"2024-04-25T05:39:44.787621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:41.325007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:35:41.325007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787182+0000","last_change":"2024-04-25T05:39:29.809187+0000","last_active":"2024-04-25T05:39:44.787182+0000","last_peered":"2024-04-25T05:39:44.787182+0000","last_clean":"2024-04-25T05:39:44.787182+0000","last_became_active":"2024-04-25T05:39:29.785290+0000","last_became_peered":"2024-04-25T05:39:29.785290+0000","last_unstale":"2024-04-25T05:39:44.787182+0000","last_undegraded":"2024-04-25T05:39:44.787182+0000","last_fullsized":"2024-04-25T05:39:44.787182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:39:00.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.543557+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.705834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787794+0000","last_change":"2024-04-25T05:39:33.820019+0000","last_active":"2024-04-25T05:39:44.787794+0000","last_peered":"2024-04-25T05:39:44.787794+0000","last_clean":"2024-04-25T05:39:44.787794+0000","last_became_active":"2024-04-25T05:39:33.795217+0000","last_became_peered":"2024-04-25T05:39:33.795217+0000","last_unstale":"2024-04-25T05:39:44.787794+0000","last_undegraded":"2024-04-25T05:39:44.787794+0000","last_fullsized":"2024-04-25T05:39:44.787794+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:38:55.603358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.603358+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:57.436092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":126,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779226+0000","last_change":"2024-04-25T05:39:43.132193+0000","last_active":"2024-04-25T05:39:44.779226+0000","last_peered":"2024-04-25T05:39:44.779226+0000","last_clean":"2024-04-25T05:39:44.779226+0000","last_became_active":"2024-04-25T05:38:13.246544+0000","last_became_peered":"2024-04-25T05:38:13.246544+0000","last_unstale":"2024-04-25T05:39:44.779226+0000","last_undegraded":"2024-04-25T05:39:44.779226+0000","last_fullsized":"2024-04-25T05:39:44.779226+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:08.353453+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"160'9","reported_seq":402,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783386+0000","last_change":"2024-04-25T05:38:55.388653+0000","last_active":"2024-04-25T05:39:44.783386+0000","last_peered":"2024-04-25T05:39:44.783386+0000","last_clean":"2024-04-25T05:39:44.783386+0000","last_became_active":"2024-04-25T05:38:13.295760+0000","last_became_peered":"2024-04-25T05:38:13.295760+0000","last_unstale":"2024-04-25T05:39:44.783386+0000","last_undegraded":"2024-04-25T05:39:44.783386+0000","last_fullsized":"2024-04-25T05:39:44.783386+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'9","last_scrub_stamp":"2024-04-25T05:38:55.388487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.388487+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:57.015303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.778667+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:39:44.778667+0000","last_peered":"2024-04-25T05:39:44.778667+0000","last_clean":"2024-04-25T05:39:44.778667+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:39:44.778667+0000","last_undegraded":"2024-04-25T05:39:44.778667+0000","last_fullsized":"2024-04-25T05:39:44.778667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":348,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.794094+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:39:44.794094+0000","last_peered":"2024-04-25T05:39:44.794094+0000","last_clean":"2024-04-25T05:39:44.794094+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:39:44.794094+0000","last_undegraded":"2024-04-25T05:39:44.794094+0000","last_fullsized":"2024-04-25T05:39:44.794094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787467+0000","last_change":"2024-04-25T05:36:24.670134+0000","last_active":"2024-04-25T05:39:44.787467+0000","last_peered":"2024-04-25T05:39:44.787467+0000","last_clean":"2024-04-25T05:39:44.787467+0000","last_became_active":"2024-04-25T05:34:04.715372+0000","last_became_peered":"2024-04-25T05:34:04.715372+0000","last_unstale":"2024-04-25T05:39:44.787467+0000","last_undegraded":"2024-04-25T05:39:44.787467+0000","last_fullsized":"2024-04-25T05:39:44.787467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.670008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.670008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":375,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.794494+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:39:44.794494+0000","last_peered":"2024-04-25T05:39:44.794494+0000","last_clean":"2024-04-25T05:39:44.794494+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:39:44.794494+0000","last_undegraded":"2024-04-25T05:39:44.794494+0000","last_fullsized":"2024-04-25T05:39:44.794494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779208+0000","last_change":"2024-04-25T05:39:44.112703+0000","last_active":"2024-04-25T05:39:44.779208+0000","last_peered":"2024-04-25T05:39:44.779208+0000","last_clean":"2024-04-25T05:39:44.779208+0000","last_became_active":"2024-04-25T05:38:13.282059+0000","last_became_peered":"2024-04-25T05:38:13.282059+0000","last_unstale":"2024-04-25T05:39:44.779208+0000","last_undegraded":"2024-04-25T05:39:44.779208+0000","last_fullsized":"2024-04-25T05:39:44.779208+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.115488+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":350,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783585+0000","last_change":"2024-04-25T05:39:03.365135+0000","last_active":"2024-04-25T05:39:44.783585+0000","last_peered":"2024-04-25T05:39:44.783585+0000","last_clean":"2024-04-25T05:39:44.783585+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:39:44.783585+0000","last_undegraded":"2024-04-25T05:39:44.783585+0000","last_fullsized":"2024-04-25T05:39:44.783585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:39:03.364988+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.364988+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.589680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783532+0000","last_change":"2024-04-25T05:39:43.707526+0000","last_active":"2024-04-25T05:39:44.783532+0000","last_peered":"2024-04-25T05:39:44.783532+0000","last_clean":"2024-04-25T05:39:44.783532+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:39:44.783532+0000","last_undegraded":"2024-04-25T05:39:44.783532+0000","last_fullsized":"2024-04-25T05:39:44.783532+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.707414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.707414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.778892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":297,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.795229+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:39:44.795229+0000","last_peered":"2024-04-25T05:39:44.795229+0000","last_clean":"2024-04-25T05:39:44.795229+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:39:44.795229+0000","last_undegraded":"2024-04-25T05:39:44.795229+0000","last_fullsized":"2024-04-25T05:39:44.795229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779643+0000","last_change":"2024-04-25T05:39:29.798683+0000","last_active":"2024-04-25T05:39:44.779643+0000","last_peered":"2024-04-25T05:39:44.779643+0000","last_clean":"2024-04-25T05:39:44.779643+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:39:44.779643+0000","last_undegraded":"2024-04-25T05:39:44.779643+0000","last_fullsized":"2024-04-25T05:39:44.779643+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":191,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.792280+0000","last_change":"2024-04-25T05:38:13.255668+0000","last_active":"2024-04-25T05:39:44.792280+0000","last_peered":"2024-04-25T05:39:44.792280+0000","last_clean":"2024-04-25T05:39:44.792280+0000","last_became_active":"2024-04-25T05:38:13.245172+0000","last_became_peered":"2024-04-25T05:38:13.245172+0000","last_unstale":"2024-04-25T05:39:44.792280+0000","last_undegraded":"2024-04-25T05:39:44.792280+0000","last_fullsized":"2024-04-25T05:39:44.792280+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781393+0000","last_change":"2024-04-25T05:39:29.799422+0000","last_active":"2024-04-25T05:39:44.781393+0000","last_peered":"2024-04-25T05:39:44.781393+0000","last_clean":"2024-04-25T05:39:44.781393+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:39:44.781393+0000","last_undegraded":"2024-04-25T05:39:44.781393+0000","last_fullsized":"2024-04-25T05:39:44.781393+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":305,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782352+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:39:44.782352+0000","last_peered":"2024-04-25T05:39:44.782352+0000","last_clean":"2024-04-25T05:39:44.782352+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:39:44.782352+0000","last_undegraded":"2024-04-25T05:39:44.782352+0000","last_fullsized":"2024-04-25T05:39:44.782352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781266+0000","last_change":"2024-04-25T05:39:29.796336+0000","last_active":"2024-04-25T05:39:44.781266+0000","last_peered":"2024-04-25T05:39:44.781266+0000","last_clean":"2024-04-25T05:39:44.781266+0000","last_became_active":"2024-04-25T05:39:29.795963+0000","last_became_peered":"2024-04-25T05:39:29.795963+0000","last_unstale":"2024-04-25T05:39:44.781266+0000","last_undegraded":"2024-04-25T05:39:44.781266+0000","last_fullsized":"2024-04-25T05:39:44.781266+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:26.418226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:38:26.418226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:54.730209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":330,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.795324+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:39:44.795324+0000","last_peered":"2024-04-25T05:39:44.795324+0000","last_clean":"2024-04-25T05:39:44.795324+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:39:44.795324+0000","last_undegraded":"2024-04-25T05:39:44.795324+0000","last_fullsized":"2024-04-25T05:39:44.795324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"213'2389","reported_seq":2429,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.688411+0000","last_change":"2024-04-25T05:39:19.228971+0000","last_active":"2024-04-25T05:39:47.688411+0000","last_peered":"2024-04-25T05:39:47.688411+0000","last_clean":"2024-04-25T05:39:47.688411+0000","last_became_active":"2024-04-25T05:39:19.228682+0000","last_became_peered":"2024-04-25T05:39:19.228682+0000","last_unstale":"2024-04-25T05:39:47.688411+0000","last_undegraded":"2024-04-25T05:39:47.688411+0000","last_fullsized":"2024-04-25T05:39:47.688411+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":152896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":327,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782650+0000","last_change":"2024-04-25T05:39:29.794673+0000","last_active":"2024-04-25T05:39:44.782650+0000","last_peered":"2024-04-25T05:39:44.782650+0000","last_clean":"2024-04-25T05:39:44.782650+0000","last_became_active":"2024-04-25T05:39:29.792652+0000","last_became_peered":"2024-04-25T05:39:29.792652+0000","last_unstale":"2024-04-25T05:39:44.782650+0000","last_undegraded":"2024-04-25T05:39:44.782650+0000","last_fullsized":"2024-04-25T05:39:44.782650+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.778991+0000","last_change":"2024-04-25T05:38:13.313381+0000","last_active":"2024-04-25T05:39:44.778991+0000","last_peered":"2024-04-25T05:39:44.778991+0000","last_clean":"2024-04-25T05:39:44.778991+0000","last_became_active":"2024-04-25T05:38:13.295320+0000","last_became_peered":"2024-04-25T05:38:13.295320+0000","last_unstale":"2024-04-25T05:39:44.778991+0000","last_undegraded":"2024-04-25T05:39:44.778991+0000","last_fullsized":"2024-04-25T05:39:44.778991+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":418,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782011+0000","last_change":"2024-04-25T05:39:30.386374+0000","last_active":"2024-04-25T05:39:44.782011+0000","last_peered":"2024-04-25T05:39:44.782011+0000","last_clean":"2024-04-25T05:39:44.782011+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:39:44.782011+0000","last_undegraded":"2024-04-25T05:39:44.782011+0000","last_fullsized":"2024-04-25T05:39:44.782011+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:39:30.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.386275+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:32.056327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":423,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779834+0000","last_change":"2024-04-25T05:39:33.128642+0000","last_active":"2024-04-25T05:39:44.779834+0000","last_peered":"2024-04-25T05:39:44.779834+0000","last_clean":"2024-04-25T05:39:44.779834+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:39:44.779834+0000","last_undegraded":"2024-04-25T05:39:44.779834+0000","last_fullsized":"2024-04-25T05:39:44.779834+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'10","last_scrub_stamp":"2024-04-25T05:39:33.128477+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.128477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:48.900732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"213'2476","reported_seq":2705,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.767111+0000","last_change":"2024-04-25T05:39:35.780958+0000","last_active":"2024-04-25T05:39:47.767111+0000","last_peered":"2024-04-25T05:39:47.767111+0000","last_clean":"2024-04-25T05:39:47.767111+0000","last_became_active":"2024-04-25T05:39:35.778659+0000","last_became_peered":"2024-04-25T05:39:35.778659+0000","last_unstale":"2024-04-25T05:39:47.767111+0000","last_undegraded":"2024-04-25T05:39:47.767111+0000","last_fullsized":"2024-04-25T05:39:47.767111+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2476,"log_dups_size":0,"ondisk_log_size":2476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.504973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162267136,"num_objects":2476,"num_object_clones":0,"num_object_copies":7428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":158464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":293,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793272+0000","last_change":"2024-04-25T05:34:04.724535+0000","last_active":"2024-04-25T05:39:44.793272+0000","last_peered":"2024-04-25T05:39:44.793272+0000","last_clean":"2024-04-25T05:39:44.793272+0000","last_became_active":"2024-04-25T05:34:04.723743+0000","last_became_peered":"2024-04-25T05:34:04.723743+0000","last_unstale":"2024-04-25T05:39:44.793272+0000","last_undegraded":"2024-04-25T05:39:44.793272+0000","last_fullsized":"2024-04-25T05:39:44.793272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:34:03.694746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":329,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779455+0000","last_change":"2024-04-25T05:39:38.030757+0000","last_active":"2024-04-25T05:39:44.779455+0000","last_peered":"2024-04-25T05:39:44.779455+0000","last_clean":"2024-04-25T05:39:44.779455+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:39:44.779455+0000","last_undegraded":"2024-04-25T05:39:44.779455+0000","last_fullsized":"2024-04-25T05:39:44.779455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.030627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.030627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.843179+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":385,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787466+0000","last_change":"2024-04-25T05:39:33.829399+0000","last_active":"2024-04-25T05:39:44.787466+0000","last_peered":"2024-04-25T05:39:44.787466+0000","last_clean":"2024-04-25T05:39:44.787466+0000","last_became_active":"2024-04-25T05:39:33.801595+0000","last_became_peered":"2024-04-25T05:39:33.801595+0000","last_unstale":"2024-04-25T05:39:44.787466+0000","last_undegraded":"2024-04-25T05:39:44.787466+0000","last_fullsized":"2024-04-25T05:39:44.787466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:39:10.593601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:10.593601+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:30.934793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782261+0000","last_change":"2024-04-25T05:39:39.789694+0000","last_active":"2024-04-25T05:39:44.782261+0000","last_peered":"2024-04-25T05:39:44.782261+0000","last_clean":"2024-04-25T05:39:44.782261+0000","last_became_active":"2024-04-25T05:38:13.302285+0000","last_became_peered":"2024-04-25T05:38:13.302285+0000","last_unstale":"2024-04-25T05:39:44.782261+0000","last_undegraded":"2024-04-25T05:39:44.782261+0000","last_fullsized":"2024-04-25T05:39:44.782261+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:46.925548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"213'2419","reported_seq":3270,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.774763+0000","last_change":"2024-04-25T05:39:34.781545+0000","last_active":"2024-04-25T05:39:47.774763+0000","last_peered":"2024-04-25T05:39:47.774763+0000","last_clean":"2024-04-25T05:39:47.774763+0000","last_became_active":"2024-04-25T05:39:34.778772+0000","last_became_peered":"2024-04-25T05:39:34.778772+0000","last_unstale":"2024-04-25T05:39:47.774763+0000","last_undegraded":"2024-04-25T05:39:47.774763+0000","last_fullsized":"2024-04-25T05:39:47.774763+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2419,"log_dups_size":0,"ondisk_log_size":2419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:44.773418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158531584,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2419,"num_write_kb":154816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":26214400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":309,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.778434+0000","last_change":"2024-04-25T05:39:36.074193+0000","last_active":"2024-04-25T05:39:44.778434+0000","last_peered":"2024-04-25T05:39:44.778434+0000","last_clean":"2024-04-25T05:39:44.778434+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:39:44.778434+0000","last_undegraded":"2024-04-25T05:39:44.778434+0000","last_fullsized":"2024-04-25T05:39:44.778434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.074043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.074043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:38.732336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783070+0000","last_change":"2024-04-25T05:39:40.796539+0000","last_active":"2024-04-25T05:39:44.783070+0000","last_peered":"2024-04-25T05:39:44.783070+0000","last_clean":"2024-04-25T05:39:44.783070+0000","last_became_active":"2024-04-25T05:38:13.236574+0000","last_became_peered":"2024-04-25T05:38:13.236574+0000","last_unstale":"2024-04-25T05:39:44.783070+0000","last_undegraded":"2024-04-25T05:39:44.783070+0000","last_fullsized":"2024-04-25T05:39:44.783070+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:41.763860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781745+0000","last_change":"2024-04-25T05:39:29.821064+0000","last_active":"2024-04-25T05:39:44.781745+0000","last_peered":"2024-04-25T05:39:44.781745+0000","last_clean":"2024-04-25T05:39:44.781745+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:39:44.781745+0000","last_undegraded":"2024-04-25T05:39:44.781745+0000","last_fullsized":"2024-04-25T05:39:44.781745+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:30.395681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.395681+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":59,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782723+0000","last_change":"2024-04-25T05:39:30.591246+0000","last_active":"2024-04-25T05:39:44.782723+0000","last_peered":"2024-04-25T05:39:44.782723+0000","last_clean":"2024-04-25T05:39:44.782723+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:39:44.782723+0000","last_undegraded":"2024-04-25T05:39:44.782723+0000","last_fullsized":"2024-04-25T05:39:44.782723+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:39:30.591091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.591091+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:41.617716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"213'2489","reported_seq":3411,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.031480+0000","last_change":"2024-04-25T05:39:35.779484+0000","last_active":"2024-04-25T05:39:48.031480+0000","last_peered":"2024-04-25T05:39:48.031480+0000","last_clean":"2024-04-25T05:39:48.031480+0000","last_became_active":"2024-04-25T05:39:35.777746+0000","last_became_peered":"2024-04-25T05:39:35.777746+0000","last_unstale":"2024-04-25T05:39:48.031480+0000","last_undegraded":"2024-04-25T05:39:48.031480+0000","last_fullsized":"2024-04-25T05:39:48.031480+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.582782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":7467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":159296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":126,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779183+0000","last_change":"2024-04-25T05:39:06.050576+0000","last_active":"2024-04-25T05:39:44.779183+0000","last_peered":"2024-04-25T05:39:44.779183+0000","last_clean":"2024-04-25T05:39:44.779183+0000","last_became_active":"2024-04-25T05:38:13.240067+0000","last_became_peered":"2024-04-25T05:38:13.240067+0000","last_unstale":"2024-04-25T05:39:44.779183+0000","last_undegraded":"2024-04-25T05:39:44.779183+0000","last_fullsized":"2024-04-25T05:39:44.779183+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:09.492020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":312,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787452+0000","last_change":"2024-04-25T05:39:35.686849+0000","last_active":"2024-04-25T05:39:44.787452+0000","last_peered":"2024-04-25T05:39:44.787452+0000","last_clean":"2024-04-25T05:39:44.787452+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:39:44.787452+0000","last_undegraded":"2024-04-25T05:39:44.787452+0000","last_fullsized":"2024-04-25T05:39:44.787452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:35.686769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:35.686769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:59.630972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":333,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783243+0000","last_change":"2024-04-25T05:39:29.796897+0000","last_active":"2024-04-25T05:39:44.783243+0000","last_peered":"2024-04-25T05:39:44.783243+0000","last_clean":"2024-04-25T05:39:44.783243+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:39:44.783243+0000","last_undegraded":"2024-04-25T05:39:44.783243+0000","last_fullsized":"2024-04-25T05:39:44.783243+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787274+0000","last_change":"2024-04-25T05:39:41.766832+0000","last_active":"2024-04-25T05:39:44.787274+0000","last_peered":"2024-04-25T05:39:44.787274+0000","last_clean":"2024-04-25T05:39:44.787274+0000","last_became_active":"2024-04-25T05:38:13.294321+0000","last_became_peered":"2024-04-25T05:38:13.294321+0000","last_unstale":"2024-04-25T05:39:44.787274+0000","last_undegraded":"2024-04-25T05:39:44.787274+0000","last_fullsized":"2024-04-25T05:39:44.787274+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:45.433374+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"213'2433","reported_seq":2473,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.714948+0000","last_change":"2024-04-25T05:39:19.233374+0000","last_active":"2024-04-25T05:39:47.714948+0000","last_peered":"2024-04-25T05:39:47.714948+0000","last_clean":"2024-04-25T05:39:47.714948+0000","last_became_active":"2024-04-25T05:39:19.233055+0000","last_became_peered":"2024-04-25T05:39:19.233055+0000","last_unstale":"2024-04-25T05:39:47.714948+0000","last_undegraded":"2024-04-25T05:39:47.714948+0000","last_fullsized":"2024-04-25T05:39:47.714948+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2433,"log_dups_size":0,"ondisk_log_size":2433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:20.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449088,"num_objects":2433,"num_object_clones":0,"num_object_copies":7299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":155712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":488,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781604+0000","last_change":"2024-04-25T05:38:58.389048+0000","last_active":"2024-04-25T05:39:44.781604+0000","last_peered":"2024-04-25T05:39:44.781604+0000","last_clean":"2024-04-25T05:39:44.781604+0000","last_became_active":"2024-04-25T05:38:34.759567+0000","last_became_peered":"2024-04-25T05:38:34.759567+0000","last_unstale":"2024-04-25T05:39:44.781604+0000","last_undegraded":"2024-04-25T05:39:44.781604+0000","last_fullsized":"2024-04-25T05:39:44.781604+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.388898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.388898+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:06.835407+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":301,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.792373+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:39:44.792373+0000","last_peered":"2024-04-25T05:39:44.792373+0000","last_clean":"2024-04-25T05:39:44.792373+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:39:44.792373+0000","last_undegraded":"2024-04-25T05:39:44.792373+0000","last_fullsized":"2024-04-25T05:39:44.792373+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782416+0000","last_change":"2024-04-25T05:39:29.827029+0000","last_active":"2024-04-25T05:39:44.782416+0000","last_peered":"2024-04-25T05:39:44.782416+0000","last_clean":"2024-04-25T05:39:44.782416+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:39:44.782416+0000","last_undegraded":"2024-04-25T05:39:44.782416+0000","last_fullsized":"2024-04-25T05:39:44.782416+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:16.768175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":330,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787039+0000","last_change":"2024-04-25T05:39:29.785844+0000","last_active":"2024-04-25T05:39:44.787039+0000","last_peered":"2024-04-25T05:39:44.787039+0000","last_clean":"2024-04-25T05:39:44.787039+0000","last_became_active":"2024-04-25T05:39:29.785506+0000","last_became_peered":"2024-04-25T05:39:29.785506+0000","last_unstale":"2024-04-25T05:39:44.787039+0000","last_undegraded":"2024-04-25T05:39:44.787039+0000","last_fullsized":"2024-04-25T05:39:44.787039+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:37:43.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":458,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786428+0000","last_change":"2024-04-25T05:39:42.724143+0000","last_active":"2024-04-25T05:39:44.786428+0000","last_peered":"2024-04-25T05:39:44.786428+0000","last_clean":"2024-04-25T05:39:44.786428+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:39:44.786428+0000","last_undegraded":"2024-04-25T05:39:44.786428+0000","last_fullsized":"2024-04-25T05:39:44.786428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:39:42.723973+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.723973+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:05.917525+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":124,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786571+0000","last_change":"2024-04-25T05:39:29.798694+0000","last_active":"2024-04-25T05:39:44.786571+0000","last_peered":"2024-04-25T05:39:44.786571+0000","last_clean":"2024-04-25T05:39:44.786571+0000","last_became_active":"2024-04-25T05:39:29.798140+0000","last_became_peered":"2024-04-25T05:39:29.798140+0000","last_unstale":"2024-04-25T05:39:44.786571+0000","last_undegraded":"2024-04-25T05:39:44.786571+0000","last_fullsized":"2024-04-25T05:39:44.786571+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"213'2414","reported_seq":2454,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.113835+0000","last_change":"2024-04-25T05:39:19.228876+0000","last_active":"2024-04-25T05:39:48.113835+0000","last_peered":"2024-04-25T05:39:48.113835+0000","last_clean":"2024-04-25T05:39:48.113835+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:39:48.113835+0000","last_undegraded":"2024-04-25T05:39:48.113835+0000","last_fullsized":"2024-04-25T05:39:48.113835+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2414,"log_dups_size":0,"ondisk_log_size":2414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.356518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158203904,"num_objects":2414,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2414,"num_write_kb":154496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":619,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779335+0000","last_change":"2024-04-25T05:39:00.133654+0000","last_active":"2024-04-25T05:39:44.779335+0000","last_peered":"2024-04-25T05:39:44.779335+0000","last_clean":"2024-04-25T05:39:44.779335+0000","last_became_active":"2024-04-25T05:38:34.758423+0000","last_became_peered":"2024-04-25T05:38:34.758423+0000","last_unstale":"2024-04-25T05:39:44.779335+0000","last_undegraded":"2024-04-25T05:39:44.779335+0000","last_fullsized":"2024-04-25T05:39:44.779335+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:00.133580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.133580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:12.718787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781922+0000","last_change":"2024-04-25T05:39:29.799473+0000","last_active":"2024-04-25T05:39:44.781922+0000","last_peered":"2024-04-25T05:39:44.781922+0000","last_clean":"2024-04-25T05:39:44.781922+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:39:44.781922+0000","last_undegraded":"2024-04-25T05:39:44.781922+0000","last_fullsized":"2024-04-25T05:39:44.781922+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.790269+0000","last_change":"2024-04-25T05:39:29.792837+0000","last_active":"2024-04-25T05:39:44.790269+0000","last_peered":"2024-04-25T05:39:44.790269+0000","last_clean":"2024-04-25T05:39:44.790269+0000","last_became_active":"2024-04-25T05:39:29.792494+0000","last_became_peered":"2024-04-25T05:39:29.792494+0000","last_unstale":"2024-04-25T05:39:44.790269+0000","last_undegraded":"2024-04-25T05:39:44.790269+0000","last_fullsized":"2024-04-25T05:39:44.790269+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:52.054641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:52.054641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":342,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782390+0000","last_change":"2024-04-25T05:39:38.764826+0000","last_active":"2024-04-25T05:39:44.782390+0000","last_peered":"2024-04-25T05:39:44.782390+0000","last_clean":"2024-04-25T05:39:44.782390+0000","last_became_active":"2024-04-25T05:38:13.284281+0000","last_became_peered":"2024-04-25T05:38:13.284281+0000","last_unstale":"2024-04-25T05:39:44.782390+0000","last_undegraded":"2024-04-25T05:39:44.782390+0000","last_fullsized":"2024-04-25T05:39:44.782390+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.764632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.764632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:40.459074+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":374,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786568+0000","last_change":"2024-04-25T05:38:13.314279+0000","last_active":"2024-04-25T05:39:44.786568+0000","last_peered":"2024-04-25T05:39:44.786568+0000","last_clean":"2024-04-25T05:39:44.786568+0000","last_became_active":"2024-04-25T05:38:13.295547+0000","last_became_peered":"2024-04-25T05:38:13.295547+0000","last_unstale":"2024-04-25T05:39:44.786568+0000","last_undegraded":"2024-04-25T05:39:44.786568+0000","last_fullsized":"2024-04-25T05:39:44.786568+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:35:39.311807+0000","last_clean_scrub_stamp":"2024-04-25T05:35:39.311807+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":312,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.790718+0000","last_change":"2024-04-25T05:38:59.769234+0000","last_active":"2024-04-25T05:39:44.790718+0000","last_peered":"2024-04-25T05:39:44.790718+0000","last_clean":"2024-04-25T05:39:44.790718+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:39:44.790718+0000","last_undegraded":"2024-04-25T05:39:44.790718+0000","last_fullsized":"2024-04-25T05:39:44.790718+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.769120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.769120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.633366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"213'2461","reported_seq":3369,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.765537+0000","last_change":"2024-04-25T05:39:34.781500+0000","last_active":"2024-04-25T05:39:47.765537+0000","last_peered":"2024-04-25T05:39:47.765537+0000","last_clean":"2024-04-25T05:39:47.765537+0000","last_became_active":"2024-04-25T05:39:34.778800+0000","last_became_peered":"2024-04-25T05:39:34.778800+0000","last_unstale":"2024-04-25T05:39:47.765537+0000","last_undegraded":"2024-04-25T05:39:47.765537+0000","last_fullsized":"2024-04-25T05:39:47.765537+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.108740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161284096,"num_objects":2461,"num_object_clones":0,"num_object_copies":7383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":157504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":28049408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":309,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782634+0000","last_change":"2024-04-25T05:39:09.353431+0000","last_active":"2024-04-25T05:39:44.782634+0000","last_peered":"2024-04-25T05:39:44.782634+0000","last_clean":"2024-04-25T05:39:44.782634+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:39:44.782634+0000","last_undegraded":"2024-04-25T05:39:44.782634+0000","last_fullsized":"2024-04-25T05:39:44.782634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.353310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:10.766600+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":620,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.779947+0000","last_change":"2024-04-25T05:39:32.801925+0000","last_active":"2024-04-25T05:39:44.779947+0000","last_peered":"2024-04-25T05:39:44.779947+0000","last_clean":"2024-04-25T05:39:44.779947+0000","last_became_active":"2024-04-25T05:39:32.801558+0000","last_became_peered":"2024-04-25T05:39:32.801558+0000","last_unstale":"2024-04-25T05:39:44.779947+0000","last_undegraded":"2024-04-25T05:39:44.779947+0000","last_fullsized":"2024-04-25T05:39:44.779947+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.104548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.104548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:11.373025+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.791670+0000","last_change":"2024-04-25T05:39:29.797447+0000","last_active":"2024-04-25T05:39:44.791670+0000","last_peered":"2024-04-25T05:39:44.791670+0000","last_clean":"2024-04-25T05:39:44.791670+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:39:44.791670+0000","last_undegraded":"2024-04-25T05:39:44.791670+0000","last_fullsized":"2024-04-25T05:39:44.791670+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":50,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782885+0000","last_change":"2024-04-25T05:39:29.827544+0000","last_active":"2024-04-25T05:39:44.782885+0000","last_peered":"2024-04-25T05:39:44.782885+0000","last_clean":"2024-04-25T05:39:44.782885+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:39:44.782885+0000","last_undegraded":"2024-04-25T05:39:44.782885+0000","last_fullsized":"2024-04-25T05:39:44.782885+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'18","last_scrub_stamp":"2024-04-25T05:39:04.573693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.573693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781637+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:39:44.781637+0000","last_peered":"2024-04-25T05:39:44.781637+0000","last_clean":"2024-04-25T05:39:44.781637+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:39:44.781637+0000","last_undegraded":"2024-04-25T05:39:44.781637+0000","last_fullsized":"2024-04-25T05:39:44.781637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"213'2501","reported_seq":2541,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.032667+0000","last_change":"2024-04-25T05:39:19.231225+0000","last_active":"2024-04-25T05:39:48.032667+0000","last_peered":"2024-04-25T05:39:48.032667+0000","last_clean":"2024-04-25T05:39:48.032667+0000","last_became_active":"2024-04-25T05:39:19.230926+0000","last_became_peered":"2024-04-25T05:39:19.230926+0000","last_unstale":"2024-04-25T05:39:48.032667+0000","last_undegraded":"2024-04-25T05:39:48.032667+0000","last_fullsized":"2024-04-25T05:39:48.032667+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2501,"log_dups_size":0,"ondisk_log_size":2501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.268052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2501,"num_write_kb":160064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":503,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786821+0000","last_change":"2024-04-25T05:39:32.814133+0000","last_active":"2024-04-25T05:39:44.786821+0000","last_peered":"2024-04-25T05:39:44.786821+0000","last_clean":"2024-04-25T05:39:44.786821+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:39:44.786821+0000","last_undegraded":"2024-04-25T05:39:44.786821+0000","last_fullsized":"2024-04-25T05:39:44.786821+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.010323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.010323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.554151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786103+0000","last_change":"2024-04-25T05:39:00.922795+0000","last_active":"2024-04-25T05:39:44.786103+0000","last_peered":"2024-04-25T05:39:44.786103+0000","last_clean":"2024-04-25T05:39:44.786103+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:39:44.786103+0000","last_undegraded":"2024-04-25T05:39:44.786103+0000","last_fullsized":"2024-04-25T05:39:44.786103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.922679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.922679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:18.355708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":118,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787102+0000","last_change":"2024-04-25T05:38:13.239349+0000","last_active":"2024-04-25T05:39:44.787102+0000","last_peered":"2024-04-25T05:39:44.787102+0000","last_clean":"2024-04-25T05:39:44.787102+0000","last_became_active":"2024-04-25T05:38:13.238969+0000","last_became_peered":"2024-04-25T05:38:13.238969+0000","last_unstale":"2024-04-25T05:39:44.787102+0000","last_undegraded":"2024-04-25T05:39:44.787102+0000","last_fullsized":"2024-04-25T05:39:44.787102+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.153092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.153092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":685,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:46.441352+0000","last_change":"2024-04-25T05:38:13.325017+0000","last_active":"2024-04-25T05:39:46.441352+0000","last_peered":"2024-04-25T05:39:46.441352+0000","last_clean":"2024-04-25T05:39:46.441352+0000","last_became_active":"2024-04-25T05:38:13.305327+0000","last_became_peered":"2024-04-25T05:38:13.305327+0000","last_unstale":"2024-04-25T05:39:46.441352+0000","last_undegraded":"2024-04-25T05:39:46.441352+0000","last_fullsized":"2024-04-25T05:39:46.441352+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:50.092317+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":339,"num_read_kb":334,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":147,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782253+0000","last_change":"2024-04-25T05:37:56.295161+0000","last_active":"2024-04-25T05:39:44.782253+0000","last_peered":"2024-04-25T05:39:44.782253+0000","last_clean":"2024-04-25T05:39:44.782253+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:39:44.782253+0000","last_undegraded":"2024-04-25T05:39:44.782253+0000","last_fullsized":"2024-04-25T05:39:44.782253+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:36:32.151512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:32.151512+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783916+0000","last_change":"2024-04-25T05:38:13.296713+0000","last_active":"2024-04-25T05:39:44.783916+0000","last_peered":"2024-04-25T05:39:44.783916+0000","last_clean":"2024-04-25T05:39:44.783916+0000","last_became_active":"2024-04-25T05:38:13.296013+0000","last_became_peered":"2024-04-25T05:38:13.296013+0000","last_unstale":"2024-04-25T05:39:44.783916+0000","last_undegraded":"2024-04-25T05:39:44.783916+0000","last_fullsized":"2024-04-25T05:39:44.783916+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"213'2565","reported_seq":2605,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.123578+0000","last_change":"2024-04-25T05:39:19.229227+0000","last_active":"2024-04-25T05:39:48.123578+0000","last_peered":"2024-04-25T05:39:48.123578+0000","last_clean":"2024-04-25T05:39:48.123578+0000","last_became_active":"2024-04-25T05:39:19.228912+0000","last_became_peered":"2024-04-25T05:39:19.228912+0000","last_unstale":"2024-04-25T05:39:48.123578+0000","last_undegraded":"2024-04-25T05:39:48.123578+0000","last_fullsized":"2024-04-25T05:39:48.123578+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2565,"log_dups_size":0,"ondisk_log_size":2565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:37.352621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2565,"num_write_kb":164160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":492,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783054+0000","last_change":"2024-04-25T05:39:32.802517+0000","last_active":"2024-04-25T05:39:44.783054+0000","last_peered":"2024-04-25T05:39:44.783054+0000","last_clean":"2024-04-25T05:39:44.783054+0000","last_became_active":"2024-04-25T05:39:32.801996+0000","last_became_peered":"2024-04-25T05:39:32.801996+0000","last_unstale":"2024-04-25T05:39:44.783054+0000","last_undegraded":"2024-04-25T05:39:44.783054+0000","last_fullsized":"2024-04-25T05:39:44.783054+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.081204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.081204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:01.911147+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786185+0000","last_change":"2024-04-25T05:39:30.760584+0000","last_active":"2024-04-25T05:39:44.786185+0000","last_peered":"2024-04-25T05:39:44.786185+0000","last_clean":"2024-04-25T05:39:44.786185+0000","last_became_active":"2024-04-25T05:39:29.792454+0000","last_became_peered":"2024-04-25T05:39:29.792454+0000","last_unstale":"2024-04-25T05:39:44.786185+0000","last_undegraded":"2024-04-25T05:39:44.786185+0000","last_fullsized":"2024-04-25T05:39:44.786185+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:30.760468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.760468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.143766+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786339+0000","last_change":"2024-04-25T05:36:17.550073+0000","last_active":"2024-04-25T05:39:44.786339+0000","last_peered":"2024-04-25T05:39:44.786339+0000","last_clean":"2024-04-25T05:39:44.786339+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:39:44.786339+0000","last_undegraded":"2024-04-25T05:39:44.786339+0000","last_fullsized":"2024-04-25T05:39:44.786339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:17.549952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:17.549952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783710+0000","last_change":"2024-04-25T05:38:22.405354+0000","last_active":"2024-04-25T05:39:44.783710+0000","last_peered":"2024-04-25T05:39:44.783710+0000","last_clean":"2024-04-25T05:39:44.783710+0000","last_became_active":"2024-04-25T05:38:13.296480+0000","last_became_peered":"2024-04-25T05:38:13.296480+0000","last_unstale":"2024-04-25T05:39:44.783710+0000","last_undegraded":"2024-04-25T05:39:44.783710+0000","last_fullsized":"2024-04-25T05:39:44.783710+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:38:22.404901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:22.404901+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782916+0000","last_change":"2024-04-25T05:39:36.496099+0000","last_active":"2024-04-25T05:39:44.782916+0000","last_peered":"2024-04-25T05:39:44.782916+0000","last_clean":"2024-04-25T05:39:44.782916+0000","last_became_active":"2024-04-25T05:39:29.797329+0000","last_became_peered":"2024-04-25T05:39:29.797329+0000","last_unstale":"2024-04-25T05:39:44.782916+0000","last_undegraded":"2024-04-25T05:39:44.782916+0000","last_fullsized":"2024-04-25T05:39:44.782916+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.495957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.495957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.120296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"213'2401","reported_seq":2440,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.930874+0000","last_change":"2024-04-25T05:39:19.229498+0000","last_active":"2024-04-25T05:39:47.930874+0000","last_peered":"2024-04-25T05:39:47.930874+0000","last_clean":"2024-04-25T05:39:47.930874+0000","last_became_active":"2024-04-25T05:39:19.229203+0000","last_became_peered":"2024-04-25T05:39:19.229203+0000","last_unstale":"2024-04-25T05:39:47.930874+0000","last_undegraded":"2024-04-25T05:39:47.930874+0000","last_fullsized":"2024-04-25T05:39:47.930874+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2401,"log_dups_size":0,"ondisk_log_size":2401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:42.361829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157351936,"num_objects":2401,"num_object_clones":0,"num_object_copies":7203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2401,"num_write_kb":153664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":492,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782824+0000","last_change":"2024-04-25T05:39:32.805680+0000","last_active":"2024-04-25T05:39:44.782824+0000","last_peered":"2024-04-25T05:39:44.782824+0000","last_clean":"2024-04-25T05:39:44.782824+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:39:44.782824+0000","last_undegraded":"2024-04-25T05:39:44.782824+0000","last_fullsized":"2024-04-25T05:39:44.782824+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.069332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.069332+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.710721+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":118,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788040+0000","last_change":"2024-04-25T05:38:13.244062+0000","last_active":"2024-04-25T05:39:44.788040+0000","last_peered":"2024-04-25T05:39:44.788040+0000","last_clean":"2024-04-25T05:39:44.788040+0000","last_became_active":"2024-04-25T05:38:13.243650+0000","last_became_peered":"2024-04-25T05:38:13.243650+0000","last_unstale":"2024-04-25T05:39:44.788040+0000","last_undegraded":"2024-04-25T05:39:44.788040+0000","last_fullsized":"2024-04-25T05:39:44.788040+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783485+0000","last_change":"2024-04-25T05:39:38.370074+0000","last_active":"2024-04-25T05:39:44.783485+0000","last_peered":"2024-04-25T05:39:44.783485+0000","last_clean":"2024-04-25T05:39:44.783485+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:39:44.783485+0000","last_undegraded":"2024-04-25T05:39:44.783485+0000","last_fullsized":"2024-04-25T05:39:44.783485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:48.827472+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":281,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783935+0000","last_change":"2024-04-25T05:39:09.887432+0000","last_active":"2024-04-25T05:39:44.783935+0000","last_peered":"2024-04-25T05:39:44.783935+0000","last_clean":"2024-04-25T05:39:44.783935+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:39:44.783935+0000","last_undegraded":"2024-04-25T05:39:44.783935+0000","last_fullsized":"2024-04-25T05:39:44.783935+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.887340+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.423904+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"208'11","reported_seq":36,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783897+0000","last_change":"2024-04-25T05:39:29.809774+0000","last_active":"2024-04-25T05:39:44.783897+0000","last_peered":"2024-04-25T05:39:44.783897+0000","last_clean":"2024-04-25T05:39:44.783897+0000","last_became_active":"2024-04-25T05:39:29.795733+0000","last_became_peered":"2024-04-25T05:39:29.795733+0000","last_unstale":"2024-04-25T05:39:44.783897+0000","last_undegraded":"2024-04-25T05:39:44.783897+0000","last_fullsized":"2024-04-25T05:39:44.783897+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'10","last_scrub_stamp":"2024-04-25T05:38:24.406643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:24.406643+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:52.726430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"213'2433","reported_seq":2473,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.705612+0000","last_change":"2024-04-25T05:39:19.228938+0000","last_active":"2024-04-25T05:39:47.705612+0000","last_peered":"2024-04-25T05:39:47.705612+0000","last_clean":"2024-04-25T05:39:47.705612+0000","last_became_active":"2024-04-25T05:39:19.228620+0000","last_became_peered":"2024-04-25T05:39:19.228620+0000","last_unstale":"2024-04-25T05:39:47.705612+0000","last_undegraded":"2024-04-25T05:39:47.705612+0000","last_fullsized":"2024-04-25T05:39:47.705612+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2433,"log_dups_size":0,"ondisk_log_size":2433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:28.198905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449088,"num_objects":2433,"num_object_clones":0,"num_object_copies":7299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":155712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":488,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793647+0000","last_change":"2024-04-25T05:38:58.078848+0000","last_active":"2024-04-25T05:39:44.793647+0000","last_peered":"2024-04-25T05:39:44.793647+0000","last_clean":"2024-04-25T05:39:44.793647+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:39:44.793647+0000","last_undegraded":"2024-04-25T05:39:44.793647+0000","last_fullsized":"2024-04-25T05:39:44.793647+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.078483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.078483+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.982642+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.778997+0000","last_change":"2024-04-25T05:39:29.798367+0000","last_active":"2024-04-25T05:39:44.778997+0000","last_peered":"2024-04-25T05:39:44.778997+0000","last_clean":"2024-04-25T05:39:44.778997+0000","last_became_active":"2024-04-25T05:39:29.798026+0000","last_became_peered":"2024-04-25T05:39:29.798026+0000","last_unstale":"2024-04-25T05:39:44.778997+0000","last_undegraded":"2024-04-25T05:39:44.778997+0000","last_fullsized":"2024-04-25T05:39:44.778997+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":207,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787210+0000","last_change":"2024-04-25T05:39:34.741064+0000","last_active":"2024-04-25T05:39:44.787210+0000","last_peered":"2024-04-25T05:39:44.787210+0000","last_clean":"2024-04-25T05:39:44.787210+0000","last_became_active":"2024-04-25T05:38:13.289662+0000","last_became_peered":"2024-04-25T05:38:13.289662+0000","last_unstale":"2024-04-25T05:39:44.787210+0000","last_undegraded":"2024-04-25T05:39:44.787210+0000","last_fullsized":"2024-04-25T05:39:44.787210+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:52.993594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":414,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787258+0000","last_change":"2024-04-25T05:39:29.835092+0000","last_active":"2024-04-25T05:39:44.787258+0000","last_peered":"2024-04-25T05:39:44.787258+0000","last_clean":"2024-04-25T05:39:44.787258+0000","last_became_active":"2024-04-25T05:39:29.793081+0000","last_became_peered":"2024-04-25T05:39:29.793081+0000","last_unstale":"2024-04-25T05:39:44.787258+0000","last_undegraded":"2024-04-25T05:39:44.787258+0000","last_fullsized":"2024-04-25T05:39:44.787258+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":383,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788512+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:39:44.788512+0000","last_peered":"2024-04-25T05:39:44.788512+0000","last_clean":"2024-04-25T05:39:44.788512+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:39:44.788512+0000","last_undegraded":"2024-04-25T05:39:44.788512+0000","last_fullsized":"2024-04-25T05:39:44.788512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"213'2456","reported_seq":2496,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.109691+0000","last_change":"2024-04-25T05:39:19.232542+0000","last_active":"2024-04-25T05:39:48.109691+0000","last_peered":"2024-04-25T05:39:48.109691+0000","last_clean":"2024-04-25T05:39:48.109691+0000","last_became_active":"2024-04-25T05:39:19.232224+0000","last_became_peered":"2024-04-25T05:39:19.232224+0000","last_unstale":"2024-04-25T05:39:48.109691+0000","last_undegraded":"2024-04-25T05:39:48.109691+0000","last_fullsized":"2024-04-25T05:39:48.109691+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:45.474301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160956416,"num_objects":2456,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2456,"num_write_kb":157184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":497,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787875+0000","last_change":"2024-04-25T05:39:32.795454+0000","last_active":"2024-04-25T05:39:44.787875+0000","last_peered":"2024-04-25T05:39:44.787875+0000","last_clean":"2024-04-25T05:39:44.787875+0000","last_became_active":"2024-04-25T05:39:32.795085+0000","last_became_peered":"2024-04-25T05:39:32.795085+0000","last_unstale":"2024-04-25T05:39:44.787875+0000","last_undegraded":"2024-04-25T05:39:44.787875+0000","last_fullsized":"2024-04-25T05:39:44.787875+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.548529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.548529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:39:59.253966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788197+0000","last_change":"2024-04-25T05:36:19.732582+0000","last_active":"2024-04-25T05:39:44.788197+0000","last_peered":"2024-04-25T05:39:44.788197+0000","last_clean":"2024-04-25T05:39:44.788197+0000","last_became_active":"2024-04-25T05:34:04.718166+0000","last_became_peered":"2024-04-25T05:34:04.718166+0000","last_unstale":"2024-04-25T05:39:44.788197+0000","last_undegraded":"2024-04-25T05:39:44.788197+0000","last_fullsized":"2024-04-25T05:39:44.788197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:19.732424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.732424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.795039+0000","last_change":"2024-04-25T05:38:13.247469+0000","last_active":"2024-04-25T05:39:44.795039+0000","last_peered":"2024-04-25T05:39:44.795039+0000","last_clean":"2024-04-25T05:39:44.795039+0000","last_became_active":"2024-04-25T05:38:13.246750+0000","last_became_peered":"2024-04-25T05:38:13.246750+0000","last_unstale":"2024-04-25T05:39:44.795039+0000","last_undegraded":"2024-04-25T05:39:44.795039+0000","last_fullsized":"2024-04-25T05:39:44.795039+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788413+0000","last_change":"2024-04-25T05:38:13.326846+0000","last_active":"2024-04-25T05:39:44.788413+0000","last_peered":"2024-04-25T05:39:44.788413+0000","last_clean":"2024-04-25T05:39:44.788413+0000","last_became_active":"2024-04-25T05:38:13.305935+0000","last_became_peered":"2024-04-25T05:38:13.305935+0000","last_unstale":"2024-04-25T05:39:44.788413+0000","last_undegraded":"2024-04-25T05:39:44.788413+0000","last_fullsized":"2024-04-25T05:39:44.788413+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786858+0000","last_change":"2024-04-25T05:36:16.784378+0000","last_active":"2024-04-25T05:39:44.786858+0000","last_peered":"2024-04-25T05:39:44.786858+0000","last_clean":"2024-04-25T05:39:44.786858+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:39:44.786858+0000","last_undegraded":"2024-04-25T05:39:44.786858+0000","last_fullsized":"2024-04-25T05:39:44.786858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:16.784084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:16.784084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"213'2403","reported_seq":3191,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.029162+0000","last_change":"2024-04-25T05:39:34.781974+0000","last_active":"2024-04-25T05:39:48.029162+0000","last_peered":"2024-04-25T05:39:48.029162+0000","last_clean":"2024-04-25T05:39:48.029162+0000","last_became_active":"2024-04-25T05:39:34.780422+0000","last_became_peered":"2024-04-25T05:39:34.780422+0000","last_unstale":"2024-04-25T05:39:48.029162+0000","last_undegraded":"2024-04-25T05:39:48.029162+0000","last_fullsized":"2024-04-25T05:39:48.029162+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2403,"log_dups_size":0,"ondisk_log_size":2403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:38.827751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157483008,"num_objects":2403,"num_object_clones":0,"num_object_copies":7209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":153792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":24117248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":497,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781736+0000","last_change":"2024-04-25T05:39:32.794193+0000","last_active":"2024-04-25T05:39:44.781736+0000","last_peered":"2024-04-25T05:39:44.781736+0000","last_clean":"2024-04-25T05:39:44.781736+0000","last_became_active":"2024-04-25T05:39:32.793746+0000","last_became_peered":"2024-04-25T05:39:32.793746+0000","last_unstale":"2024-04-25T05:39:44.781736+0000","last_undegraded":"2024-04-25T05:39:44.781736+0000","last_fullsized":"2024-04-25T05:39:44.781736+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:08.227492+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":315,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782823+0000","last_change":"2024-04-25T05:38:13.284810+0000","last_active":"2024-04-25T05:39:44.782823+0000","last_peered":"2024-04-25T05:39:44.782823+0000","last_clean":"2024-04-25T05:39:44.782823+0000","last_became_active":"2024-04-25T05:38:13.284451+0000","last_became_peered":"2024-04-25T05:38:13.284451+0000","last_unstale":"2024-04-25T05:39:44.782823+0000","last_undegraded":"2024-04-25T05:39:44.782823+0000","last_fullsized":"2024-04-25T05:39:44.782823+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"160'9","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787723+0000","last_change":"2024-04-25T05:38:13.244991+0000","last_active":"2024-04-25T05:39:44.787723+0000","last_peered":"2024-04-25T05:39:44.787723+0000","last_clean":"2024-04-25T05:39:44.787723+0000","last_became_active":"2024-04-25T05:38:13.238743+0000","last_became_peered":"2024-04-25T05:38:13.238743+0000","last_unstale":"2024-04-25T05:39:44.787723+0000","last_undegraded":"2024-04-25T05:39:44.787723+0000","last_fullsized":"2024-04-25T05:39:44.787723+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:37:48.604653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:48.604653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788319+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:39:44.788319+0000","last_peered":"2024-04-25T05:39:44.788319+0000","last_clean":"2024-04-25T05:39:44.788319+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:39:44.788319+0000","last_undegraded":"2024-04-25T05:39:44.788319+0000","last_fullsized":"2024-04-25T05:39:44.788319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":349,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786670+0000","last_change":"2024-04-25T05:39:39.713342+0000","last_active":"2024-04-25T05:39:44.786670+0000","last_peered":"2024-04-25T05:39:44.786670+0000","last_clean":"2024-04-25T05:39:44.786670+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:39:44.786670+0000","last_undegraded":"2024-04-25T05:39:44.786670+0000","last_fullsized":"2024-04-25T05:39:44.786670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.713165+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.240786+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"213'2407","reported_seq":2447,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:47.705554+0000","last_change":"2024-04-25T05:39:19.230917+0000","last_active":"2024-04-25T05:39:47.705554+0000","last_peered":"2024-04-25T05:39:47.705554+0000","last_clean":"2024-04-25T05:39:47.705554+0000","last_became_active":"2024-04-25T05:39:19.230541+0000","last_became_peered":"2024-04-25T05:39:19.230541+0000","last_unstale":"2024-04-25T05:39:47.705554+0000","last_undegraded":"2024-04-25T05:39:47.705554+0000","last_fullsized":"2024-04-25T05:39:47.705554+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2407,"log_dups_size":0,"ondisk_log_size":2407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:29.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":7221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":154048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":313,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793991+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:39:44.793991+0000","last_peered":"2024-04-25T05:39:44.793991+0000","last_clean":"2024-04-25T05:39:44.793991+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:39:44.793991+0000","last_undegraded":"2024-04-25T05:39:44.793991+0000","last_fullsized":"2024-04-25T05:39:44.793991+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:01.546178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783821+0000","last_change":"2024-04-25T05:39:29.794362+0000","last_active":"2024-04-25T05:39:44.783821+0000","last_peered":"2024-04-25T05:39:44.783821+0000","last_clean":"2024-04-25T05:39:44.783821+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:39:44.783821+0000","last_undegraded":"2024-04-25T05:39:44.783821+0000","last_fullsized":"2024-04-25T05:39:44.783821+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787014+0000","last_change":"2024-04-25T05:39:28.539337+0000","last_active":"2024-04-25T05:39:44.787014+0000","last_peered":"2024-04-25T05:39:44.787014+0000","last_clean":"2024-04-25T05:39:44.787014+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:39:44.787014+0000","last_undegraded":"2024-04-25T05:39:44.787014+0000","last_fullsized":"2024-04-25T05:39:44.787014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:28.539190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:28.539190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.676380+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":379,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787612+0000","last_change":"2024-04-25T05:39:29.836257+0000","last_active":"2024-04-25T05:39:44.787612+0000","last_peered":"2024-04-25T05:39:44.787612+0000","last_clean":"2024-04-25T05:39:44.787612+0000","last_became_active":"2024-04-25T05:39:29.787446+0000","last_became_peered":"2024-04-25T05:39:29.787446+0000","last_unstale":"2024-04-25T05:39:44.787612+0000","last_undegraded":"2024-04-25T05:39:44.787612+0000","last_fullsized":"2024-04-25T05:39:44.787612+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:36:15.740682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:15.740682+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"213'2461","reported_seq":2501,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.036283+0000","last_change":"2024-04-25T05:39:19.229731+0000","last_active":"2024-04-25T05:39:48.036283+0000","last_peered":"2024-04-25T05:39:48.036283+0000","last_clean":"2024-04-25T05:39:48.036283+0000","last_became_active":"2024-04-25T05:39:19.229438+0000","last_became_peered":"2024-04-25T05:39:19.229438+0000","last_unstale":"2024-04-25T05:39:48.036283+0000","last_undegraded":"2024-04-25T05:39:48.036283+0000","last_fullsized":"2024-04-25T05:39:48.036283+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:32.701047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161284096,"num_objects":2461,"num_object_clones":0,"num_object_copies":7383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":157504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":488,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.781869+0000","last_change":"2024-04-25T05:38:59.374948+0000","last_active":"2024-04-25T05:39:44.781869+0000","last_peered":"2024-04-25T05:39:44.781869+0000","last_clean":"2024-04-25T05:39:44.781869+0000","last_became_active":"2024-04-25T05:38:34.759755+0000","last_became_peered":"2024-04-25T05:38:34.759755+0000","last_unstale":"2024-04-25T05:39:44.781869+0000","last_undegraded":"2024-04-25T05:39:44.781869+0000","last_fullsized":"2024-04-25T05:39:44.781869+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:00.299246+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783000+0000","last_change":"2024-04-25T05:39:00.794514+0000","last_active":"2024-04-25T05:39:44.783000+0000","last_peered":"2024-04-25T05:39:44.783000+0000","last_clean":"2024-04-25T05:39:44.783000+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:39:44.783000+0000","last_undegraded":"2024-04-25T05:39:44.783000+0000","last_fullsized":"2024-04-25T05:39:44.783000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.794388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.794388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.097831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.786578+0000","last_change":"2024-04-25T05:39:29.798267+0000","last_active":"2024-04-25T05:39:44.786578+0000","last_peered":"2024-04-25T05:39:44.786578+0000","last_clean":"2024-04-25T05:39:44.786578+0000","last_became_active":"2024-04-25T05:39:29.797924+0000","last_became_peered":"2024-04-25T05:39:29.797924+0000","last_unstale":"2024-04-25T05:39:44.786578+0000","last_undegraded":"2024-04-25T05:39:44.786578+0000","last_fullsized":"2024-04-25T05:39:44.786578+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.782819+0000","last_change":"2024-04-25T05:39:37.518793+0000","last_active":"2024-04-25T05:39:44.782819+0000","last_peered":"2024-04-25T05:39:44.782819+0000","last_clean":"2024-04-25T05:39:44.782819+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:39:44.782819+0000","last_undegraded":"2024-04-25T05:39:44.782819+0000","last_fullsized":"2024-04-25T05:39:44.782819+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.518653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.518653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:46.935057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":344,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.788513+0000","last_change":"2024-04-25T05:39:34.605820+0000","last_active":"2024-04-25T05:39:44.788513+0000","last_peered":"2024-04-25T05:39:44.788513+0000","last_clean":"2024-04-25T05:39:44.788513+0000","last_became_active":"2024-04-25T05:34:00.692107+0000","last_became_peered":"2024-04-25T05:34:00.692107+0000","last_unstale":"2024-04-25T05:39:44.788513+0000","last_undegraded":"2024-04-25T05:39:44.788513+0000","last_fullsized":"2024-04-25T05:39:44.788513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:34.605704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.605704+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:41.043070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"213'2493","reported_seq":2533,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:48.113837+0000","last_change":"2024-04-25T05:39:19.231808+0000","last_active":"2024-04-25T05:39:48.113837+0000","last_peered":"2024-04-25T05:39:48.113837+0000","last_clean":"2024-04-25T05:39:48.113837+0000","last_became_active":"2024-04-25T05:39:19.231525+0000","last_became_peered":"2024-04-25T05:39:19.231525+0000","last_unstale":"2024-04-25T05:39:48.113837+0000","last_undegraded":"2024-04-25T05:39:48.113837+0000","last_fullsized":"2024-04-25T05:39:48.113837+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":2493,"log_dups_size":0,"ondisk_log_size":2493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.331924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":7479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":159552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":619,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.778506+0000","last_change":"2024-04-25T05:39:01.087263+0000","last_active":"2024-04-25T05:39:44.778506+0000","last_peered":"2024-04-25T05:39:44.778506+0000","last_clean":"2024-04-25T05:39:44.778506+0000","last_became_active":"2024-04-25T05:38:34.761642+0000","last_became_peered":"2024-04-25T05:38:34.761642+0000","last_unstale":"2024-04-25T05:39:44.778506+0000","last_undegraded":"2024-04-25T05:39:44.778506+0000","last_fullsized":"2024-04-25T05:39:44.778506+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:01.087145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:01.087145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:14.030738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787811+0000","last_change":"2024-04-25T05:36:28.676820+0000","last_active":"2024-04-25T05:39:44.787811+0000","last_peered":"2024-04-25T05:39:44.787811+0000","last_clean":"2024-04-25T05:39:44.787811+0000","last_became_active":"2024-04-25T05:34:04.719358+0000","last_became_peered":"2024-04-25T05:34:04.719358+0000","last_unstale":"2024-04-25T05:39:44.787811+0000","last_undegraded":"2024-04-25T05:39:44.787811+0000","last_fullsized":"2024-04-25T05:39:44.787811+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:36:28.676660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":394,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.783457+0000","last_change":"2024-04-25T05:39:33.738444+0000","last_active":"2024-04-25T05:39:44.783457+0000","last_peered":"2024-04-25T05:39:44.783457+0000","last_clean":"2024-04-25T05:39:44.783457+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:39:44.783457+0000","last_undegraded":"2024-04-25T05:39:44.783457+0000","last_fullsized":"2024-04-25T05:39:44.783457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:39:33.738307+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.738307+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:59.211618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.793004+0000","last_change":"2024-04-25T05:38:13.286297+0000","last_active":"2024-04-25T05:39:44.793004+0000","last_peered":"2024-04-25T05:39:44.793004+0000","last_clean":"2024-04-25T05:39:44.793004+0000","last_became_active":"2024-04-25T05:38:13.285812+0000","last_became_peered":"2024-04-25T05:38:13.285812+0000","last_unstale":"2024-04-25T05:39:44.793004+0000","last_undegraded":"2024-04-25T05:39:44.793004+0000","last_fullsized":"2024-04-25T05:39:44.793004+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":359,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-04-25T05:39:44.787335+0000","last_change":"2024-04-25T05:39:29.815696+0000","last_active":"2024-04-25T05:39:44.787335+0000","last_peered":"2024-04-25T05:39:44.787335+0000","last_clean":"2024-04-25T05:39:44.787335+0000","last_became_active":"2024-04-25T05:39:29.785808+0000","last_became_peered":"2024-04-25T05:39:29.785808+0000","last_unstale":"2024-04-25T05:39:44.787335+0000","last_undegraded":"2024-04-25T05:39:44.787335+0000","last_fullsized":"2024-04-25T05:39:44.787335+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2569076736,"num_objects":39201,"num_object_clones":0,"num_object_copies":117603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39201,"num_write_kb":2508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":134873088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7690715136,"data_stored":7690715136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39201,"ondisk_log_size":39201,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":339,"num_read_kb":334,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":3828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":12800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":647,"ondisk_log_size":647,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070539,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102748,"kb_used_data":900820,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92642404,"statfs":{"total":95995035648,"available":94865821696,"internally_reserved":0,"allocated":922439680,"data_stored":921731583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":206764490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364879,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":921460,"kb_used_data":773676,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92823692,"statfs":{"total":95995035648,"available":95051460608,"internally_reserved":0,"allocated":792244224,"data_stored":791575613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1324980,"kb_used_data":1081388,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92420172,"statfs":{"total":95995035648,"available":94638256128,"internally_reserved":0,"allocated":1107341312,"data_stored":1106607613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326024,"kb_used_data":1096320,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92419128,"statfs":{"total":95995035648,"available":94637187072,"internally_reserved":0,"allocated":1122631680,"data_stored":1121942525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518364,"kb_used_data":1220628,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92226788,"statfs":{"total":95995035648,"available":94440230912,"internally_reserved":0,"allocated":1249923072,"data_stored":1249214006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304871879},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919440,"kb_used_data":771656,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92825712,"statfs":{"total":95995035648,"available":95053529088,"internally_reserved":0,"allocated":790175744,"data_stored":789478643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691924,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683648,"kb_used_data":1526136,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92061504,"statfs":{"total":95995035648,"available":94270980096,"internally_reserved":0,"allocated":1562763264,"data_stored":1562082495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1133051904,"data_stored":1133051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1126498304,"data_stored":1126498304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1302528000,"data_stored":1302528000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":789577728,"data_stored":789577728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1587150848,"data_stored":1587150848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":798752768,"data_stored":798752768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":953155584,"data_stored":953155584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:39:51.767 INFO:tasks.thrashosds.thrasher:Setting 6.b to [7, 0, 6, 5, 1, 2] 2024-04-25T05:39:51.767 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.b', '7', '0', '6', '5', '1', '2'] 2024-04-25T05:39:51.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 6.b 7 0 6 5 1 2 2024-04-25T05:39:51.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:51.793 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:51.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:51.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:51 smithi043 ceph-mon[27010]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:51.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:51 smithi043 ceph-mon[37492]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:51.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:51.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:51.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:51 smithi060 ceph-mon[30898]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:39:52.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:52.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:52.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:52.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:52.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 15 46856 46841 117.089 105.562 0.0103692 0.00853527 2024-04-25T05:39:52.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:52.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:52.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:52.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:52.503 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:52.504 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:52.587 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:39:52.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:52.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:52.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:52.887 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:52 smithi043 ceph-mon[27010]: pgmap v300: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 732 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-04-25T05:39:52.887 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:52 smithi043 ceph-mon[37492]: pgmap v300: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 732 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-04-25T05:39:52.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:52.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:52.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:52.976 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:52 smithi060 ceph-mon[30898]: pgmap v300: 158 pgs: 158 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 732 B/s rd, 88 MiB/s wr, 1.42k op/s 2024-04-25T05:39:53.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:53.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:53.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:53.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:53.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:39:52 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:39:53.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 48441 48425 116.393 99 0.0130255 0.00858636 2024-04-25T05:39:53.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:53.335 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:53.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:53.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:53.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:53.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:53.661 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:39:53.661 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:53.662 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:53.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:53.662 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:53.725 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:39:53.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:53.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:53.779 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:39:53.827 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:39:53.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:53.875 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:39:53.898 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:53.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:53.916 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:39:53.937 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:39:53.937 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:39:53.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:39:53.937 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:39:53.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165408 7.c 7:30f51da7:::benchmark_data_smithi043_74356_object48539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 122.800923789, 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.013940488, 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165408 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130983+0000", 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3799999999999998e-07 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130986+0000", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9830000000000001e-06 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130992+0000", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8289999999999996e-06 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.130997+0000", 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4129999999999997e-06 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.131160+0000", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016310700000000001 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.131169+0000", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3159999999999997e-06 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132057+0000", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.000888287 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132198+0000", 2024-04-25T05:39:53.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014108 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.132336+0000", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013787700000000001 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135299+0000", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002962648 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141378+0000", 2024-04-25T05:39:53.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060790439999999996 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144822+0000", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034443059999999998 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144875+0000", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2799000000000001e-05 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144923+0000", 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8361e-05 2024-04-25T05:39:53.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165436 7.c 7:3742aa0a:::benchmark_data_smithi043_74356_object48567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 122.76586090799999, 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97903490800000004, 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165436 2024-04-25T05:39:53.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166046+0000", 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166047+0000", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3e-06 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166051+0000", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.048e-06 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166059+0000", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9319999999999996e-06 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.166065+0000", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3849999999999999e-06 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.167368+0000", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001302316 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.167377+0000", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5529999999999998e-06 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135334+0000", 2024-04-25T05:39:53.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96795652399999998 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135460+0000", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012674600000000001 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.135592+0000", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013188100000000001 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141420+0000", 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058273129999999998 2024-04-25T05:39:53.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141456+0000", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6055e-05 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144990+0000", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035342210000000002 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145044+0000", 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3715999999999999e-05 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145080+0000", 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6918000000000001e-05 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165470 7.c 7:382b4df9:::benchmark_data_smithi043_74356_object48601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 122.720581651, 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93388353000000002, 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.950 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165470 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211325+0000", 2024-04-25T05:39:53.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211326+0000", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5850000000000001e-06 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211332+0000", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2499999999999997e-06 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211343+0000", 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1252e-05 2024-04-25T05:39:53.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.211351+0000", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9850000000000006e-06 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.213644+0000", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022933939999999998 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.213653+0000", 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1449999999999997e-06 2024-04-25T05:39:53.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.138752+0000", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.925098902 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.138925+0000", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017265999999999999 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.139075+0000", 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000150354 2024-04-25T05:39:53.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141492+0000", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024170149999999998 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145087+0000", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003594683 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145128+0000", 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0615000000000003e-05 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145172+0000", 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4524999999999998e-05 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.145208+0000", 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6164999999999997e-05 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:165480 7.c 7:3433329a:::benchmark_data_smithi043_74356_object48611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-04-25T05:39:53.956 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 122.706654997, 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92329429500000004, 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 165480 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:39:53.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225251+0000", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225253+0000", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.781e-06 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225260+0000", 2024-04-25T05:39:53.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3899999999999998e-06 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225275+0000", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5044e-05 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225283+0000", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.439e-06 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225337+0000", 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3766e-05 2024-04-25T05:39:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:51.225346+0000", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2099999999999999e-06 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141525+0000", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91617920900000005 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141659+0000", 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013405700000000001 2024-04-25T05:39:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.141786+0000", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012625700000000001 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.144809+0000", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003022921 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.147708+0000", 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002899956 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148449+0000", 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00074005000000000002 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148496+0000", 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7524000000000001e-05 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:52.148546+0000", 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9691000000000002e-05 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196420 7.a 7:547ca08e:::benchmark_data_smithi043_74356_object79551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.895509+0000", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.036397179999994, 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17897647699999999, 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196420 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895509+0000", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895509+0000", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895511+0000", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2920000000000001e-06 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895514+0000", 2024-04-25T05:39:53.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7910000000000002e-06 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895523+0000", 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.422e-06 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895529+0000", 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6810000000000001e-06 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895575+0000", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5843999999999997e-05 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895685+0000", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010949499999999999 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895965+0000", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00028048999999999999 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901053+0000", 2024-04-25T05:39:53.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050876139999999999 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074377+0000", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.173324109 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074407+0000", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0454999999999999e-05 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074449+0000", 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2014999999999997e-05 2024-04-25T05:39:53.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074486+0000", 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6269000000000001e-05 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196423 7.2 7:4bd23bac:::benchmark_data_smithi043_74356_object79554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.895546+0000", 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.03636066, 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.178797134, 2024-04-25T05:39:53.969 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196423 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895546+0000", 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895546+0000", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895546+0000", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7399999999999998e-07 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895549+0000", 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.666e-06 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895555+0000", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8549999999999998e-06 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895560+0000", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6389999999999997e-06 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895695+0000", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013424500000000001 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895826+0000", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013122200000000001 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.895951+0000", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000125537 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.900968+0000", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050163070000000002 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074225+0000", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17325695199999999 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074257+0000", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1953999999999999e-05 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074302+0000", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4906e-05 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.074343+0000", 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1377e-05 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:196430 7.6 e175/166 7:67a0b441:::benchmark_data_smithi043_74356_object79561:head v 175'12177, mlcod=175'12175)", 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.899881+0000", 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.032025804, 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17614164900000001, 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899881+0000", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899881+0000", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899882+0000", 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0720000000000001e-06 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899888+0000", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5579999999999997e-06 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899897+0000", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0669999999999998e-06 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.899904+0000", 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7909999999999999e-06 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.900692+0000", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00078802899999999999 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.900724+0000", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2325e-05 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.075913+0000", 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.175188761 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076022+0000", 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109046 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196433 7.a 7:56d47a7f:::benchmark_data_smithi043_74356_object79564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.901726+0000", 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.030180462000004, 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17850012300000001, 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.979 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196433 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901726+0000", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901726+0000", 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901727+0000", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.285e-06 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901732+0000", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4889999999999997e-06 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901743+0000", 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1415e-05 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.901751+0000", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0080000000000002e-06 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.904548+0000", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027965569999999999 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.904662+0000", 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114282 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.904776+0000", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113539 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076028+0000", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.171252667 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076285+0000", 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025684599999999997 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.080135+0000", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038497660000000001 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.080185+0000", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0146999999999999e-05 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.080226+0000", 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1121999999999997e-05 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.984 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196435 7.3 7:c444ebfb:::benchmark_data_smithi043_74356_object79566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.902027+0000", 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.029879467000001, 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.177823801, 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196435 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902027+0000", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902027+0000", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902028+0000", 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6700000000000002e-07 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902033+0000", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8509999999999998e-06 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902043+0000", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9340000000000001e-06 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902050+0000", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.323e-06 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.903626+0000", 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001576519 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.903743+0000", 2024-04-25T05:39:53.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116877 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.903867+0000", 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123879 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.073496+0000", 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.169628415 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.073663+0000", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000167517 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.079752+0000", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060886430000000004 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.079805+0000", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3551000000000003e-05 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.079851+0000", 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5425e-05 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:196432 7.4 e175/166 7:201f4639:::benchmark_data_smithi043_74356_object79563:head v 175'12230, mlcod=175'12226)", 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.902304+0000", 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.029602746999998, 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.184325286, 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902304+0000", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902304+0000", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902304+0000", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4600000000000001e-07 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902308+0000", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5669999999999999e-06 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902312+0000", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7820000000000001e-06 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902317+0000", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4039999999999997e-06 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902453+0000", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013590799999999999 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902470+0000", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7405999999999999e-05 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.086553+0000", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18408308900000001 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.086629+0000", 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5883999999999994e-05 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:196431 7.6 e175/166 7:64208f17:::benchmark_data_smithi043_74356_object79562:head v 175'12178, mlcod=175'12175)", 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.902377+0000", 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.029529394999997, 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.177244606, 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902377+0000", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902377+0000", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902377+0000", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0100000000000001e-07 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:53.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902381+0000", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3409999999999999e-06 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902385+0000", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0899999999999998e-06 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902389+0000", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2740000000000001e-06 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:53.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.073724+0000", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.171334773 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.073745+0000", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1665e-05 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.079559+0000", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058137320000000003 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:53.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.079622+0000", 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2529999999999999e-05 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.24734.0:196438 7.6 e175/166 7:6894f35c:::benchmark_data_smithi043_74356_object79569:head v 175'12179, mlcod=175'12175)", 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.905211+0000", 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.026695408999998, 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17887660799999999, 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:53.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905211+0000", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905211+0000", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905212+0000", 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6599999999999996e-07 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905217+0000", 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1309999999999997e-06 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905226+0000", 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6770000000000001e-06 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.001 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.905231+0000", 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.959e-06 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076473+0000", 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17124156200000001 2024-04-25T05:39:54.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076497+0000", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3743e-05 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.084046+0000", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0075488350000000003 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.084088+0000", 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2035e-05 2024-04-25T05:39:54.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198428 7.7 7:eca14a66:::benchmark_data_smithi043_74356_object81559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.773735+0000", 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.158171422999999, 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0409333709999999, 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198428 2024-04-25T05:39:54.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773735+0000", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773735+0000", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773736+0000", 2024-04-25T05:39:54.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4410000000000001e-06 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773743+0000", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1340000000000001e-06 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773758+0000", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5727e-05 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773767+0000", 2024-04-25T05:39:54.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4819999999999994e-06 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773834+0000", 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7677999999999998e-05 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773844+0000", 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3559999999999999e-06 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.809759+0000", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0359155840000001 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.809892+0000", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000132335 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.810017+0000", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012498399999999999 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814491+0000", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0044740689999999998 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814532+0000", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1350999999999998e-05 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814578+0000", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6020000000000003e-05 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814628+0000", 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9357000000000002e-05 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814668+0000", 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0852999999999997e-05 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198434 7.7 7:e5a8b8ce:::benchmark_data_smithi043_74356_object81565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.781147+0000", 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.150759874000002, 2024-04-25T05:39:54.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.043880157, 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198434 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781147+0000", 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781147+0000", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781148+0000", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8299999999999995e-07 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781153+0000", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0180000000000001e-06 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781162+0000", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8020000000000003e-06 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781170+0000", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5510000000000002e-06 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781227+0000", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7015999999999998e-05 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.781235+0000", 2024-04-25T05:39:54.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5099999999999998e-06 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814702+0000", 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0334669750000001 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814798+0000", 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5582999999999996e-05 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:39:54.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.814913+0000", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114982 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.824839+0000", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0099256629999999995 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.824891+0000", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2500000000000002e-05 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.824934+0000", 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2867999999999997e-05 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.824987+0000", 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3195999999999997e-05 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825027+0000", 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9509999999999999e-05 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198441 7.f 7:f5cb7407:::benchmark_data_smithi043_74356_object81572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.791709+0000", 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.140197568999994, 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0220609940000001, 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198441 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791709+0000", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791709+0000", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791710+0000", 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8299999999999996e-07 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791713+0000", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7079999999999999e-06 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791722+0000", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7639999999999994e-06 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791730+0000", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4109999999999999e-06 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791786+0000", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6149000000000001e-05 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.791794+0000", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3979999999999999e-06 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.774762+0000", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98296808899999999 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.775011+0000", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000249115 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.775170+0000", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015843000000000001 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.779356+0000", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041860170000000002 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.813647+0000", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.034291441999999998 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.813683+0000", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5364000000000002e-05 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.813731+0000", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8909999999999998e-05 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.813770+0000", 2024-04-25T05:39:54.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8414000000000002e-05 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198450 7.7 7:ebfbc493:::benchmark_data_smithi043_74356_object81581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.798719+0000", 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.133187890000002, 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0264793649999999, 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198450 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798719+0000", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798719+0000", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798719+0000", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0599999999999999e-07 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798724+0000", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5700000000000003e-06 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798732+0000", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5369999999999997e-06 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798740+0000", 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7020000000000001e-06 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798803+0000", 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2372999999999994e-05 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.798812+0000", 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1300000000000007e-06 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.817081+0000", 2024-04-25T05:39:54.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0182694450000001 2024-04-25T05:39:54.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:54.028 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:39:54.028 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:39:54.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.817186+0000", 2024-04-25T05:39:54.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105351 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.817301+0000", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114275 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.824845+0000", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.007543864 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825068+0000", 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00022358799999999999 2024-04-25T05:39:54.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825107+0000", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8894999999999999e-05 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825158+0000", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0994e-05 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825198+0000", 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9835000000000001e-05 2024-04-25T05:39:54.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198462 7.7 7:ed7eaf65:::benchmark_data_smithi043_74356_object81593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.804989+0000", 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.126917105000004, 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.023171689, 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.031 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198462 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804989+0000", 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804989+0000", 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804990+0000", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7599999999999997e-07 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804993+0000", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8779999999999998e-06 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804999+0000", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9059999999999999e-06 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805007+0000", 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3399999999999998e-06 2024-04-25T05:39:54.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805065+0000", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8276000000000002e-05 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805074+0000", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4740000000000005e-06 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.822200+0000", 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0171261600000001 2024-04-25T05:39:54.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.822297+0000", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7276999999999997e-05 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.822407+0000", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109672 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825294+0000", 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002887559 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825389+0000", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4859999999999996e-05 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828055+0000", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026653179999999999 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828111+0000", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6362e-05 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828161+0000", 2024-04-25T05:39:54.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0130999999999998e-05 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198464 7.f 7:fe21449e:::benchmark_data_smithi043_74356_object81595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.809385+0000", 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.122521372999998, 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0242616470000001, 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.037 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198464 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809385+0000", 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809385+0000", 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809386+0000", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.341e-06 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809392+0000", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6720000000000001e-06 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809404+0000", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1963e-05 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809412+0000", 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3229999999999998e-06 2024-04-25T05:39:54.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809472+0000", 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9760999999999999e-05 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.809482+0000", 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3840000000000003e-06 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825431+0000", 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0159492919999999 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825525+0000", 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4587999999999996e-05 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825630+0000", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010433399999999999 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828209+0000", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025788030000000002 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828254+0000", 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.57e-05 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833553+0000", 2024-04-25T05:39:54.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0052983919999999999 2024-04-25T05:39:54.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833603+0000", 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0658999999999997e-05 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833647+0000", 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3435e-05 2024-04-25T05:39:54.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198469 7.f 7:fdc5fa8d:::benchmark_data_smithi043_74356_object81600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.810616+0000", 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.121290025999997, 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0233205750000001, 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.049 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198469 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810616+0000", 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810616+0000", 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810617+0000", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1200000000000002e-07 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810622+0000", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8099999999999997e-06 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810634+0000", 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1554e-05 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810641+0000", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5839999999999997e-06 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810684+0000", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2892000000000002e-05 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.810689+0000", 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2399999999999998e-06 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828290+0000", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0176008700000001 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828407+0000", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011671300000000001 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828537+0000", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013007599999999999 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833690+0000", 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051527530000000004 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833701+0000", 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1313e-05 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833863+0000", 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016165600000000001 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833904+0000", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1480000000000003e-05 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833937+0000", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2821999999999998e-05 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198475 7.7 7:ee1c6c06:::benchmark_data_smithi043_74356_object81606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.817124+0000", 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 78.114782180999995, 2024-04-25T05:39:54.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0166865169999999, 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198475 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817124+0000", 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.056 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817124+0000", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817125+0000", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6599999999999994e-07 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817129+0000", 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0090000000000001e-06 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817139+0000", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3530000000000001e-06 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817144+0000", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.575e-06 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817191+0000", 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6805000000000003e-05 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.817197+0000", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6470000000000003e-06 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825456+0000", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0082596850000001 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825571+0000", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115127 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:39:54.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.825749+0000", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000177529 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828041+0000", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002292377 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.828168+0000", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000127031 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:39:54.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833733+0000", 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055644880000000002 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833774+0000", 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1409999999999998e-05 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.833811+0000", 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6514999999999999e-05 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:39:54.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:39:54.062 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:39:54.063 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:39:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:39:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:39:54.065 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:39:54.065 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:39:54.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:54.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:54.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 49866 49850 115.38 89.0625 0.0142156 0.00866068 2024-04-25T05:39:54.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.330 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:54.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:54.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.608 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:54.663 INFO:teuthology.orchestra.run.smithi043.stderr:set 6.b pg_upmap_items mapping to [7->0,6->5,1->2] 2024-04-25T05:39:54.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:54.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:54.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:54.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:54.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:54.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: osdmap e216: 8 total, 8 up, 7 in 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2552664786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: pgmap v302: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:39:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]: dispatch 2024-04-25T05:39:55.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:55.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: osdmap e216: 8 total, 8 up, 7 in 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2552664786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: pgmap v302: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:39:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]: dispatch 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1983603924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: osdmap e216: 8 total, 8 up, 7 in 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2552664786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: pgmap v302: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:39:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]: dispatch 2024-04-25T05:39:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:55.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:55.273 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 51268 51252 114.389 87.625 0.00632666 0.0087217 2024-04-25T05:39:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.296 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:55.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.433 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:55.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:55.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:55.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:55.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.829 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:55.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:55.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:55.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:55.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]': finished 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: osdmap e217: 8 total, 8 up, 7 in 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:55 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:56.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:56.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]': finished 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: osdmap e217: 8 total, 8 up, 7 in 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:56.124 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1252038226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [7, 0, 6, 5, 1, 2]}]': finished 2024-04-25T05:39:56.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: osdmap e217: 8 total, 8 up, 7 in 2024-04-25T05:39:56.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:56.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:39:56.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:56.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:55 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:56.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:56.250 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:56.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 52730 52714 113.595 91.375 0.0131105 0.0087982 2024-04-25T05:39:56.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:56.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:56.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:56.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:56.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:56.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:56.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:56.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:56.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:56.975 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:56.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:56.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:56 smithi060 ceph-mon[30898]: osdmap e218: 8 total, 8 up, 7 in 2024-04-25T05:39:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:56 smithi060 ceph-mon[30898]: pgmap v305: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 854 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:39:57.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:57.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:56 smithi043 ceph-mon[27010]: osdmap e218: 8 total, 8 up, 7 in 2024-04-25T05:39:57.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:56 smithi043 ceph-mon[27010]: pgmap v305: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 854 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:39:57.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:56 smithi043 ceph-mon[37492]: osdmap e218: 8 total, 8 up, 7 in 2024-04-25T05:39:57.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:56 smithi043 ceph-mon[37492]: pgmap v305: 158 pgs: 158 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 854 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:39:57.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:57.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 54456 54440 113.404 107.875 0.0130809 0.00881322 2024-04-25T05:39:57.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:57.385 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:57.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:39:57.521 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:39:57.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.660 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:57.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:57.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:57.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:57.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:57.931 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:57.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:58.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:58.067 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:58.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:58.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:39:58.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:58.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 56135 56119 113.131 104.938 0.00776454 0.00883431 2024-04-25T05:39:58.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:58.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:58.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:58.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:58.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:39:58.623 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:39:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.629 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:58.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:39:58.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:58.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:58.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:39:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:58.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:39:58.950 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[27010]: osdmap e219: 8 total, 8 up, 7 in 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[27010]: pgmap v307: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 467 op/s 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[37492]: osdmap e219: 8 total, 8 up, 7 in 2024-04-25T05:39:58.951 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:39:58 smithi043 ceph-mon[37492]: pgmap v307: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 467 op/s 2024-04-25T05:39:58.952 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:58 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:39:58.953 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:58 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:39:58.953 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:58 smithi060 ceph-mon[30898]: osdmap e219: 8 total, 8 up, 7 in 2024-04-25T05:39:58.953 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:39:58 smithi060 ceph-mon[30898]: pgmap v307: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 467 op/s 2024-04-25T05:39:59.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:59.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:59.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:39:59.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:39:59.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 15 57687 57672 112.628 97.0625 0.00949793 0.00887366 2024-04-25T05:39:59.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:57718 8.2 8:44720569:::benchmark_data_smithi043_104001_object57717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:59.296393+0000", 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0070805570000000003, 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0071195019999999998, 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:39:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 57718 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296393+0000", 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296393+0000", 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296394+0000", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.916e-06 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296433+0000", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8356999999999999e-05 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296454+0000", 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1392000000000001e-05 2024-04-25T05:39:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296466+0000", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1651000000000001e-05 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296533+0000", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7157999999999996e-05 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296683+0000", 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015032899999999999 2024-04-25T05:39:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.296858+0000", 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017492000000000001 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:57722 8.5 e219/209 8:a6d22804:::benchmark_data_smithi043_104001_object57721:head v 219'3631, mlcod=219'3629)", 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:59.299226+0000", 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0042469910000000003, 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043756230000000004, 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:39:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299226+0000", 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299226+0000", 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299227+0000", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3699999999999999e-07 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299261+0000", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4384000000000001e-05 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299273+0000", 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.117e-05 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299280+0000", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4680000000000003e-06 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299557+0000", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027655900000000001 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299577+0000", 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0805000000000002e-05 2024-04-25T05:39:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:57718 8.2 e219/198)", 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:59.299422+0000", 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0040507090000000004, 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042440749999999999, 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299422+0000", 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299422+0000", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299423+0000", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0800000000000004e-07 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299427+0000", 2024-04-25T05:39:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0539999999999996e-06 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299436+0000", 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9490000000000002e-06 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299443+0000", 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9179999999999997e-06 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:57718 8.2 e219/198)", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:59.299878+0000", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0035951059999999998, 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038394929999999998, 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299878+0000", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:39:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299878+0000", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299879+0000", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2900000000000003e-07 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299883+0000", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.403e-06 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:39:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299892+0000", 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9430000000000006e-06 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.299899+0000", 2024-04-25T05:39:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.63e-06 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:57728 8.4 e219/198 8:251e3529:::benchmark_data_smithi043_104001_object57727:head v 219'3602, mlcod=219'3600)", 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:59.301241+0000", 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0022322280000000002, 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025324380000000001, 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:39:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301241+0000", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301241+0000", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301242+0000", 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9400000000000004e-07 2024-04-25T05:39:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301510+0000", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026784900000000002 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301518+0000", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3590000000000008e-06 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301524+0000", 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5270000000000004e-06 2024-04-25T05:39:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301580+0000", 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5823999999999999e-05 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:59.301599+0000", 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8621e-05 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:39:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:39:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 5 2024-04-25T05:39:59.323 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:39:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:39:59.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:39:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:39:59.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:59.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:39:59.744 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:39:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:39:59.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:39:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:39:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:39:59.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:39:59.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:00.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:00.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:00.142 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:39:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:39:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:00.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:00.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:40:00.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 59187 59171 112.054 93.6875 0.014669 0.0089188 2024-04-25T05:40:00.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:00.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:00.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.454 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:00.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:00.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:00.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:00.602 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:00.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:00.734 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:00.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:40:00.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:00.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:40:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:00.885 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:00.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[27010]: pgmap v308: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-04-25T05:40:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[27010]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[27010]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[37492]: pgmap v308: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[37492]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:00 smithi043 ceph-mon[37492]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.899 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.899 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:00.899 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:00 smithi060 ceph-mon[30898]: pgmap v308: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-04-25T05:40:00.899 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:00 smithi060 ceph-mon[30898]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.899 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:00 smithi060 ceph-mon[30898]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:40:00.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:01.016 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:01.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:01.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:01.213 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:01.268 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:40:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:40:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:40:01.268 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:01.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 60566 60550 111.293 86.1875 0.011139 0.00897922 2024-04-25T05:40:01.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:01.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:01.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:01.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:01.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:01.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:01.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:01.827 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:01.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:01.878 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:01.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:01.926 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:01 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:40:01.926 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:01 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:40:01.926 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:01 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:01.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:40:01.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:40:01.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:01.972 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:40:01.972 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:40:01.972 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:01 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:01.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:40:01.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:02.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:02.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:02.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:02.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 62033 62017 110.731 91.6875 0.0130365 0.0090253 2024-04-25T05:40:02.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:02.397 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:02.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:02.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:02.538 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:02.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:02.569 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:02.577 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:02.608 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:02.649 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:02.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:02.685 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:40:02.688 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:02.723 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:02.740 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:02.776 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:02.810 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:02.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:02.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:02.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:02.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:02.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[27010]: pgmap v309: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 719 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:40:02.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3076295372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1420062817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[37492]: pgmap v309: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 719 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3076295372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.832 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1420062817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:02.861 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:02.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:02.987 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-mon[30898]: pgmap v309: 158 pgs: 158 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 719 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:40:02.988 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:02.988 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3076295372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.988 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:02.988 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1420062817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:02.997 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:02.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:03.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:03.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:03.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:03.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:02 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:03.254 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:03.274 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 63413 63397 110.05 86.25 0.00559812 0.00908202 2024-04-25T05:40:03.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:03.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:03.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:03.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:03.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:03.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:03.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:03.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:03.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:03.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:03.965 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:03 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:03.965 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:03 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:03.965 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:03 smithi060 ceph-mon[30898]: osdmap e220: 8 total, 8 up, 7 in 2024-04-25T05:40:03.965 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:03 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:40:03.965 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:04.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.041 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:04.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[27010]: osdmap e220: 8 total, 8 up, 7 in 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[37492]: osdmap e220: 8 total, 8 up, 7 in 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:40:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:04.143 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.180 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:04.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:04.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 64716 64700 109.277 81.4375 0.00318823 0.00914079 2024-04-25T05:40:04.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:04.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:04.340 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:04.340 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:04.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:04.438 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:40:04.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:40:04.438 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-25T05:40:04.438 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.610751 2024-04-25T05:40:04.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 0.6107509478453164 2024-04-25T05:40:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:04.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:04.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:04.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:04.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:04.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:04.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:04.996 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:04.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:04.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:40:04.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:40:04.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:40:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: pgmap v311: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:40:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:40:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:40:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:40:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:04 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:40:05.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:40:05.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:05.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: pgmap v311: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:40:05.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: pgmap v311: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:40:05.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:04 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:40:05.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:05.276 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 66037 66021 108.574 82.5625 0.0114377 0.00920504 2024-04-25T05:40:05.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:05.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:05.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:05.521 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:05.528 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:05.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:05.664 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:05.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:05.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:05.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:05.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:05.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:05.942 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:05 smithi060 ceph-mon[30898]: osdmap e221: 8 total, 8 up, 7 in 2024-04-25T05:40:06.034 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:05 smithi043 ceph-mon[27010]: osdmap e221: 8 total, 8 up, 7 in 2024-04-25T05:40:06.034 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:05 smithi043 ceph-mon[37492]: osdmap e221: 8 total, 8 up, 7 in 2024-04-25T05:40:06.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:06.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:06.168 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:06.169 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:06.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:06.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.213 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:06.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:06.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 67376 67360 107.936 83.6875 0.0106638 0.00925982 2024-04-25T05:40:06.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.352 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:06.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:06.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:06.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:06.608 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:06.710 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.734 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:06.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:06.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:06.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:06.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:06.972 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:06.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:06 smithi060 ceph-mon[30898]: pgmap v313: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-04-25T05:40:07.001 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.016 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:07.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:06 smithi043 ceph-mon[27010]: pgmap v313: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-04-25T05:40:07.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:06 smithi043 ceph-mon[37492]: pgmap v313: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-04-25T05:40:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:07.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:40:07.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:07.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:40:07.276291+0000 min lat: 0.00119975 max lat: 1.57603 avg lat: 0.00931497 2024-04-25T05:40:07.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:40:07.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 68696 68680 107.299 82.5 0.0165218 0.00931497 2024-04-25T05:40:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:07.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:07.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:07.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:07.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:07.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:07.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:07.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:07.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:07.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:07.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:40:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:40:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:40:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:40:07.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:40:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:40:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:40:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:07 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:40:07.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:07.989 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:07 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:40:07.990 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:07 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:40:07.990 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:07 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:40:07.990 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:07 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:40:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:08.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:40:08.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:08.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:08.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 70032 70016 106.718 83.5 0.0117169 0.00936525 2024-04-25T05:40:08.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:08.350 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:08.350 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:08.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:08.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:08.506 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:08.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.579 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:08.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:08.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:08.717 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:08.726 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 0.610751 (8400262) 2024-04-25T05:40:08.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:08.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:40:08.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:40:08.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:40:08.933 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: pgmap v314: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3111778127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: pgmap v314: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:40:08.934 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3111778127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:08.935 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:08.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:08.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:08.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: pgmap v314: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3111778127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]: dispatch 2024-04-25T05:40:09.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:09.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:09.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:09.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 71361 71345 106.155 83.0625 0.00913326 0.00941487 2024-04-25T05:40:09.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:09.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:09.415 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:09.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:40:09.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:09.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:09.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:09.703 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:09.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:09.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:09.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:09.900 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:09.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:09.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:09.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:09.986 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:09.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]': finished 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: osdmap e222: 8 total, 8 up, 7 in 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/304720796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]': finished 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: osdmap e222: 8 total, 8 up, 7 in 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:40:09.988 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:40:09.989 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:40:09.989 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:40:09.989 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/304720796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.6107509478453164}]': finished 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: osdmap e222: 8 total, 8 up, 7 in 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:40:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/304720796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:10.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:10.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:10.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:10.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:10.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 72698 72682 105.629 83.5625 0.0133391 0.00946213 2024-04-25T05:40:10.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:10.392 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:10.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:10.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:10.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.655 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:10.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:10.721 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:10.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:10.788 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:10.818 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:10.818 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:40:10.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:40:10.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:129259 7.0 7:050cda31:::benchmark_data_smithi043_74356_object12390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 188.21560719999999, 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098052218999999996, 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129259 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347242+0000", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347244+0000", 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.364e-06 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347250+0000", 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1349999999999997e-06 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347266+0000", 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5911000000000001e-05 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347274+0000", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1769999999999998e-06 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347332+0000", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7521e-05 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347467+0000", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013588000000000001 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.347601+0000", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013331399999999999 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.350192+0000", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002590958 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.353155+0000", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029637579999999999 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445182+0000", 2024-04-25T05:40:10.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.092026636999999994 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445245+0000", 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3274000000000004e-05 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.445295+0000", 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-05 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130360 7.0 7:05078405:::benchmark_data_smithi043_74356_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 186.65011813500001, 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100930387, 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130360 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912731+0000", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912733+0000", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.516e-06 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912738+0000", 2024-04-25T05:40:10.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0490000000000003e-06 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912749+0000", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0635999999999999e-05 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912755+0000", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9429999999999997e-06 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912807+0000", 2024-04-25T05:40:10.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2732000000000003e-05 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912916+0000", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109107 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913025+0000", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108824 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919839+0000", 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068137730000000004 2024-04-25T05:40:10.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998798+0000", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078959219999999997 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013581+0000", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014783053000000001 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013621+0000", 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9190999999999998e-05 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013662+0000", 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1343e-05 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130361 7.0 7:0c1643bc:::benchmark_data_smithi043_74356_object13492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 186.650054633, 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.101036217, 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.830 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130361 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912795+0000", 2024-04-25T05:40:10.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912796+0000", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7999999999999995e-07 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912802+0000", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6789999999999999e-06 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912809+0000", 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9929999999999998e-06 2024-04-25T05:40:10.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.912821+0000", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1637999999999999e-05 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915433+0000", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026120610000000002 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915541+0000", 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108274 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.915609+0000", 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8141999999999999e-05 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919843+0000", 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042331419999999996 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998857+0000", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079014847999999999 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013723+0000", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014866047 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013784+0000", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0198999999999997e-05 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013831+0000", 2024-04-25T05:40:10.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7614000000000002e-05 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130363 7.0 7:0185dbbe:::benchmark_data_smithi043_74356_object13494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 186.649523668, 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10060899700000001, 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.836 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130363 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913326+0000", 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913327+0000", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1999999999999999e-07 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913331+0000", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0940000000000001e-06 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913342+0000", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0767999999999999e-05 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.913359+0000", 2024-04-25T05:40:10.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7198999999999999e-05 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917317+0000", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039578080000000002 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917403+0000", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6774999999999995e-05 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.917472+0000", 2024-04-25T05:40:10.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8664000000000005e-05 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919845+0000", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002373165 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.998895+0000", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.079049509000000004 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013869+0000", 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014974315 2024-04-25T05:40:10.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013898+0000", 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9014999999999999e-05 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013935+0000", 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6965000000000003e-05 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130364 7.0 7:00060f6c:::benchmark_data_smithi043_74356_object13495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.841 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 186.64840335599999, 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099641968999999997, 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130364 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:40:10.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914446+0000", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914447+0000", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6599999999999995e-07 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914451+0000", 2024-04-25T05:40:10.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7570000000000002e-06 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914460+0000", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2329999999999995e-06 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.914467+0000", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3130000000000001e-06 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918588+0000", 2024-04-25T05:40:10.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041202340000000004 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918660+0000", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2220999999999999e-05 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.918726+0000", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5771000000000004e-05 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:03.919932+0000", 2024-04-25T05:40:10.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012063740000000001 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.002732+0000", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082799625000000002 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.013993+0000", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011261567 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.014048+0000", 2024-04-25T05:40:10.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4951999999999998e-05 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.014088+0000", 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0156e-05 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130499 7.0 7:06be1c12:::benchmark_data_smithi043_74356_object13630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 186.43569201, 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.113647552, 2024-04-25T05:40:10.847 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130499 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127158+0000", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127159+0000", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1689999999999999e-06 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127163+0000", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4950000000000002e-06 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127173+0000", 2024-04-25T05:40:10.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086e-05 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127179+0000", 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127225+0000", 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5946999999999999e-05 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127358+0000", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013307600000000001 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127504+0000", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014553799999999999 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.130178+0000", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002674091 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.233209+0000", 2024-04-25T05:40:10.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10303095499999999 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240692+0000", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0074828560000000004 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240751+0000", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9342000000000002e-05 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.240805+0000", 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4143e-05 2024-04-25T05:40:10.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:143465 7.0 7:0114b0cc:::benchmark_data_smithi043_74356_object26596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 169.78781920599999, 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098687036000000006, 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 143465 2024-04-25T05:40:10.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775030+0000", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775032+0000", 2024-04-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.113e-06 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775038+0000", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0249999999999999e-06 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775050+0000", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2473999999999999e-05 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775059+0000", 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4199999999999996e-06 2024-04-25T05:40:10.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775109+0000", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9225999999999999e-05 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775244+0000", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000135537 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.775370+0000", 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012596100000000001 2024-04-25T05:40:10.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.777416+0000", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020459409999999999 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.833867+0000", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.056450475 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873607+0000", 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039740635000000003 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873672+0000", 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4692999999999999e-05 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:20.873717+0000", 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5535999999999999e-05 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:163814 7.0 7:09b1f4e8:::benchmark_data_smithi043_74356_object46945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:40:10.858 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.20872852900001, 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082701040000000003, 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 163814 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354121+0000", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354123+0000", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5579999999999999e-06 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354130+0000", 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3900000000000004e-06 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354148+0000", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8185999999999998e-05 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354159+0000", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0832999999999999e-05 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354211+0000", 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1708e-05 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354308+0000", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7609999999999995e-05 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.354403+0000", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4476999999999997e-05 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.362773+0000", 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0083697449999999996 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.424762+0000", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.061988986000000003 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436757+0000", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011995176999999999 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436792+0000", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5092000000000002e-05 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436822+0000", 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0278e-05 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:163817 7.0 7:04abd7d8:::benchmark_data_smithi043_74356_object46948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.20777175800001, 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.081835871000000004, 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 163817 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355078+0000", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355079+0000", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.61e-07 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355084+0000", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9230000000000001e-06 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355094+0000", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0546e-05 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.355100+0000", 2024-04-25T05:40:10.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7250000000000002e-06 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357186+0000", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020863679999999999 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357361+0000", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175024 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.357481+0000", 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119339 2024-04-25T05:40:10.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.362780+0000", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005299158 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.424826+0000", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.062046077999999998 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436861+0000", 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012035426 2024-04-25T05:40:10.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436891+0000", 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9765e-05 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:48.436914+0000", 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2758000000000001e-05 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.869 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:175169 7.0 7:0d4e44d2:::benchmark_data_smithi043_74356_object58300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 127.08331880599999, 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13254632699999999, 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175169 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479532+0000", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.288e-06 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479537+0000", 2024-04-25T05:40:10.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9239999999999998e-06 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479548+0000", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185e-05 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479556+0000", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3139999999999998e-06 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479603+0000", 2024-04-25T05:40:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7651000000000002e-05 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479689+0000", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5328999999999996e-05 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479769+0000", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0508999999999998e-05 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.481886+0000", 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021172999999999999 2024-04-25T05:40:10.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.492164+0000", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010277632 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612007+0000", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.119842849 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612041+0000", 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3958999999999998e-05 2024-04-25T05:40:10.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612077+0000", 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6387000000000002e-05 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195392 7.0 7:00172623:::benchmark_data_smithi043_74356_object78523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 99.815343506000005, 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.091797856999999997, 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.875 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195392 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747506+0000", 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747508+0000", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2330000000000001e-06 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.747518+0000", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3039999999999994e-06 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760394+0000", 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012876258 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760415+0000", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0922000000000001e-05 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760811+0000", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039616199999999998 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.760938+0000", 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012737299999999999 2024-04-25T05:40:10.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.761064+0000", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012562900000000001 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.766816+0000", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057523770000000004 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.831492+0000", 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.064675361000000001 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839182+0000", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0076902949999999998 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839272+0000", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0019000000000007e-05 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:30.839304+0000", 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1924000000000001e-05 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.880 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195836 7.0 7:0dc9af6b:::benchmark_data_smithi043_74356_object78967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 99.307036624999995, 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.103185819, 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195836 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255813+0000", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255814+0000", 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0240000000000001e-06 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255819+0000", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8130000000000003e-06 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255830+0000", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0781000000000001e-05 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255836+0000", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4729999999999997e-06 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255883+0000", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6786999999999998e-05 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.255989+0000", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010607600000000001 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256077+0000", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8104999999999998e-05 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.257931+0000", 2024-04-25T05:40:10.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018537289999999999 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.261718+0000", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003787159 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358884+0000", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097166081000000001 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358938+0000", 2024-04-25T05:40:10.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4336999999999998e-05 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.358999+0000", 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0454000000000001e-05 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:215815 7.0 7:0c272839:::benchmark_data_smithi043_74356_object98946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:49.767697+0000", 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 80.795153103000004, 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12199568399999999, 2024-04-25T05:40:10.886 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 215815 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767697+0000", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767697+0000", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767698+0000", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0279999999999999e-06 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767701+0000", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.512e-06 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767710+0000", 2024-04-25T05:40:10.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6789999999999994e-06 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767715+0000", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0839999999999999e-06 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767752+0000", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7203000000000003e-05 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767838+0000", 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6007999999999994e-05 2024-04-25T05:40:10.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767905+0000", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6729000000000004e-05 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.771122+0000", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032174460000000001 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.843317+0000", 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072194581999999993 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889626+0000", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.046309013000000003 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889660+0000", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4173e-05 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889692+0000", 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2227000000000002e-05 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:215816 7.0 7:0d74933b:::benchmark_data_smithi043_74356_object98947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:49.767898+0000", 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 80.794952076000001, 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12203220400000001, 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 215816 2024-04-25T05:40:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767898+0000", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767898+0000", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767898+0000", 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.46e-07 2024-04-25T05:40:10.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767901+0000", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5500000000000001e-06 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767907+0000", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0700000000000003e-06 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.767911+0000", 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7029999999999997e-06 2024-04-25T05:40:10.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.769874+0000", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019627899999999998 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.769961+0000", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7149000000000002e-05 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.770063+0000", 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010184 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.771200+0000", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011365349999999999 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.843353+0000", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072152885 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889864+0000", 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.046511916 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889895+0000", 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0599999999999998e-05 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:49.889930+0000", 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.472e-05 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:72929 8.c e223/198 8:3c7ed20e:::benchmark_data_smithi043_104001_object72928:head v 223'4567, mlcod=223'4563)", 2024-04-25T05:40:10.897 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.440982+0000", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.121867459, 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.086831979000000004, 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.440982+0000", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.440982+0000", 2024-04-25T05:40:10.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.440983+0000", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8500000000000001e-07 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.441014+0000", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1454000000000001e-05 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.441024+0000", 2024-04-25T05:40:10.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7170000000000003e-06 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.441031+0000", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6830000000000002e-06 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.445113+0000", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004082072 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.445133+0000", 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0123999999999999e-05 2024-04-25T05:40:10.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.527776+0000", 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082642934000000001 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.527814+0000", 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8309999999999997e-05 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.901 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:72925 8.9 e223/208 8:916c4b15:::benchmark_data_smithi043_104001_object72924:head v 223'4568, mlcod=223'4566)", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.442187+0000", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.120662535, 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.083611879, 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442187+0000", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442187+0000", 2024-04-25T05:40:10.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442188+0000", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3099999999999997e-07 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442235+0000", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7401000000000002e-05 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442244+0000", 2024-04-25T05:40:10.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7469999999999994e-06 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442251+0000", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2069999999999998e-06 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442383+0000", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131452 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.442401+0000", 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8471e-05 2024-04-25T05:40:10.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.525750+0000", 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.083348958000000001 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.525799+0000", 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9011999999999998e-05 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.905 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:72930 8.0 e223/198 8:0911f971:::benchmark_data_smithi043_104001_object72929:head v 223'4553, mlcod=223'4550)", 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.444588+0000", 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.118261936, 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.084913208000000004, 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444588+0000", 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444588+0000", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444588+0000", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4299999999999999e-07 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444617+0000", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9393000000000001e-05 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444623+0000", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8100000000000003e-06 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444628+0000", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9849999999999997e-06 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444691+0000", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2982000000000001e-05 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.444714+0000", 2024-04-25T05:40:10.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2946e-05 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.529462+0000", 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.084747490999999994 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.529501+0000", 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9258000000000003e-05 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.909 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:72937 8.8 8:1ab337a5:::benchmark_data_smithi043_104001_object72936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.447223+0000", 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.115626223, 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082378543999999998, 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 72937 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447223+0000", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447223+0000", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447225+0000", 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1400000000000001e-06 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447266+0000", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1876999999999999e-05 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447277+0000", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0740000000000001e-05 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447295+0000", 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7927999999999998e-05 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447526+0000", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023124699999999999 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447786+0000", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025988399999999999 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.447940+0000", 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000154189 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.526300+0000", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078359749000000006 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.526346+0000", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5735999999999997e-05 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.529510+0000", 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031643119999999999 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.529558+0000", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7540000000000002e-05 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.529602+0000", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4202000000000001e-05 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:72939 8.1 e223/198 8:84ad0433:::benchmark_data_smithi043_104001_object72938:head v 223'4669, mlcod=223'4667)", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.450676+0000", 2024-04-25T05:40:10.915 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.112173195, 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082018507000000004, 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.450676+0000", 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.450676+0000", 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.450677+0000", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3900000000000003e-07 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.450719+0000", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2110999999999997e-05 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.523225+0000", 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072505112999999996 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.523245+0000", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0146999999999999e-05 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.524619+0000", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013746279999999999 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.524639+0000", 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9541000000000001e-05 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.532587+0000", 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0079483950000000005 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.532695+0000", 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107633 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:72943 8.2 e223/198 8:4d275e0c:::benchmark_data_smithi043_104001_object72942:head v 223'4524, mlcod=223'4522)", 2024-04-25T05:40:10.919 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:10.451222+0000", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.111627701, 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.092385969999999998, 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.451222+0000", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.451222+0000", 2024-04-25T05:40:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.451223+0000", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2399999999999997e-07 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.451248+0000", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5567000000000001e-05 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.523429+0000", 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072180566000000002 2024-04-25T05:40:10.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.523434+0000", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9200000000000003e-06 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.525981+0000", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025472490000000001 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.525999+0000", 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8389e-05 2024-04-25T05:40:10.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.543582+0000", 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.017583046000000001 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:10.543608+0000", 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5409e-05 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:10.923 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:10.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:10.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:10 smithi043 ceph-mon[27010]: osdmap e223: 8 total, 8 up, 7 in 2024-04-25T05:40:10.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:10 smithi043 ceph-mon[27010]: pgmap v317: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:40:10.925 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:10 smithi043 ceph-mon[37492]: osdmap e223: 8 total, 8 up, 7 in 2024-04-25T05:40:10.925 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:10 smithi043 ceph-mon[37492]: pgmap v317: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:40:10.959 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:10.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:10.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:10 smithi060 ceph-mon[30898]: osdmap e223: 8 total, 8 up, 7 in 2024-04-25T05:40:10.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:10 smithi060 ceph-mon[30898]: pgmap v317: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:40:11.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:11.094 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:11.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:11.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:11.275 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 73937 73921 104.989 77.4375 0.005207 0.00951903 2024-04-25T05:40:11.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:11.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:11.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:11.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:11.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:11.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:11.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:40:11.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:11.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:11.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:11.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 6.17 scrub starts 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 6.17 scrub ok 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:40:11.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:11 smithi060 ceph-mon[30898]: 6.b scrub starts 2024-04-25T05:40:12.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:12.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:40:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:40:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 6.17 scrub starts 2024-04-25T05:40:12.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 6.17 scrub ok 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[27010]: 6.b scrub starts 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 6.17 scrub starts 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 6.17 scrub ok 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:40:12.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:11 smithi043 ceph-mon[37492]: 6.b scrub starts 2024-04-25T05:40:12.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:12.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:12.207 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:12.276 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 75253 75237 104.483 82.25 0.00990383 0.00956518 2024-04-25T05:40:12.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:12.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:12.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.482 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:12.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:12.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:12.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:12.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:40:12.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:12.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:12.766 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:40:12.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:12.899 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:12.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: 6.b scrub ok 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: pgmap v318: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 721 B/s rd, 59 MiB/s wr, 936 op/s 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1055703462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:12 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:13.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:13.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:13.124 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:40:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:76409 8.9 8:9a9916da:::benchmark_data_smithi043_104001_object76408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:13.116638+0000", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0062341810000000001, 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062718009999999996, 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 76409 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116638+0000", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116638+0000", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116639+0000", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0109999999999999e-06 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116683+0000", 2024-04-25T05:40:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3692000000000001e-05 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116698+0000", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5157000000000001e-05 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116707+0000", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8829999999999995e-06 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116777+0000", 2024-04-25T05:40:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0119999999999999e-05 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116930+0000", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015327299999999999 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.117154+0000", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000223582 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120276+0000", 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003122531 2024-04-25T05:40:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120398+0000", 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121458 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:76410 8.5 8:a6e11d37:::benchmark_data_smithi043_104001_object76409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:13.116730+0000", 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0061418870000000004, 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062899549999999999, 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 76410 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116730+0000", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116730+0000", 2024-04-25T05:40:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116731+0000", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0699999999999997e-07 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116764+0000", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2610000000000001e-05 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116775+0000", 2024-04-25T05:40:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0917999999999999e-05 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.116783+0000", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0320000000000003e-06 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120101+0000", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033186000000000001 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120244+0000", 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014285900000000001 2024-04-25T05:40:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120613+0000", 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036857600000000002 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:76406 8.5 e224/209)", 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:13.120245+0000", 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.002627243, 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002849978, 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120245+0000", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120245+0000", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120246+0000", 2024-04-25T05:40:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0890000000000001e-06 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120252+0000", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7450000000000001e-06 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120271+0000", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9415e-05 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.120279+0000", 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3710000000000002e-06 2024-04-25T05:40:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:76415 8.3 e224/208 8:c5bdae88:::benchmark_data_smithi043_104001_object76414:head v 224'4684, mlcod=224'4682)", 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:13.121533+0000", 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0013390520000000001, 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001615294, 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121533+0000", 2024-04-25T05:40:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121533+0000", 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121534+0000", 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5400000000000001e-07 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121586+0000", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1856e-05 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121599+0000", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3399e-05 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121606+0000", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6590000000000001e-06 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121669+0000", 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2723000000000002e-05 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.121687+0000", 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8726999999999999e-05 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:76406 8.5 8:aac13138:::benchmark_data_smithi043_104001_object76405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:13.115005+0000", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0078676679999999995, 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0082147820000000003, 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 76406 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115005+0000", 2024-04-25T05:40:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115005+0000", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115006+0000", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.516e-06 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115048+0000", 2024-04-25T05:40:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1526000000000002e-05 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115114+0000", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6651999999999998e-05 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115125+0000", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0686999999999999e-05 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115338+0000", 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000212919 2024-04-25T05:40:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115486+0000", 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000148491 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:13.115651+0000", 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016434800000000001 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:40:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 5 2024-04-25T05:40:13.144 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:13.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: 6.b scrub ok 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: pgmap v318: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 721 B/s rd, 59 MiB/s wr, 936 op/s 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1055703462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: 6.b scrub ok 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: pgmap v318: 158 pgs: 158 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 721 B/s rd, 59 MiB/s wr, 936 op/s 2024-04-25T05:40:13.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:40:13.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1055703462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:13.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:12 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:13.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:13.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 76593 76577 104.032 83.75 0.0123265 0.00960774 2024-04-25T05:40:13.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:13.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:13.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:13.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:13.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:13.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:13.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:13.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:13.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:13.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:13.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:13.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:13.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:40:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:14.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:14.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:40:14.017 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: osdmap e224: 8 total, 8 up, 7 in 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:40:14.018 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:13 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:14.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: osdmap e224: 8 total, 8 up, 7 in 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:40:14.137 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: osdmap e224: 8 total, 8 up, 7 in 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:40:14.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:13 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:14.140 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:14.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:14.276 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 77927 77911 103.593 83.375 0.014841 0.009647 2024-04-25T05:40:14.280 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:14.304 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:14.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:14.389 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:14.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:14.557 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:14.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:14.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:14.768 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:40:14.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:40:14.768 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T05:40:14.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:40:14.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.829 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:14.885 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:14 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:40:14.885 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:14 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:40:14.885 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:14 smithi060 ceph-mon[30898]: pgmap v320: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-04-25T05:40:14.885 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:14 smithi060 ceph-mon[30898]: 5.13 scrub starts 2024-04-25T05:40:14.885 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:14 smithi060 ceph-mon[30898]: 5.13 scrub ok 2024-04-25T05:40:14.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:14.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:14.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:15.061 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[27010]: pgmap v320: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[27010]: 5.13 scrub starts 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[27010]: 5.13 scrub ok 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[37492]: pgmap v320: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[37492]: 5.13 scrub starts 2024-04-25T05:40:15.063 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:14 smithi043 ceph-mon[37492]: 5.13 scrub ok 2024-04-25T05:40:15.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:15.089 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:15.089 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:40:15.089 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:40:15.089 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:40:15.089 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:15.105 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:15.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:15.146 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:15.190 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:15.208 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:15.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:15.238 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:15.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 79144 79128 103.017 76.0625 0.0229468 0.00970175 2024-04-25T05:40:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:15.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:15.284 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:15.326 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:15.367 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:15.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:15.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:15.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:15.554 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:15.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:15.690 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:15.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:15.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 4.7 deep-scrub starts 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 4.7 deep-scrub ok 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: osdmap e225: 8 total, 8 up, 7 in 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/516410085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 4.1d scrub starts 2024-04-25T05:40:15.830 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:15 smithi060 ceph-mon[30898]: 4.1d scrub ok 2024-04-25T05:40:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:15.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:15.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:15.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:15.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:16.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 4.7 deep-scrub starts 2024-04-25T05:40:16.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 4.7 deep-scrub ok 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: osdmap e225: 8 total, 8 up, 7 in 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/516410085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 4.1d scrub starts 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[27010]: 4.1d scrub ok 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 4.7 deep-scrub starts 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 4.7 deep-scrub ok 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: osdmap e225: 8 total, 8 up, 7 in 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/516410085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:16.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 4.1d scrub starts 2024-04-25T05:40:16.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:15 smithi043 ceph-mon[37492]: 4.1d scrub ok 2024-04-25T05:40:16.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:40:16.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:16.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:16.276 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 80327 80311 102.424 73.9375 0.0202438 0.00975793 2024-04-25T05:40:16.304 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:16.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:16.407 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:16.509 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:16.548 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:16.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:16.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:16.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:16.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:16.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[27010]: pgmap v322: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:40:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:40:16.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:16 smithi043 ceph-mon[37492]: pgmap v322: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-25T05:40:16.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:16.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:16.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:17.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:17.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:17.100 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:40:17.100 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:40:17.100 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:40:17.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:16 smithi060 ceph-mon[30898]: pgmap v322: 158 pgs: 158 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-04-25T05:40:17.130 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:17.181 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:40:17.181 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 225, 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:40:14.753001+0000", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:39:29.246791+0000", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:40:17.182 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 8, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:40:17.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.184 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.185 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "174", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "171", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.186 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.187 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:40:17.188 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.1700000762939453, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.1700000762939453, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.189 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.190 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.191 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.191 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.191 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.191 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.191 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.192 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.193 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:40:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:40:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.201 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.202 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.203 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.204 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.205 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.206 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.207 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.208 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.7400000095367432, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.7400000095367432, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.4100000858306885, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.4100000858306885, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.209 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.210 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.211 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.212 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.213 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.214 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.4900000095367432, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.4900000095367432, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.216 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.217 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.218 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.219 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:40:17.220 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.221 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 8, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_1", 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:39:18.082095+0000", 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:40:17.222 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:40:17.223 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "201", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:40:17.224 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:40:17.225 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:40:17.226 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.9900000095367432, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.9900000095367432, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:40:17.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:40:17.228 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 157, 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:40:17.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:40:17.231 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.232 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:40:17.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.237 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.240 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.241 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.242 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:40:17.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.245 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0.610748291015625, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 207, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.249 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:40:17.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:40:17.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 222, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:40:17.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:39.615659+0000", 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:40:17.263 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:40:04.079146+0000", 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.264 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:28.355579+0000", 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:49.561033+0000", 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.265 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:40:13.986479+0000", 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.266 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:55.009819+0000", 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:19.229076+0000", 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:39:57.193068+0000", 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.2", 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.9", 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.12", 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.b", 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:40:17.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [], 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:40:17.273 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:40:17.274 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:40:17.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:17.276 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:40:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:17.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:17.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 15 81458 81443 101.79 70.75 0.00792473 0.00981926 2024-04-25T05:40:17.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:17.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:17.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:17.538 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:17.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:17.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:17.809 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:17.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:17.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:18.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:18.136 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:17 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:40:18.136 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:17 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:40:18.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/175044449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:40:18.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:18.137 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:18.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:40:18.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:40:18.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/175044449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:40:18.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:18.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:40:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:40:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/175044449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:40:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:18.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:18.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:18.225 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:18.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 82716 82700 101.334 78.5625 0.00947036 0.00986339 2024-04-25T05:40:18.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:18.361 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:18.361 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:18.361 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:18.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:40:18.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:18.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:18.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:18.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:18.750 INFO:tasks.thrashosds.thrasher:j is {'epoch': 225, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:40:14.753001+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:39:29.246791+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '171', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.740000009536743, 'score_stable': 2.740000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.4100000858306885, 'raw_score_stable': 2.4100000858306885, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-25T05:39:18.082095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 157, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.610748291015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 207, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:39:39.615659+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:40:04.079146+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:28.355579+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:49.561033+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:40:13.986479+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:55.009819+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:19.229076+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:39:57.193068+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:40:18.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:40:18.777 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:18.795 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:40:18.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:18.911 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:18.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:19.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:19.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:19.076 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:18 smithi060 ceph-mon[30898]: pgmap v323: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-25T05:40:19.076 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:18 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:40:19.076 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:18 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:40:19.076 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:19.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:40:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[27010]: pgmap v323: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-25T05:40:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:40:19.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:40:19.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[37492]: pgmap v323: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-04-25T05:40:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:40:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:40:19.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:19.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:19.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:19.277 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 84001 83985 100.93 80.3125 0.0184187 0.0099024 2024-04-25T05:40:19.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:19.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:19.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.491 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:19.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:19.560 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:19.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:19.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:19.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:19.670 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:19.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:19.781 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.787 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:19.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:19.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:19.920 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:19.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:19.994 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:40:19.995 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:19.995 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:19 smithi060 ceph-mon[30898]: osdmap e226: 8 total, 8 up, 7 in 2024-04-25T05:40:20.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:20.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[27010]: osdmap e226: 8 total, 8 up, 7 in 2024-04-25T05:40:20.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:40:20.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:40:20.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:40:20.050 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:40:20.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:40:20.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:20.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:40:20.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1529140729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:20.051 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:19 smithi043 ceph-mon[37492]: osdmap e226: 8 total, 8 up, 7 in 2024-04-25T05:40:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:20.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:20.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:20.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:20.287 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:20.295 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 85344 85328 100.576 83.9375 0.0277132 0.0099373 2024-04-25T05:40:20.323 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:20.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:20.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:20.494 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:20.495 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:20.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:20.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:20.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:20.734 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:20.746 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:20.791 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:20.833 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:20.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:20.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:20.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:20.880 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:20.916 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:20.957 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:20.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:21.003 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:21.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[27010]: pgmap v325: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 145 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-25T05:40:21.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[27010]: osdmap e227: 8 total, 8 up, 7 in 2024-04-25T05:40:21.003 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1258488907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:21.004 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[37492]: pgmap v325: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 145 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-25T05:40:21.004 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[37492]: osdmap e227: 8 total, 8 up, 7 in 2024-04-25T05:40:21.004 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1258488907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:21.032 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:40:21.039 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:40:21.050 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:21.054 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:21.055 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:21.055 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:21.055 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:21.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:21.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:21.196 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:20 smithi060 ceph-mon[30898]: pgmap v325: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 145 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-25T05:40:21.196 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:20 smithi060 ceph-mon[30898]: osdmap e227: 8 total, 8 up, 7 in 2024-04-25T05:40:21.196 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1258488907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:21.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:21.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:40:21.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:21.295 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 86551 86535 100.11 75.4375 0.00686169 0.00998402 2024-04-25T05:40:21.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:21.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:21.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:21.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:40:21.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:21.682 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:21.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.846 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:21.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:21 smithi043 ceph-mon[27010]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:40:21.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:21 smithi043 ceph-mon[37492]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:40:21.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:21.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:21.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:22.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:22.124 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:21 smithi060 ceph-mon[30898]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:40:22.125 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:22.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:22.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:22.295 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 87825 87809 99.7377 79.625 0.00651264 0.0100212 2024-04-25T05:40:22.346 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:22.347 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:40:22.347 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:40:22.347 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:40:22.347 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:22.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:22.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:22.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:22.524 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:22.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:22.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:22.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:22.898 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:22.997 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:22 smithi060 ceph-mon[30898]: pgmap v327: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 729 B/s rd, 54 MiB/s wr, 872 op/s 2024-04-25T05:40:23.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:23.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:40:23.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:23.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:22 smithi043 ceph-mon[27010]: pgmap v327: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 729 B/s rd, 54 MiB/s wr, 872 op/s 2024-04-25T05:40:23.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:22 smithi043 ceph-mon[37492]: pgmap v327: 158 pgs: 158 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 729 B/s rd, 54 MiB/s wr, 872 op/s 2024-04-25T05:40:23.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:23.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:23.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:23.295 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 89084 89068 99.3618 78.6875 0.0143619 0.0100591 2024-04-25T05:40:23.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:23.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:23.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:23.509 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:40:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.522 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:23.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:23.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:23.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:23.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:23.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:23.865 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-25T05:40:19.805682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5393947163,"num_objects":82526,"num_object_clones":0,"num_object_copies":247578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82526,"num_whiteouts":0,"num_read":1360,"num_read_kb":1080,"num_write":82924,"num_write_kb":5266989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":134877822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73243,"ondisk_log_size":73243,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17310764,"kb_used_data":15663984,"kb_used_omap":10,"kb_used_meta":1646709,"kb_avail":638905300,"statfs":{"total":671965249536,"available":654239027200,"internally_reserved":0,"allocated":16039919616,"data_stored":16034586910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1686230123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":401014784,"num_objects":6119,"num_object_clones":0,"num_object_copies":18357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6119,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6119,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.024511"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775766+0000","last_change":"2024-04-25T05:40:09.734535+0000","last_active":"2024-04-25T05:40:14.775766+0000","last_peered":"2024-04-25T05:40:14.775766+0000","last_clean":"2024-04-25T05:40:14.775766+0000","last_became_active":"2024-04-25T05:40:09.734217+0000","last_became_peered":"2024-04-25T05:40:09.734217+0000","last_unstale":"2024-04-25T05:40:14.775766+0000","last_undegraded":"2024-04-25T05:40:14.775766+0000","last_fullsized":"2024-04-25T05:40:14.775766+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:49.635538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:49.635538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771273+0000","last_change":"2024-04-25T05:39:08.797448+0000","last_active":"2024-04-25T05:40:14.771273+0000","last_peered":"2024-04-25T05:40:14.771273+0000","last_clean":"2024-04-25T05:40:14.771273+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:40:14.771273+0000","last_undegraded":"2024-04-25T05:40:14.771273+0000","last_fullsized":"2024-04-25T05:40:14.771273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:08.797334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.797334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.918004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":157,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776945+0000","last_change":"2024-04-25T05:39:29.795039+0000","last_active":"2024-04-25T05:40:14.776945+0000","last_peered":"2024-04-25T05:40:14.776945+0000","last_clean":"2024-04-25T05:40:14.776945+0000","last_became_active":"2024-04-25T05:39:29.792781+0000","last_became_peered":"2024-04-25T05:39:29.792781+0000","last_unstale":"2024-04-25T05:40:14.776945+0000","last_undegraded":"2024-04-25T05:40:14.776945+0000","last_fullsized":"2024-04-25T05:40:14.776945+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":409,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769054+0000","last_change":"2024-04-25T05:39:42.695596+0000","last_active":"2024-04-25T05:40:14.769054+0000","last_peered":"2024-04-25T05:40:14.769054+0000","last_clean":"2024-04-25T05:40:14.769054+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:40:14.769054+0000","last_undegraded":"2024-04-25T05:40:14.769054+0000","last_fullsized":"2024-04-25T05:40:14.769054+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.695422+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:55.591355+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767078+0000","last_change":"2024-04-25T05:39:29.799097+0000","last_active":"2024-04-25T05:40:14.767078+0000","last_peered":"2024-04-25T05:40:14.767078+0000","last_clean":"2024-04-25T05:40:14.767078+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:40:14.767078+0000","last_undegraded":"2024-04-25T05:40:14.767078+0000","last_fullsized":"2024-04-25T05:40:14.767078+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.118690+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":313,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769377+0000","last_change":"2024-04-25T05:40:13.865954+0000","last_active":"2024-04-25T05:40:14.769377+0000","last_peered":"2024-04-25T05:40:14.769377+0000","last_clean":"2024-04-25T05:40:14.769377+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:40:14.769377+0000","last_undegraded":"2024-04-25T05:40:14.769377+0000","last_fullsized":"2024-04-25T05:40:14.769377+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:13.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:13.865834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:43.860963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765557+0000","last_change":"2024-04-25T05:39:29.797215+0000","last_active":"2024-04-25T05:40:14.765557+0000","last_peered":"2024-04-25T05:40:14.765557+0000","last_clean":"2024-04-25T05:40:14.765557+0000","last_became_active":"2024-04-25T05:39:29.796847+0000","last_became_peered":"2024-04-25T05:39:29.796847+0000","last_unstale":"2024-04-25T05:40:14.765557+0000","last_undegraded":"2024-04-25T05:40:14.765557+0000","last_fullsized":"2024-04-25T05:40:14.765557+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:27.201749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:27.201749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":436,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766249+0000","last_change":"2024-04-25T05:39:41.220106+0000","last_active":"2024-04-25T05:40:14.766249+0000","last_peered":"2024-04-25T05:40:14.766249+0000","last_clean":"2024-04-25T05:40:14.766249+0000","last_became_active":"2024-04-25T05:38:13.246798+0000","last_became_peered":"2024-04-25T05:38:13.246798+0000","last_unstale":"2024-04-25T05:40:14.766249+0000","last_undegraded":"2024-04-25T05:40:14.766249+0000","last_fullsized":"2024-04-25T05:40:14.766249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:39:41.219950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.219950+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.988927+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":150,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770929+0000","last_change":"2024-04-25T05:38:59.924478+0000","last_active":"2024-04-25T05:40:14.770929+0000","last_peered":"2024-04-25T05:40:14.770929+0000","last_clean":"2024-04-25T05:40:14.770929+0000","last_became_active":"2024-04-25T05:38:13.304062+0000","last_became_peered":"2024-04-25T05:38:13.304062+0000","last_unstale":"2024-04-25T05:40:14.770929+0000","last_undegraded":"2024-04-25T05:40:14.770929+0000","last_fullsized":"2024-04-25T05:40:14.770929+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:15.372015+0000","last_change":"2024-04-25T05:40:15.372049+0000","last_active":"2024-04-25T05:40:15.372015+0000","last_peered":"2024-04-25T05:40:15.372015+0000","last_clean":"2024-04-25T05:40:15.372015+0000","last_became_active":"2024-04-25T05:38:13.237479+0000","last_became_peered":"2024-04-25T05:38:13.237479+0000","last_unstale":"2024-04-25T05:40:15.372015+0000","last_undegraded":"2024-04-25T05:40:15.372015+0000","last_fullsized":"2024-04-25T05:40:15.372015+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:29.724242+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":365,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776783+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:40:14.776783+0000","last_peered":"2024-04-25T05:40:14.776783+0000","last_clean":"2024-04-25T05:40:14.776783+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:40:14.776783+0000","last_undegraded":"2024-04-25T05:40:14.776783+0000","last_fullsized":"2024-04-25T05:40:14.776783+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:42.090484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":76,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765637+0000","last_change":"2024-04-25T05:40:09.736112+0000","last_active":"2024-04-25T05:40:14.765637+0000","last_peered":"2024-04-25T05:40:14.765637+0000","last_clean":"2024-04-25T05:40:14.765637+0000","last_became_active":"2024-04-25T05:40:09.734944+0000","last_became_peered":"2024-04-25T05:40:09.734944+0000","last_unstale":"2024-04-25T05:40:14.765637+0000","last_undegraded":"2024-04-25T05:40:14.765637+0000","last_fullsized":"2024-04-25T05:40:14.765637+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775798+0000","last_change":"2024-04-25T05:37:36.187700+0000","last_active":"2024-04-25T05:40:14.775798+0000","last_peered":"2024-04-25T05:40:14.775798+0000","last_clean":"2024-04-25T05:40:14.775798+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:40:14.775798+0000","last_undegraded":"2024-04-25T05:40:14.775798+0000","last_fullsized":"2024-04-25T05:40:14.775798+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:36.187622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.187622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779568+0000","last_change":"2024-04-25T05:37:32.999333+0000","last_active":"2024-04-25T05:40:14.779568+0000","last_peered":"2024-04-25T05:40:14.779568+0000","last_clean":"2024-04-25T05:40:14.779568+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:40:14.779568+0000","last_undegraded":"2024-04-25T05:40:14.779568+0000","last_fullsized":"2024-04-25T05:40:14.779568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:32.999254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:32.999254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776033+0000","last_change":"2024-04-25T05:36:52.994419+0000","last_active":"2024-04-25T05:40:14.776033+0000","last_peered":"2024-04-25T05:40:14.776033+0000","last_clean":"2024-04-25T05:40:14.776033+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:40:14.776033+0000","last_undegraded":"2024-04-25T05:40:14.776033+0000","last_fullsized":"2024-04-25T05:40:14.776033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.994302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.994302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":57,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766405+0000","last_change":"2024-04-25T05:40:09.735486+0000","last_active":"2024-04-25T05:40:14.766405+0000","last_peered":"2024-04-25T05:40:14.766405+0000","last_clean":"2024-04-25T05:40:14.766405+0000","last_became_active":"2024-04-25T05:40:09.734815+0000","last_became_peered":"2024-04-25T05:40:09.734815+0000","last_unstale":"2024-04-25T05:40:14.766405+0000","last_undegraded":"2024-04-25T05:40:14.766405+0000","last_fullsized":"2024-04-25T05:40:14.766405+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:36:20.720604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.720604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.837413+0000","last_change":"2024-04-25T05:40:14.837450+0000","last_active":"2024-04-25T05:40:14.837413+0000","last_peered":"2024-04-25T05:40:14.837413+0000","last_clean":"2024-04-25T05:40:14.837413+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:40:14.837413+0000","last_undegraded":"2024-04-25T05:40:14.837413+0000","last_fullsized":"2024-04-25T05:40:14.837413+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.837215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.837215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:33.768510+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772551+0000","last_change":"2024-04-25T05:38:13.285789+0000","last_active":"2024-04-25T05:40:14.772551+0000","last_peered":"2024-04-25T05:40:14.772551+0000","last_clean":"2024-04-25T05:40:14.772551+0000","last_became_active":"2024-04-25T05:38:13.285434+0000","last_became_peered":"2024-04-25T05:38:13.285434+0000","last_unstale":"2024-04-25T05:40:14.772551+0000","last_undegraded":"2024-04-25T05:40:14.772551+0000","last_fullsized":"2024-04-25T05:40:14.772551+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:52.721698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:52.721698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:15.750292+0000","last_change":"2024-04-25T05:40:15.750335+0000","last_active":"2024-04-25T05:40:15.750292+0000","last_peered":"2024-04-25T05:40:15.750292+0000","last_clean":"2024-04-25T05:40:15.750292+0000","last_became_active":"2024-04-25T05:40:09.731002+0000","last_became_peered":"2024-04-25T05:40:09.731002+0000","last_unstale":"2024-04-25T05:40:15.750292+0000","last_undegraded":"2024-04-25T05:40:15.750292+0000","last_fullsized":"2024-04-25T05:40:15.750292+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:17.206142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":380,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766157+0000","last_change":"2024-04-25T05:39:42.757071+0000","last_active":"2024-04-25T05:40:14.766157+0000","last_peered":"2024-04-25T05:40:14.766157+0000","last_clean":"2024-04-25T05:40:14.766157+0000","last_became_active":"2024-04-25T05:34:00.690519+0000","last_became_peered":"2024-04-25T05:34:00.690519+0000","last_unstale":"2024-04-25T05:40:14.766157+0000","last_undegraded":"2024-04-25T05:40:14.766157+0000","last_fullsized":"2024-04-25T05:40:14.766157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:39:42.756928+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.756928+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.676365+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766345+0000","last_change":"2024-04-25T05:39:29.799140+0000","last_active":"2024-04-25T05:40:14.766345+0000","last_peered":"2024-04-25T05:40:14.766345+0000","last_clean":"2024-04-25T05:40:14.766345+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:40:14.766345+0000","last_undegraded":"2024-04-25T05:40:14.766345+0000","last_fullsized":"2024-04-25T05:40:14.766345+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.963090+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767278+0000","last_change":"2024-04-25T05:40:09.741925+0000","last_active":"2024-04-25T05:40:14.767278+0000","last_peered":"2024-04-25T05:40:14.767278+0000","last_clean":"2024-04-25T05:40:14.767278+0000","last_became_active":"2024-04-25T05:40:09.741588+0000","last_became_peered":"2024-04-25T05:40:09.741588+0000","last_unstale":"2024-04-25T05:40:14.767278+0000","last_undegraded":"2024-04-25T05:40:14.767278+0000","last_fullsized":"2024-04-25T05:40:14.767278+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":344,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771787+0000","last_change":"2024-04-25T05:40:12.775576+0000","last_active":"2024-04-25T05:40:14.771787+0000","last_peered":"2024-04-25T05:40:14.771787+0000","last_clean":"2024-04-25T05:40:14.771787+0000","last_became_active":"2024-04-25T05:33:58.677098+0000","last_became_peered":"2024-04-25T05:33:58.677098+0000","last_unstale":"2024-04-25T05:40:14.771787+0000","last_undegraded":"2024-04-25T05:40:14.771787+0000","last_fullsized":"2024-04-25T05:40:14.771787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.775467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.775467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:15.858860+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":368,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:15.536890+0000","last_change":"2024-04-25T05:40:15.536952+0000","last_active":"2024-04-25T05:40:15.536890+0000","last_peered":"2024-04-25T05:40:15.536890+0000","last_clean":"2024-04-25T05:40:15.536890+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:40:15.536890+0000","last_undegraded":"2024-04-25T05:40:15.536890+0000","last_fullsized":"2024-04-25T05:40:15.536890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:40:15.536802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.536802+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:25.840839+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":342,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782042+0000","last_change":"2024-04-25T05:40:10.850430+0000","last_active":"2024-04-25T05:40:14.782042+0000","last_peered":"2024-04-25T05:40:14.782042+0000","last_clean":"2024-04-25T05:40:14.782042+0000","last_became_active":"2024-04-25T05:34:04.725397+0000","last_became_peered":"2024-04-25T05:34:04.725397+0000","last_unstale":"2024-04-25T05:40:14.782042+0000","last_undegraded":"2024-04-25T05:40:14.782042+0000","last_fullsized":"2024-04-25T05:40:14.782042+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:40:10.849562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:10.849562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.994016+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":74,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770844+0000","last_change":"2024-04-25T05:39:38.521412+0000","last_active":"2024-04-25T05:40:14.770844+0000","last_peered":"2024-04-25T05:40:14.770844+0000","last_clean":"2024-04-25T05:40:14.770844+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:40:14.770844+0000","last_undegraded":"2024-04-25T05:40:14.770844+0000","last_fullsized":"2024-04-25T05:40:14.770844+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:39:38.521265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.521265+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.573956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":346,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771179+0000","last_change":"2024-04-25T05:36:50.826572+0000","last_active":"2024-04-25T05:40:14.771179+0000","last_peered":"2024-04-25T05:40:14.771179+0000","last_clean":"2024-04-25T05:40:14.771179+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:40:14.771179+0000","last_undegraded":"2024-04-25T05:40:14.771179+0000","last_fullsized":"2024-04-25T05:40:14.771179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:36:50.826445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:50.826445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":217,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782336+0000","last_change":"2024-04-25T05:39:02.184665+0000","last_active":"2024-04-25T05:40:14.782336+0000","last_peered":"2024-04-25T05:40:14.782336+0000","last_clean":"2024-04-25T05:40:14.782336+0000","last_became_active":"2024-04-25T05:38:13.281967+0000","last_became_peered":"2024-04-25T05:38:13.281967+0000","last_unstale":"2024-04-25T05:40:14.782336+0000","last_undegraded":"2024-04-25T05:40:14.782336+0000","last_fullsized":"2024-04-25T05:40:14.782336+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:15.703093+0000","last_change":"2024-04-25T05:40:15.703133+0000","last_active":"2024-04-25T05:40:15.703093+0000","last_peered":"2024-04-25T05:40:15.703093+0000","last_clean":"2024-04-25T05:40:15.703093+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:40:15.703093+0000","last_undegraded":"2024-04-25T05:40:15.703093+0000","last_fullsized":"2024-04-25T05:40:15.703093+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.702999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.702999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:22.622246+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":151,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782197+0000","last_change":"2024-04-25T05:39:04.108082+0000","last_active":"2024-04-25T05:40:14.782197+0000","last_peered":"2024-04-25T05:40:14.782197+0000","last_clean":"2024-04-25T05:40:14.782197+0000","last_became_active":"2024-04-25T05:38:13.282204+0000","last_became_peered":"2024-04-25T05:38:13.282204+0000","last_unstale":"2024-04-25T05:40:14.782197+0000","last_undegraded":"2024-04-25T05:40:14.782197+0000","last_fullsized":"2024-04-25T05:40:14.782197+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.765638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767759+0000","last_change":"2024-04-25T05:38:13.305988+0000","last_active":"2024-04-25T05:40:14.767759+0000","last_peered":"2024-04-25T05:40:14.767759+0000","last_clean":"2024-04-25T05:40:14.767759+0000","last_became_active":"2024-04-25T05:38:13.305466+0000","last_became_peered":"2024-04-25T05:38:13.305466+0000","last_unstale":"2024-04-25T05:40:14.767759+0000","last_undegraded":"2024-04-25T05:40:14.767759+0000","last_fullsized":"2024-04-25T05:40:14.767759+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:35:36.324465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":75,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771027+0000","last_change":"2024-04-25T05:39:40.474408+0000","last_active":"2024-04-25T05:40:14.771027+0000","last_peered":"2024-04-25T05:40:14.771027+0000","last_clean":"2024-04-25T05:40:14.771027+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:40:14.771027+0000","last_undegraded":"2024-04-25T05:40:14.771027+0000","last_fullsized":"2024-04-25T05:40:14.771027+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:40.474140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.474140+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:51.243595+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":334,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779787+0000","last_change":"2024-04-25T05:37:39.074683+0000","last_active":"2024-04-25T05:40:14.779787+0000","last_peered":"2024-04-25T05:40:14.779787+0000","last_clean":"2024-04-25T05:40:14.779787+0000","last_became_active":"2024-04-25T05:34:04.725374+0000","last_became_peered":"2024-04-25T05:34:04.725374+0000","last_unstale":"2024-04-25T05:40:14.779787+0000","last_undegraded":"2024-04-25T05:40:14.779787+0000","last_fullsized":"2024-04-25T05:40:14.779787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:37:39.074529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:39.074529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":53,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.773783+0000","last_change":"2024-04-25T05:39:29.802693+0000","last_active":"2024-04-25T05:40:14.773783+0000","last_peered":"2024-04-25T05:40:14.773783+0000","last_clean":"2024-04-25T05:40:14.773783+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:40:14.773783+0000","last_undegraded":"2024-04-25T05:40:14.773783+0000","last_fullsized":"2024-04-25T05:40:14.773783+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:27.553199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:27.553199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.099188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":156,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777016+0000","last_change":"2024-04-25T05:39:37.412459+0000","last_active":"2024-04-25T05:40:14.777016+0000","last_peered":"2024-04-25T05:40:14.777016+0000","last_clean":"2024-04-25T05:40:14.777016+0000","last_became_active":"2024-04-25T05:39:29.794349+0000","last_became_peered":"2024-04-25T05:39:29.794349+0000","last_unstale":"2024-04-25T05:40:14.777016+0000","last_undegraded":"2024-04-25T05:40:14.777016+0000","last_fullsized":"2024-04-25T05:40:14.777016+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:50.967870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":363,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767745+0000","last_change":"2024-04-25T05:39:29.824926+0000","last_active":"2024-04-25T05:40:14.767745+0000","last_peered":"2024-04-25T05:40:14.767745+0000","last_clean":"2024-04-25T05:40:14.767745+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:40:14.767745+0000","last_undegraded":"2024-04-25T05:40:14.767745+0000","last_fullsized":"2024-04-25T05:40:14.767745+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:36:19.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:19.405058+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767276+0000","last_change":"2024-04-25T05:39:29.799258+0000","last_active":"2024-04-25T05:40:14.767276+0000","last_peered":"2024-04-25T05:40:14.767276+0000","last_clean":"2024-04-25T05:40:14.767276+0000","last_became_active":"2024-04-25T05:39:29.798189+0000","last_became_peered":"2024-04-25T05:39:29.798189+0000","last_unstale":"2024-04-25T05:40:14.767276+0000","last_undegraded":"2024-04-25T05:40:14.767276+0000","last_fullsized":"2024-04-25T05:40:14.767276+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.160752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:17.103753+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":355,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782892+0000","last_change":"2024-04-25T05:39:29.797510+0000","last_active":"2024-04-25T05:40:14.782892+0000","last_peered":"2024-04-25T05:40:14.782892+0000","last_clean":"2024-04-25T05:40:14.782892+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:40:14.782892+0000","last_undegraded":"2024-04-25T05:40:14.782892+0000","last_fullsized":"2024-04-25T05:40:14.782892+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:00.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:00.574897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776684+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:40:14.776684+0000","last_peered":"2024-04-25T05:40:14.776684+0000","last_clean":"2024-04-25T05:40:14.776684+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:40:14.776684+0000","last_undegraded":"2024-04-25T05:40:14.776684+0000","last_fullsized":"2024-04-25T05:40:14.776684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.554804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":404,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766110+0000","last_change":"2024-04-25T05:39:29.832661+0000","last_active":"2024-04-25T05:40:14.766110+0000","last_peered":"2024-04-25T05:40:14.766110+0000","last_clean":"2024-04-25T05:40:14.766110+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:40:14.766110+0000","last_undegraded":"2024-04-25T05:40:14.766110+0000","last_fullsized":"2024-04-25T05:40:14.766110+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:35:38.337999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:38.337999+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":156,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777857+0000","last_change":"2024-04-25T05:39:29.797305+0000","last_active":"2024-04-25T05:40:14.777857+0000","last_peered":"2024-04-25T05:40:14.777857+0000","last_clean":"2024-04-25T05:40:14.777857+0000","last_became_active":"2024-04-25T05:39:29.794256+0000","last_became_peered":"2024-04-25T05:39:29.794256+0000","last_unstale":"2024-04-25T05:40:14.777857+0000","last_undegraded":"2024-04-25T05:40:14.777857+0000","last_fullsized":"2024-04-25T05:40:14.777857+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:20.270747+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":335,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766991+0000","last_change":"2024-04-25T05:40:14.526918+0000","last_active":"2024-04-25T05:40:14.766991+0000","last_peered":"2024-04-25T05:40:14.766991+0000","last_clean":"2024-04-25T05:40:14.766991+0000","last_became_active":"2024-04-25T05:38:13.286756+0000","last_became_peered":"2024-04-25T05:38:13.286756+0000","last_unstale":"2024-04-25T05:40:14.766991+0000","last_undegraded":"2024-04-25T05:40:14.766991+0000","last_fullsized":"2024-04-25T05:40:14.766991+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.526768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.526768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:41.727492+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779671+0000","last_change":"2024-04-25T05:37:06.741927+0000","last_active":"2024-04-25T05:40:14.779671+0000","last_peered":"2024-04-25T05:40:14.779671+0000","last_clean":"2024-04-25T05:40:14.779671+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:40:14.779671+0000","last_undegraded":"2024-04-25T05:40:14.779671+0000","last_fullsized":"2024-04-25T05:40:14.779671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:06.741806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:37:06.741806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":239,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775322+0000","last_change":"2024-04-25T05:39:29.809885+0000","last_active":"2024-04-25T05:40:14.775322+0000","last_peered":"2024-04-25T05:40:14.775322+0000","last_clean":"2024-04-25T05:40:14.775322+0000","last_became_active":"2024-04-25T05:39:29.792929+0000","last_became_peered":"2024-04-25T05:39:29.792929+0000","last_unstale":"2024-04-25T05:40:14.775322+0000","last_undegraded":"2024-04-25T05:40:14.775322+0000","last_fullsized":"2024-04-25T05:40:14.775322+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-04-25T05:37:42.212406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:37:42.212406+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":339,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765922+0000","last_change":"2024-04-25T05:40:04.446994+0000","last_active":"2024-04-25T05:40:14.765922+0000","last_peered":"2024-04-25T05:40:14.765922+0000","last_clean":"2024-04-25T05:40:14.765922+0000","last_became_active":"2024-04-25T05:38:13.246584+0000","last_became_peered":"2024-04-25T05:38:13.246584+0000","last_unstale":"2024-04-25T05:40:14.765922+0000","last_undegraded":"2024-04-25T05:40:14.765922+0000","last_fullsized":"2024-04-25T05:40:14.765922+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:04.446776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:04.446776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:14.008503+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771633+0000","last_change":"2024-04-25T05:40:06.644197+0000","last_active":"2024-04-25T05:40:14.771633+0000","last_peered":"2024-04-25T05:40:14.771633+0000","last_clean":"2024-04-25T05:40:14.771633+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:40:14.771633+0000","last_undegraded":"2024-04-25T05:40:14.771633+0000","last_fullsized":"2024-04-25T05:40:14.771633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.644083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.644083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:08.206753+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":25,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.781948+0000","last_change":"2024-04-25T05:40:09.728349+0000","last_active":"2024-04-25T05:40:14.781948+0000","last_peered":"2024-04-25T05:40:14.781948+0000","last_clean":"2024-04-25T05:40:14.781948+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:40:14.781948+0000","last_undegraded":"2024-04-25T05:40:14.781948+0000","last_fullsized":"2024-04-25T05:40:14.781948+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:39:00.543557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.543557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:19.650773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":413,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770341+0000","last_change":"2024-04-25T05:39:33.820019+0000","last_active":"2024-04-25T05:40:14.770341+0000","last_peered":"2024-04-25T05:40:14.770341+0000","last_clean":"2024-04-25T05:40:14.770341+0000","last_became_active":"2024-04-25T05:39:33.795217+0000","last_became_peered":"2024-04-25T05:39:33.795217+0000","last_unstale":"2024-04-25T05:40:14.770341+0000","last_undegraded":"2024-04-25T05:40:14.770341+0000","last_fullsized":"2024-04-25T05:40:14.770341+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:38:55.603358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.603358+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":150,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.781890+0000","last_change":"2024-04-25T05:39:43.132193+0000","last_active":"2024-04-25T05:40:14.781890+0000","last_peered":"2024-04-25T05:40:14.781890+0000","last_clean":"2024-04-25T05:40:14.781890+0000","last_became_active":"2024-04-25T05:38:13.246544+0000","last_became_peered":"2024-04-25T05:38:13.246544+0000","last_unstale":"2024-04-25T05:40:14.781890+0000","last_undegraded":"2024-04-25T05:40:14.781890+0000","last_fullsized":"2024-04-25T05:40:14.781890+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:08.353453+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"160'9","reported_seq":432,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776866+0000","last_change":"2024-04-25T05:38:55.388653+0000","last_active":"2024-04-25T05:40:14.776866+0000","last_peered":"2024-04-25T05:40:14.776866+0000","last_clean":"2024-04-25T05:40:14.776866+0000","last_became_active":"2024-04-25T05:38:13.295760+0000","last_became_peered":"2024-04-25T05:38:13.295760+0000","last_unstale":"2024-04-25T05:40:14.776866+0000","last_undegraded":"2024-04-25T05:40:14.776866+0000","last_fullsized":"2024-04-25T05:40:14.776866+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'9","last_scrub_stamp":"2024-04-25T05:38:55.388487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:55.388487+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779953+0000","last_change":"2024-04-25T05:37:37.076608+0000","last_active":"2024-04-25T05:40:14.779953+0000","last_peered":"2024-04-25T05:40:14.779953+0000","last_clean":"2024-04-25T05:40:14.779953+0000","last_became_active":"2024-04-25T05:33:58.685537+0000","last_became_peered":"2024-04-25T05:33:58.685537+0000","last_unstale":"2024-04-25T05:40:14.779953+0000","last_undegraded":"2024-04-25T05:40:14.779953+0000","last_fullsized":"2024-04-25T05:40:14.779953+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:37.076529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:37.076529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":372,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767186+0000","last_change":"2024-04-25T05:36:27.335735+0000","last_active":"2024-04-25T05:40:14.767186+0000","last_peered":"2024-04-25T05:40:14.767186+0000","last_clean":"2024-04-25T05:40:14.767186+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:40:14.767186+0000","last_undegraded":"2024-04-25T05:40:14.767186+0000","last_fullsized":"2024-04-25T05:40:14.767186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:27.335583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:27.335583+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":24,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765878+0000","last_change":"2024-04-25T05:40:09.913551+0000","last_active":"2024-04-25T05:40:14.765878+0000","last_peered":"2024-04-25T05:40:14.765878+0000","last_clean":"2024-04-25T05:40:14.765878+0000","last_became_active":"2024-04-25T05:40:09.722958+0000","last_became_peered":"2024-04-25T05:40:09.722958+0000","last_unstale":"2024-04-25T05:40:14.765878+0000","last_undegraded":"2024-04-25T05:40:14.765878+0000","last_fullsized":"2024-04-25T05:40:14.765878+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:09.913324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.913324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:30.051451+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":405,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767449+0000","last_change":"2024-04-25T05:34:02.703059+0000","last_active":"2024-04-25T05:40:14.767449+0000","last_peered":"2024-04-25T05:40:14.767449+0000","last_clean":"2024-04-25T05:40:14.767449+0000","last_became_active":"2024-04-25T05:34:02.702713+0000","last_became_peered":"2024-04-25T05:34:02.702713+0000","last_unstale":"2024-04-25T05:40:14.767449+0000","last_undegraded":"2024-04-25T05:40:14.767449+0000","last_fullsized":"2024-04-25T05:40:14.767449+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":150,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779434+0000","last_change":"2024-04-25T05:39:44.112703+0000","last_active":"2024-04-25T05:40:14.779434+0000","last_peered":"2024-04-25T05:40:14.779434+0000","last_clean":"2024-04-25T05:40:14.779434+0000","last_became_active":"2024-04-25T05:38:13.282059+0000","last_became_peered":"2024-04-25T05:38:13.282059+0000","last_unstale":"2024-04-25T05:40:14.779434+0000","last_undegraded":"2024-04-25T05:40:14.779434+0000","last_fullsized":"2024-04-25T05:40:14.779434+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:58.115488+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":374,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777210+0000","last_change":"2024-04-25T05:39:03.365135+0000","last_active":"2024-04-25T05:40:14.777210+0000","last_peered":"2024-04-25T05:40:14.777210+0000","last_clean":"2024-04-25T05:40:14.777210+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:40:14.777210+0000","last_undegraded":"2024-04-25T05:40:14.777210+0000","last_fullsized":"2024-04-25T05:40:14.777210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:39:03.364988+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.364988+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":340,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766385+0000","last_change":"2024-04-25T05:39:43.707526+0000","last_active":"2024-04-25T05:40:14.766385+0000","last_peered":"2024-04-25T05:40:14.766385+0000","last_clean":"2024-04-25T05:40:14.766385+0000","last_became_active":"2024-04-25T05:34:04.719279+0000","last_became_peered":"2024-04-25T05:34:04.719279+0000","last_unstale":"2024-04-25T05:40:14.766385+0000","last_undegraded":"2024-04-25T05:40:14.766385+0000","last_fullsized":"2024-04-25T05:40:14.766385+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.707414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.707414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.778892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":321,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766200+0000","last_change":"2024-04-25T05:34:02.698102+0000","last_active":"2024-04-25T05:40:14.766200+0000","last_peered":"2024-04-25T05:40:14.766200+0000","last_clean":"2024-04-25T05:40:14.766200+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:40:14.766200+0000","last_undegraded":"2024-04-25T05:40:14.766200+0000","last_fullsized":"2024-04-25T05:40:14.766200+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:34:01.680110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":363,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779130+0000","last_change":"2024-04-25T05:39:29.798683+0000","last_active":"2024-04-25T05:40:14.779130+0000","last_peered":"2024-04-25T05:40:14.779130+0000","last_clean":"2024-04-25T05:40:14.779130+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:40:14.779130+0000","last_undegraded":"2024-04-25T05:40:14.779130+0000","last_fullsized":"2024-04-25T05:40:14.779130+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:01.599324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:01.599324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":215,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765782+0000","last_change":"2024-04-25T05:38:13.255668+0000","last_active":"2024-04-25T05:40:14.765782+0000","last_peered":"2024-04-25T05:40:14.765782+0000","last_clean":"2024-04-25T05:40:14.765782+0000","last_became_active":"2024-04-25T05:38:13.245172+0000","last_became_peered":"2024-04-25T05:38:13.245172+0000","last_unstale":"2024-04-25T05:40:14.765782+0000","last_undegraded":"2024-04-25T05:40:14.765782+0000","last_fullsized":"2024-04-25T05:40:14.765782+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.778888+0000","last_change":"2024-04-25T05:39:29.799422+0000","last_active":"2024-04-25T05:40:14.778888+0000","last_peered":"2024-04-25T05:40:14.778888+0000","last_clean":"2024-04-25T05:40:14.778888+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:40:14.778888+0000","last_undegraded":"2024-04-25T05:40:14.778888+0000","last_fullsized":"2024-04-25T05:40:14.778888+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:41.064156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:37:41.064156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":329,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776613+0000","last_change":"2024-04-25T05:36:24.037537+0000","last_active":"2024-04-25T05:40:14.776613+0000","last_peered":"2024-04-25T05:40:14.776613+0000","last_clean":"2024-04-25T05:40:14.776613+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:40:14.776613+0000","last_undegraded":"2024-04-25T05:40:14.776613+0000","last_fullsized":"2024-04-25T05:40:14.776613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.037408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.918816+0000","last_change":"2024-04-25T05:40:14.918851+0000","last_active":"2024-04-25T05:40:14.918816+0000","last_peered":"2024-04-25T05:40:14.918816+0000","last_clean":"2024-04-25T05:40:14.918816+0000","last_became_active":"2024-04-25T05:39:29.795963+0000","last_became_peered":"2024-04-25T05:39:29.795963+0000","last_unstale":"2024-04-25T05:40:14.918816+0000","last_undegraded":"2024-04-25T05:40:14.918816+0000","last_fullsized":"2024-04-25T05:40:14.918816+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.918723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.918723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:29.385434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":354,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766401+0000","last_change":"2024-04-25T05:35:08.318259+0000","last_active":"2024-04-25T05:40:14.766401+0000","last_peered":"2024-04-25T05:40:14.766401+0000","last_clean":"2024-04-25T05:40:14.766401+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:40:14.766401+0000","last_undegraded":"2024-04-25T05:40:14.766401+0000","last_fullsized":"2024-04-25T05:40:14.766401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:35:08.318168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:08.318168+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"225'5149","reported_seq":5213,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.736135+0000","last_change":"2024-04-25T05:39:19.228971+0000","last_active":"2024-04-25T05:40:17.736135+0000","last_peered":"2024-04-25T05:40:17.736135+0000","last_clean":"2024-04-25T05:40:17.736135+0000","last_became_active":"2024-04-25T05:39:19.228682+0000","last_became_peered":"2024-04-25T05:39:19.228682+0000","last_unstale":"2024-04-25T05:40:17.736135+0000","last_undegraded":"2024-04-25T05:40:17.736135+0000","last_fullsized":"2024-04-25T05:40:17.736135+0000","mapping_epoch":198,"log_start":"210'800","ondisk_log_start":"210'800","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:33.132061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444864,"num_objects":5149,"num_object_clones":0,"num_object_copies":15447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":329536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":351,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776216+0000","last_change":"2024-04-25T05:39:29.794673+0000","last_active":"2024-04-25T05:40:14.776216+0000","last_peered":"2024-04-25T05:40:14.776216+0000","last_clean":"2024-04-25T05:40:14.776216+0000","last_became_active":"2024-04-25T05:39:29.792652+0000","last_became_peered":"2024-04-25T05:39:29.792652+0000","last_unstale":"2024-04-25T05:40:14.776216+0000","last_undegraded":"2024-04-25T05:40:14.776216+0000","last_fullsized":"2024-04-25T05:40:14.776216+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:55.046992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:55.046992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":369,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.780662+0000","last_change":"2024-04-25T05:38:13.313381+0000","last_active":"2024-04-25T05:40:14.780662+0000","last_peered":"2024-04-25T05:40:14.780662+0000","last_clean":"2024-04-25T05:40:14.780662+0000","last_became_active":"2024-04-25T05:38:13.295320+0000","last_became_peered":"2024-04-25T05:38:13.295320+0000","last_unstale":"2024-04-25T05:40:14.780662+0000","last_undegraded":"2024-04-25T05:40:14.780662+0000","last_fullsized":"2024-04-25T05:40:14.780662+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:37:36.045291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:36.045291+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":442,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777705+0000","last_change":"2024-04-25T05:39:30.386374+0000","last_active":"2024-04-25T05:40:14.777705+0000","last_peered":"2024-04-25T05:40:14.777705+0000","last_clean":"2024-04-25T05:40:14.777705+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:40:14.777705+0000","last_undegraded":"2024-04-25T05:40:14.777705+0000","last_fullsized":"2024-04-25T05:40:14.777705+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:39:30.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.386275+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:32.056327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":453,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779327+0000","last_change":"2024-04-25T05:39:33.128642+0000","last_active":"2024-04-25T05:40:14.779327+0000","last_peered":"2024-04-25T05:40:14.779327+0000","last_clean":"2024-04-25T05:40:14.779327+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:40:14.779327+0000","last_undegraded":"2024-04-25T05:40:14.779327+0000","last_fullsized":"2024-04-25T05:40:14.779327+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'10","last_scrub_stamp":"2024-04-25T05:39:33.128477+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.128477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:48.900732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"225'5135","reported_seq":5388,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.764085+0000","last_change":"2024-04-25T05:39:35.780958+0000","last_active":"2024-04-25T05:40:17.764085+0000","last_peered":"2024-04-25T05:40:17.764085+0000","last_clean":"2024-04-25T05:40:17.764085+0000","last_became_active":"2024-04-25T05:39:35.778659+0000","last_became_peered":"2024-04-25T05:39:35.778659+0000","last_unstale":"2024-04-25T05:40:17.764085+0000","last_undegraded":"2024-04-25T05:40:17.764085+0000","last_fullsized":"2024-04-25T05:40:17.764085+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":5135,"log_dups_size":0,"ondisk_log_size":5135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.504973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336527360,"num_objects":5135,"num_object_clones":0,"num_object_copies":15405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5135,"num_write_kb":328640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":331,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767178+0000","last_change":"2024-04-25T05:40:11.579293+0000","last_active":"2024-04-25T05:40:14.767178+0000","last_peered":"2024-04-25T05:40:14.767178+0000","last_clean":"2024-04-25T05:40:14.767178+0000","last_became_active":"2024-04-25T05:39:55.767650+0000","last_became_peered":"2024-04-25T05:39:55.767650+0000","last_unstale":"2024-04-25T05:40:14.767178+0000","last_undegraded":"2024-04-25T05:40:14.767178+0000","last_fullsized":"2024-04-25T05:40:14.767178+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.579078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.579078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:27.332807+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":353,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782160+0000","last_change":"2024-04-25T05:39:38.030757+0000","last_active":"2024-04-25T05:40:14.782160+0000","last_peered":"2024-04-25T05:40:14.782160+0000","last_clean":"2024-04-25T05:40:14.782160+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:40:14.782160+0000","last_undegraded":"2024-04-25T05:40:14.782160+0000","last_fullsized":"2024-04-25T05:40:14.782160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.030627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.030627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:49.843179+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":366,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.764494+0000","last_change":"2024-04-25T05:40:09.744437+0000","last_active":"2024-04-25T05:40:14.764494+0000","last_peered":"2024-04-25T05:40:14.764494+0000","last_clean":"2024-04-25T05:40:14.764494+0000","last_became_active":"2024-04-25T05:40:09.744079+0000","last_became_peered":"2024-04-25T05:40:09.744079+0000","last_unstale":"2024-04-25T05:40:14.764494+0000","last_undegraded":"2024-04-25T05:40:14.764494+0000","last_fullsized":"2024-04-25T05:40:14.764494+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:39:10.593601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:10.593601+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:15.116825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.764720+0000","last_change":"2024-04-25T05:39:39.789694+0000","last_active":"2024-04-25T05:40:14.764720+0000","last_peered":"2024-04-25T05:40:14.764720+0000","last_clean":"2024-04-25T05:40:14.764720+0000","last_became_active":"2024-04-25T05:38:13.302285+0000","last_became_peered":"2024-04-25T05:38:13.302285+0000","last_unstale":"2024-04-25T05:40:14.764720+0000","last_undegraded":"2024-04-25T05:40:14.764720+0000","last_fullsized":"2024-04-25T05:40:14.764720+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:46.925548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"225'5091","reported_seq":5966,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.789835+0000","last_change":"2024-04-25T05:39:34.781545+0000","last_active":"2024-04-25T05:40:17.789835+0000","last_peered":"2024-04-25T05:40:17.789835+0000","last_clean":"2024-04-25T05:40:17.789835+0000","last_became_active":"2024-04-25T05:39:34.778772+0000","last_became_peered":"2024-04-25T05:39:34.778772+0000","last_unstale":"2024-04-25T05:40:17.789835+0000","last_undegraded":"2024-04-25T05:40:17.789835+0000","last_fullsized":"2024-04-25T05:40:17.789835+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:44.773418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333643776,"num_objects":5091,"num_object_clones":0,"num_object_copies":15273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":325824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":26214400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.778797+0000","last_change":"2024-04-25T05:39:36.074193+0000","last_active":"2024-04-25T05:40:14.778797+0000","last_peered":"2024-04-25T05:40:14.778797+0000","last_clean":"2024-04-25T05:40:14.778797+0000","last_became_active":"2024-04-25T05:34:04.726805+0000","last_became_peered":"2024-04-25T05:34:04.726805+0000","last_unstale":"2024-04-25T05:40:14.778797+0000","last_undegraded":"2024-04-25T05:40:14.778797+0000","last_fullsized":"2024-04-25T05:40:14.778797+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.074043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.074043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:38.732336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765372+0000","last_change":"2024-04-25T05:39:40.796539+0000","last_active":"2024-04-25T05:40:14.765372+0000","last_peered":"2024-04-25T05:40:14.765372+0000","last_clean":"2024-04-25T05:40:14.765372+0000","last_became_active":"2024-04-25T05:38:13.236574+0000","last_became_peered":"2024-04-25T05:38:13.236574+0000","last_unstale":"2024-04-25T05:40:14.765372+0000","last_undegraded":"2024-04-25T05:40:14.765372+0000","last_fullsized":"2024-04-25T05:40:14.765372+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:41.763860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":379,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779871+0000","last_change":"2024-04-25T05:40:08.880333+0000","last_active":"2024-04-25T05:40:14.779871+0000","last_peered":"2024-04-25T05:40:14.779871+0000","last_clean":"2024-04-25T05:40:14.779871+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:40:14.779871+0000","last_undegraded":"2024-04-25T05:40:14.779871+0000","last_fullsized":"2024-04-25T05:40:14.779871+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:08.880207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.880207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:28.890375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":83,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769791+0000","last_change":"2024-04-25T05:39:30.591246+0000","last_active":"2024-04-25T05:40:14.769791+0000","last_peered":"2024-04-25T05:40:14.769791+0000","last_clean":"2024-04-25T05:40:14.769791+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:40:14.769791+0000","last_undegraded":"2024-04-25T05:40:14.769791+0000","last_fullsized":"2024-04-25T05:40:14.769791+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:39:30.591091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.591091+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:41.617716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"225'5209","reported_seq":6156,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.043493+0000","last_change":"2024-04-25T05:39:35.779484+0000","last_active":"2024-04-25T05:40:18.043493+0000","last_peered":"2024-04-25T05:40:18.043493+0000","last_clean":"2024-04-25T05:40:18.043493+0000","last_became_active":"2024-04-25T05:39:35.777746+0000","last_became_peered":"2024-04-25T05:39:35.777746+0000","last_unstale":"2024-04-25T05:40:18.043493+0000","last_undegraded":"2024-04-25T05:40:18.043493+0000","last_fullsized":"2024-04-25T05:40:18.043493+0000","mapping_epoch":209,"log_start":"210'900","ondisk_log_start":"210'900","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.582782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":333376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":150,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.778847+0000","last_change":"2024-04-25T05:39:06.050576+0000","last_active":"2024-04-25T05:40:14.778847+0000","last_peered":"2024-04-25T05:40:14.778847+0000","last_clean":"2024-04-25T05:40:14.778847+0000","last_became_active":"2024-04-25T05:38:13.240067+0000","last_became_peered":"2024-04-25T05:38:13.240067+0000","last_unstale":"2024-04-25T05:40:14.778847+0000","last_undegraded":"2024-04-25T05:40:14.778847+0000","last_fullsized":"2024-04-25T05:40:14.778847+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":336,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765464+0000","last_change":"2024-04-25T05:39:35.686849+0000","last_active":"2024-04-25T05:40:14.765464+0000","last_peered":"2024-04-25T05:40:14.765464+0000","last_clean":"2024-04-25T05:40:14.765464+0000","last_became_active":"2024-04-25T05:34:02.700504+0000","last_became_peered":"2024-04-25T05:34:02.700504+0000","last_unstale":"2024-04-25T05:40:14.765464+0000","last_undegraded":"2024-04-25T05:40:14.765464+0000","last_fullsized":"2024-04-25T05:40:14.765464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:35.686769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:35.686769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:59.630972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":357,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776051+0000","last_change":"2024-04-25T05:39:29.796897+0000","last_active":"2024-04-25T05:40:14.776051+0000","last_peered":"2024-04-25T05:40:14.776051+0000","last_clean":"2024-04-25T05:40:14.776051+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:40:14.776051+0000","last_undegraded":"2024-04-25T05:40:14.776051+0000","last_fullsized":"2024-04-25T05:40:14.776051+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:30.237597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:30.237597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":214,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772053+0000","last_change":"2024-04-25T05:39:41.766832+0000","last_active":"2024-04-25T05:40:14.772053+0000","last_peered":"2024-04-25T05:40:14.772053+0000","last_clean":"2024-04-25T05:40:14.772053+0000","last_became_active":"2024-04-25T05:38:13.294321+0000","last_became_peered":"2024-04-25T05:38:13.294321+0000","last_unstale":"2024-04-25T05:40:14.772053+0000","last_undegraded":"2024-04-25T05:40:14.772053+0000","last_fullsized":"2024-04-25T05:40:14.772053+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:45.433374+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"225'5068","reported_seq":5132,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.727348+0000","last_change":"2024-04-25T05:39:19.233374+0000","last_active":"2024-04-25T05:40:17.727348+0000","last_peered":"2024-04-25T05:40:17.727348+0000","last_clean":"2024-04-25T05:40:17.727348+0000","last_became_active":"2024-04-25T05:39:19.233055+0000","last_became_peered":"2024-04-25T05:39:19.233055+0000","last_unstale":"2024-04-25T05:40:17.727348+0000","last_undegraded":"2024-04-25T05:40:17.727348+0000","last_fullsized":"2024-04-25T05:40:17.727348+0000","mapping_epoch":198,"log_start":"207'700","ondisk_log_start":"207'700","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:20.380576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332136448,"num_objects":5068,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5068,"num_write_kb":324352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":520,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775427+0000","last_change":"2024-04-25T05:40:07.411902+0000","last_active":"2024-04-25T05:40:14.775427+0000","last_peered":"2024-04-25T05:40:14.775427+0000","last_clean":"2024-04-25T05:40:14.775427+0000","last_became_active":"2024-04-25T05:38:34.759567+0000","last_became_peered":"2024-04-25T05:38:34.759567+0000","last_unstale":"2024-04-25T05:40:14.775427+0000","last_undegraded":"2024-04-25T05:40:14.775427+0000","last_fullsized":"2024-04-25T05:40:14.775427+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:07.411609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:07.411609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:36.073862+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":325,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765884+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:40:14.765884+0000","last_peered":"2024-04-25T05:40:14.765884+0000","last_clean":"2024-04-25T05:40:14.765884+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:40:14.765884+0000","last_undegraded":"2024-04-25T05:40:14.765884+0000","last_fullsized":"2024-04-25T05:40:14.765884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":386,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775986+0000","last_change":"2024-04-25T05:39:29.827029+0000","last_active":"2024-04-25T05:40:14.775986+0000","last_peered":"2024-04-25T05:40:14.775986+0000","last_clean":"2024-04-25T05:40:14.775986+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:40:14.775986+0000","last_undegraded":"2024-04-25T05:40:14.775986+0000","last_fullsized":"2024-04-25T05:40:14.775986+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":327,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770489+0000","last_change":"2024-04-25T05:40:12.849822+0000","last_active":"2024-04-25T05:40:14.770489+0000","last_peered":"2024-04-25T05:40:14.770489+0000","last_clean":"2024-04-25T05:40:14.770489+0000","last_became_active":"2024-04-25T05:40:09.730862+0000","last_became_peered":"2024-04-25T05:40:09.730862+0000","last_unstale":"2024-04-25T05:40:14.770489+0000","last_undegraded":"2024-04-25T05:40:14.770489+0000","last_fullsized":"2024-04-25T05:40:14.770489+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.849716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.849716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:17.063987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"101'45","reported_seq":489,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771592+0000","last_change":"2024-04-25T05:39:42.724143+0000","last_active":"2024-04-25T05:40:14.771592+0000","last_peered":"2024-04-25T05:40:14.771592+0000","last_clean":"2024-04-25T05:40:14.771592+0000","last_became_active":"2024-04-25T05:34:00.688367+0000","last_became_peered":"2024-04-25T05:34:00.688367+0000","last_unstale":"2024-04-25T05:40:14.771592+0000","last_undegraded":"2024-04-25T05:40:14.771592+0000","last_fullsized":"2024-04-25T05:40:14.771592+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:39:42.723973+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.723973+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:05.917525+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":149,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772617+0000","last_change":"2024-04-25T05:39:29.798694+0000","last_active":"2024-04-25T05:40:14.772617+0000","last_peered":"2024-04-25T05:40:14.772617+0000","last_clean":"2024-04-25T05:40:14.772617+0000","last_became_active":"2024-04-25T05:39:29.798140+0000","last_became_peered":"2024-04-25T05:39:29.798140+0000","last_unstale":"2024-04-25T05:40:14.772617+0000","last_undegraded":"2024-04-25T05:40:14.772617+0000","last_fullsized":"2024-04-25T05:40:14.772617+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:24.213969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:24.213969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"225'5068","reported_seq":5132,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.147754+0000","last_change":"2024-04-25T05:39:19.228876+0000","last_active":"2024-04-25T05:40:18.147754+0000","last_peered":"2024-04-25T05:40:18.147754+0000","last_clean":"2024-04-25T05:40:18.147754+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:40:18.147754+0000","last_undegraded":"2024-04-25T05:40:18.147754+0000","last_fullsized":"2024-04-25T05:40:18.147754+0000","mapping_epoch":198,"log_start":"210'1000","ondisk_log_start":"210'1000","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.356518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332136448,"num_objects":5068,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5068,"num_write_kb":324352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":643,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.779692+0000","last_change":"2024-04-25T05:39:00.133654+0000","last_active":"2024-04-25T05:40:14.779692+0000","last_peered":"2024-04-25T05:40:14.779692+0000","last_clean":"2024-04-25T05:40:14.779692+0000","last_became_active":"2024-04-25T05:38:34.758423+0000","last_became_peered":"2024-04-25T05:38:34.758423+0000","last_unstale":"2024-04-25T05:40:14.779692+0000","last_undegraded":"2024-04-25T05:40:14.779692+0000","last_fullsized":"2024-04-25T05:40:14.779692+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:00.133580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.133580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.780034+0000","last_change":"2024-04-25T05:39:29.799473+0000","last_active":"2024-04-25T05:40:14.780034+0000","last_peered":"2024-04-25T05:40:14.780034+0000","last_clean":"2024-04-25T05:40:14.780034+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:40:14.780034+0000","last_undegraded":"2024-04-25T05:40:14.780034+0000","last_fullsized":"2024-04-25T05:40:14.780034+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.626788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.626788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771282+0000","last_change":"2024-04-25T05:39:29.792837+0000","last_active":"2024-04-25T05:40:14.771282+0000","last_peered":"2024-04-25T05:40:14.771282+0000","last_clean":"2024-04-25T05:40:14.771282+0000","last_became_active":"2024-04-25T05:39:29.792494+0000","last_became_peered":"2024-04-25T05:39:29.792494+0000","last_unstale":"2024-04-25T05:40:14.771282+0000","last_undegraded":"2024-04-25T05:40:14.771282+0000","last_fullsized":"2024-04-25T05:40:14.771282+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:52.054641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:37:52.054641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":367,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.764849+0000","last_change":"2024-04-25T05:39:38.764826+0000","last_active":"2024-04-25T05:40:14.764849+0000","last_peered":"2024-04-25T05:40:14.764849+0000","last_clean":"2024-04-25T05:40:14.764849+0000","last_became_active":"2024-04-25T05:38:13.284281+0000","last_became_peered":"2024-04-25T05:38:13.284281+0000","last_unstale":"2024-04-25T05:40:14.764849+0000","last_undegraded":"2024-04-25T05:40:14.764849+0000","last_fullsized":"2024-04-25T05:40:14.764849+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.764632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.764632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:40.459074+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":406,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772982+0000","last_change":"2024-04-25T05:40:14.593162+0000","last_active":"2024-04-25T05:40:14.772982+0000","last_peered":"2024-04-25T05:40:14.772982+0000","last_clean":"2024-04-25T05:40:14.772982+0000","last_became_active":"2024-04-25T05:38:13.295547+0000","last_became_peered":"2024-04-25T05:38:13.295547+0000","last_unstale":"2024-04-25T05:40:14.772982+0000","last_undegraded":"2024-04-25T05:40:14.772982+0000","last_fullsized":"2024-04-25T05:40:14.772982+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.592911+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:23.779841+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":336,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769248+0000","last_change":"2024-04-25T05:38:59.769234+0000","last_active":"2024-04-25T05:40:14.769248+0000","last_peered":"2024-04-25T05:40:14.769248+0000","last_clean":"2024-04-25T05:40:14.769248+0000","last_became_active":"2024-04-25T05:34:02.702433+0000","last_became_peered":"2024-04-25T05:34:02.702433+0000","last_unstale":"2024-04-25T05:40:14.769248+0000","last_undegraded":"2024-04-25T05:40:14.769248+0000","last_fullsized":"2024-04-25T05:40:14.769248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.769120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.769120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.633366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"225'5118","reported_seq":6051,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.733496+0000","last_change":"2024-04-25T05:39:34.781500+0000","last_active":"2024-04-25T05:40:17.733496+0000","last_peered":"2024-04-25T05:40:17.733496+0000","last_clean":"2024-04-25T05:40:17.733496+0000","last_became_active":"2024-04-25T05:39:34.778800+0000","last_became_peered":"2024-04-25T05:39:34.778800+0000","last_unstale":"2024-04-25T05:40:17.733496+0000","last_undegraded":"2024-04-25T05:40:17.733496+0000","last_fullsized":"2024-04-25T05:40:17.733496+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.108740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335413248,"num_objects":5118,"num_object_clones":0,"num_object_copies":15354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":327552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":28049408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776802+0000","last_change":"2024-04-25T05:39:09.353431+0000","last_active":"2024-04-25T05:40:14.776802+0000","last_peered":"2024-04-25T05:40:14.776802+0000","last_clean":"2024-04-25T05:40:14.776802+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:40:14.776802+0000","last_undegraded":"2024-04-25T05:40:14.776802+0000","last_fullsized":"2024-04-25T05:40:14.776802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.353310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782749+0000","last_change":"2024-04-25T05:39:32.801925+0000","last_active":"2024-04-25T05:40:14.782749+0000","last_peered":"2024-04-25T05:40:14.782749+0000","last_clean":"2024-04-25T05:40:14.782749+0000","last_became_active":"2024-04-25T05:39:32.801558+0000","last_became_peered":"2024-04-25T05:39:32.801558+0000","last_unstale":"2024-04-25T05:40:14.782749+0000","last_undegraded":"2024-04-25T05:40:14.782749+0000","last_fullsized":"2024-04-25T05:40:14.782749+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.104548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.104548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765482+0000","last_change":"2024-04-25T05:39:29.797447+0000","last_active":"2024-04-25T05:40:14.765482+0000","last_peered":"2024-04-25T05:40:14.765482+0000","last_clean":"2024-04-25T05:40:14.765482+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:40:14.765482+0000","last_undegraded":"2024-04-25T05:40:14.765482+0000","last_fullsized":"2024-04-25T05:40:14.765482+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:26.336031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:26.336031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":80,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772826+0000","last_change":"2024-04-25T05:39:29.827544+0000","last_active":"2024-04-25T05:40:14.772826+0000","last_peered":"2024-04-25T05:40:14.772826+0000","last_clean":"2024-04-25T05:40:14.772826+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:40:14.772826+0000","last_undegraded":"2024-04-25T05:40:14.772826+0000","last_fullsized":"2024-04-25T05:40:14.772826+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'18","last_scrub_stamp":"2024-04-25T05:39:04.573693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.573693+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.160666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.782121+0000","last_change":"2024-04-25T05:37:03.680508+0000","last_active":"2024-04-25T05:40:14.782121+0000","last_peered":"2024-04-25T05:40:14.782121+0000","last_clean":"2024-04-25T05:40:14.782121+0000","last_became_active":"2024-04-25T05:34:02.700755+0000","last_became_peered":"2024-04-25T05:34:02.700755+0000","last_unstale":"2024-04-25T05:40:14.782121+0000","last_undegraded":"2024-04-25T05:40:14.782121+0000","last_fullsized":"2024-04-25T05:40:14.782121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:37:03.680381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"225'5208","reported_seq":5272,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.041493+0000","last_change":"2024-04-25T05:39:19.231225+0000","last_active":"2024-04-25T05:40:18.041493+0000","last_peered":"2024-04-25T05:40:18.041493+0000","last_clean":"2024-04-25T05:40:18.041493+0000","last_became_active":"2024-04-25T05:39:19.230926+0000","last_became_peered":"2024-04-25T05:39:19.230926+0000","last_unstale":"2024-04-25T05:40:18.041493+0000","last_undegraded":"2024-04-25T05:40:18.041493+0000","last_fullsized":"2024-04-25T05:40:18.041493+0000","mapping_epoch":198,"log_start":"210'900","ondisk_log_start":"210'900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:24.268052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341311488,"num_objects":5208,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5208,"num_write_kb":333312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":527,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771689+0000","last_change":"2024-04-25T05:39:32.814133+0000","last_active":"2024-04-25T05:40:14.771689+0000","last_peered":"2024-04-25T05:40:14.771689+0000","last_clean":"2024-04-25T05:40:14.771689+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:40:14.771689+0000","last_undegraded":"2024-04-25T05:40:14.771689+0000","last_fullsized":"2024-04-25T05:40:14.771689+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.010323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.010323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:27.554151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771215+0000","last_change":"2024-04-25T05:39:00.922795+0000","last_active":"2024-04-25T05:40:14.771215+0000","last_peered":"2024-04-25T05:40:14.771215+0000","last_clean":"2024-04-25T05:40:14.771215+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:40:14.771215+0000","last_undegraded":"2024-04-25T05:40:14.771215+0000","last_fullsized":"2024-04-25T05:40:14.771215+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.922679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.922679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:18.355708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":151,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.773125+0000","last_change":"2024-04-25T05:40:11.785614+0000","last_active":"2024-04-25T05:40:14.773125+0000","last_peered":"2024-04-25T05:40:14.773125+0000","last_clean":"2024-04-25T05:40:14.773125+0000","last_became_active":"2024-04-25T05:38:13.238969+0000","last_became_peered":"2024-04-25T05:38:13.238969+0000","last_unstale":"2024-04-25T05:40:14.773125+0000","last_undegraded":"2024-04-25T05:40:14.773125+0000","last_fullsized":"2024-04-25T05:40:14.773125+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.785355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.785355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:23.980679+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":740,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:16.490409+0000","last_change":"2024-04-25T05:38:13.325017+0000","last_active":"2024-04-25T05:40:16.490409+0000","last_peered":"2024-04-25T05:40:16.490409+0000","last_clean":"2024-04-25T05:40:16.490409+0000","last_became_active":"2024-04-25T05:38:13.305327+0000","last_became_peered":"2024-04-25T05:38:13.305327+0000","last_unstale":"2024-04-25T05:40:16.490409+0000","last_undegraded":"2024-04-25T05:40:16.490409+0000","last_fullsized":"2024-04-25T05:40:16.490409+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:37:44.995074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:37:44.995074+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":180,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776528+0000","last_change":"2024-04-25T05:40:12.370708+0000","last_active":"2024-04-25T05:40:14.776528+0000","last_peered":"2024-04-25T05:40:14.776528+0000","last_clean":"2024-04-25T05:40:14.776528+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:40:14.776528+0000","last_undegraded":"2024-04-25T05:40:14.776528+0000","last_fullsized":"2024-04-25T05:40:14.776528+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:41.662583+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777335+0000","last_change":"2024-04-25T05:38:13.296713+0000","last_active":"2024-04-25T05:40:14.777335+0000","last_peered":"2024-04-25T05:40:14.777335+0000","last_clean":"2024-04-25T05:40:14.777335+0000","last_became_active":"2024-04-25T05:38:13.296013+0000","last_became_peered":"2024-04-25T05:38:13.296013+0000","last_unstale":"2024-04-25T05:40:14.777335+0000","last_undegraded":"2024-04-25T05:40:14.777335+0000","last_fullsized":"2024-04-25T05:40:14.777335+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:07.166833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:07.166833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"225'5289","reported_seq":5353,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.141127+0000","last_change":"2024-04-25T05:39:19.229227+0000","last_active":"2024-04-25T05:40:18.141127+0000","last_peered":"2024-04-25T05:40:18.141127+0000","last_clean":"2024-04-25T05:40:18.141127+0000","last_became_active":"2024-04-25T05:39:19.228912+0000","last_became_peered":"2024-04-25T05:39:19.228912+0000","last_unstale":"2024-04-25T05:40:18.141127+0000","last_undegraded":"2024-04-25T05:40:18.141127+0000","last_fullsized":"2024-04-25T05:40:18.141127+0000","mapping_epoch":198,"log_start":"210'1200","ondisk_log_start":"210'1200","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:37.352621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346619904,"num_objects":5289,"num_object_clones":0,"num_object_copies":15867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5289,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":524,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769875+0000","last_change":"2024-04-25T05:40:02.832331+0000","last_active":"2024-04-25T05:40:14.769875+0000","last_peered":"2024-04-25T05:40:14.769875+0000","last_clean":"2024-04-25T05:40:14.769875+0000","last_became_active":"2024-04-25T05:39:32.801996+0000","last_became_peered":"2024-04-25T05:39:32.801996+0000","last_unstale":"2024-04-25T05:40:14.769875+0000","last_undegraded":"2024-04-25T05:40:14.769875+0000","last_fullsized":"2024-04-25T05:40:14.769875+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:02.832193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.832193+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:09.143782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771359+0000","last_change":"2024-04-25T05:39:30.760584+0000","last_active":"2024-04-25T05:40:14.771359+0000","last_peered":"2024-04-25T05:40:14.771359+0000","last_clean":"2024-04-25T05:40:14.771359+0000","last_became_active":"2024-04-25T05:39:29.792454+0000","last_became_peered":"2024-04-25T05:39:29.792454+0000","last_unstale":"2024-04-25T05:40:14.771359+0000","last_undegraded":"2024-04-25T05:40:14.771359+0000","last_fullsized":"2024-04-25T05:40:14.771359+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:30.760468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.760468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.143766+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":354,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:16.651714+0000","last_change":"2024-04-25T05:40:16.651753+0000","last_active":"2024-04-25T05:40:16.651714+0000","last_peered":"2024-04-25T05:40:16.651714+0000","last_clean":"2024-04-25T05:40:16.651714+0000","last_became_active":"2024-04-25T05:33:58.675276+0000","last_became_peered":"2024-04-25T05:33:58.675276+0000","last_unstale":"2024-04-25T05:40:16.651714+0000","last_undegraded":"2024-04-25T05:40:16.651714+0000","last_fullsized":"2024-04-25T05:40:16.651714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:16.651620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:16.651620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:19.462727+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":238,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775300+0000","last_change":"2024-04-25T05:38:22.405354+0000","last_active":"2024-04-25T05:40:14.775300+0000","last_peered":"2024-04-25T05:40:14.775300+0000","last_clean":"2024-04-25T05:40:14.775300+0000","last_became_active":"2024-04-25T05:38:13.296480+0000","last_became_peered":"2024-04-25T05:38:13.296480+0000","last_unstale":"2024-04-25T05:40:14.775300+0000","last_undegraded":"2024-04-25T05:40:14.775300+0000","last_fullsized":"2024-04-25T05:40:14.775300+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:38:22.404901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:22.404901+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769968+0000","last_change":"2024-04-25T05:39:36.496099+0000","last_active":"2024-04-25T05:40:14.769968+0000","last_peered":"2024-04-25T05:40:14.769968+0000","last_clean":"2024-04-25T05:40:14.769968+0000","last_became_active":"2024-04-25T05:39:29.797329+0000","last_became_peered":"2024-04-25T05:39:29.797329+0000","last_unstale":"2024-04-25T05:40:14.769968+0000","last_undegraded":"2024-04-25T05:40:14.769968+0000","last_fullsized":"2024-04-25T05:40:14.769968+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.495957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.495957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.120296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"225'5031","reported_seq":5095,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.938775+0000","last_change":"2024-04-25T05:39:19.229498+0000","last_active":"2024-04-25T05:40:17.938775+0000","last_peered":"2024-04-25T05:40:17.938775+0000","last_clean":"2024-04-25T05:40:17.938775+0000","last_became_active":"2024-04-25T05:39:19.229203+0000","last_became_peered":"2024-04-25T05:39:19.229203+0000","last_unstale":"2024-04-25T05:40:17.938775+0000","last_undegraded":"2024-04-25T05:40:17.938775+0000","last_fullsized":"2024-04-25T05:40:17.938775+0000","mapping_epoch":198,"log_start":"202'200","ondisk_log_start":"202'200","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:42.361829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329711616,"num_objects":5031,"num_object_clones":0,"num_object_copies":15093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5031,"num_write_kb":321984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":516,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770848+0000","last_change":"2024-04-25T05:39:32.805680+0000","last_active":"2024-04-25T05:40:14.770848+0000","last_peered":"2024-04-25T05:40:14.770848+0000","last_clean":"2024-04-25T05:40:14.770848+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:40:14.770848+0000","last_undegraded":"2024-04-25T05:40:14.770848+0000","last_fullsized":"2024-04-25T05:40:14.770848+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.069332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.069332+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:26.710721+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":143,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767630+0000","last_change":"2024-04-25T05:38:13.244062+0000","last_active":"2024-04-25T05:40:14.767630+0000","last_peered":"2024-04-25T05:40:14.767630+0000","last_clean":"2024-04-25T05:40:14.767630+0000","last_became_active":"2024-04-25T05:38:13.243650+0000","last_became_peered":"2024-04-25T05:38:13.243650+0000","last_unstale":"2024-04-25T05:40:14.767630+0000","last_undegraded":"2024-04-25T05:40:14.767630+0000","last_fullsized":"2024-04-25T05:40:14.767630+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":346,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777012+0000","last_change":"2024-04-25T05:39:38.370074+0000","last_active":"2024-04-25T05:40:14.777012+0000","last_peered":"2024-04-25T05:40:14.777012+0000","last_clean":"2024-04-25T05:40:14.777012+0000","last_became_active":"2024-04-25T05:33:58.678380+0000","last_became_peered":"2024-04-25T05:33:58.678380+0000","last_unstale":"2024-04-25T05:40:14.777012+0000","last_undegraded":"2024-04-25T05:40:14.777012+0000","last_fullsized":"2024-04-25T05:40:14.777012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:48.827472+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":305,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765179+0000","last_change":"2024-04-25T05:39:09.887432+0000","last_active":"2024-04-25T05:40:14.765179+0000","last_peered":"2024-04-25T05:40:14.765179+0000","last_clean":"2024-04-25T05:40:14.765179+0000","last_became_active":"2024-04-25T05:35:32.891669+0000","last_became_peered":"2024-04-25T05:35:32.891669+0000","last_unstale":"2024-04-25T05:40:14.765179+0000","last_undegraded":"2024-04-25T05:40:14.765179+0000","last_fullsized":"2024-04-25T05:40:14.765179+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.887340+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.423904+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"208'11","reported_seq":74,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.770648+0000","last_change":"2024-04-25T05:40:06.822425+0000","last_active":"2024-04-25T05:40:14.770648+0000","last_peered":"2024-04-25T05:40:14.770648+0000","last_clean":"2024-04-25T05:40:14.770648+0000","last_became_active":"2024-04-25T05:39:29.795733+0000","last_became_peered":"2024-04-25T05:39:29.795733+0000","last_unstale":"2024-04-25T05:40:14.770648+0000","last_undegraded":"2024-04-25T05:40:14.770648+0000","last_fullsized":"2024-04-25T05:40:14.770648+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:40:06.822122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.822122+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:34.211220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"225'5157","reported_seq":5221,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.722356+0000","last_change":"2024-04-25T05:39:19.228938+0000","last_active":"2024-04-25T05:40:17.722356+0000","last_peered":"2024-04-25T05:40:17.722356+0000","last_clean":"2024-04-25T05:40:17.722356+0000","last_became_active":"2024-04-25T05:39:19.228620+0000","last_became_peered":"2024-04-25T05:39:19.228620+0000","last_unstale":"2024-04-25T05:40:17.722356+0000","last_undegraded":"2024-04-25T05:40:17.722356+0000","last_fullsized":"2024-04-25T05:40:17.722356+0000","mapping_epoch":198,"log_start":"207'800","ondisk_log_start":"207'800","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4357,"log_dups_size":0,"ondisk_log_size":4357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:28.198905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337969152,"num_objects":5157,"num_object_clones":0,"num_object_copies":15471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":330048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":512,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767090+0000","last_change":"2024-04-25T05:38:58.078848+0000","last_active":"2024-04-25T05:40:14.767090+0000","last_peered":"2024-04-25T05:40:14.767090+0000","last_clean":"2024-04-25T05:40:14.767090+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:40:14.767090+0000","last_undegraded":"2024-04-25T05:40:14.767090+0000","last_fullsized":"2024-04-25T05:40:14.767090+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.078483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.078483+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":345,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.780277+0000","last_change":"2024-04-25T05:39:29.798367+0000","last_active":"2024-04-25T05:40:14.780277+0000","last_peered":"2024-04-25T05:40:14.780277+0000","last_clean":"2024-04-25T05:40:14.780277+0000","last_became_active":"2024-04-25T05:39:29.798026+0000","last_became_peered":"2024-04-25T05:39:29.798026+0000","last_unstale":"2024-04-25T05:40:14.780277+0000","last_undegraded":"2024-04-25T05:40:14.780277+0000","last_fullsized":"2024-04-25T05:40:14.780277+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:59.627680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:59.627680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":231,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772658+0000","last_change":"2024-04-25T05:39:34.741064+0000","last_active":"2024-04-25T05:40:14.772658+0000","last_peered":"2024-04-25T05:40:14.772658+0000","last_clean":"2024-04-25T05:40:14.772658+0000","last_became_active":"2024-04-25T05:38:13.289662+0000","last_became_peered":"2024-04-25T05:38:13.289662+0000","last_unstale":"2024-04-25T05:40:14.772658+0000","last_undegraded":"2024-04-25T05:40:14.772658+0000","last_fullsized":"2024-04-25T05:40:14.772658+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:52.993594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":68,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775706+0000","last_change":"2024-04-25T05:40:09.734937+0000","last_active":"2024-04-25T05:40:14.775706+0000","last_peered":"2024-04-25T05:40:14.775706+0000","last_clean":"2024-04-25T05:40:14.775706+0000","last_became_active":"2024-04-25T05:40:09.733866+0000","last_became_peered":"2024-04-25T05:40:09.733866+0000","last_unstale":"2024-04-25T05:40:14.775706+0000","last_undegraded":"2024-04-25T05:40:14.775706+0000","last_fullsized":"2024-04-25T05:40:14.775706+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:36:47.698551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:47.698551+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":414,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.768036+0000","last_change":"2024-04-25T05:36:21.409926+0000","last_active":"2024-04-25T05:40:14.768036+0000","last_peered":"2024-04-25T05:40:14.768036+0000","last_clean":"2024-04-25T05:40:14.768036+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:40:14.768036+0000","last_undegraded":"2024-04-25T05:40:14.768036+0000","last_fullsized":"2024-04-25T05:40:14.768036+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:36:21.409774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:21.409774+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"225'5234","reported_seq":5294,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.124757+0000","last_change":"2024-04-25T05:39:19.232542+0000","last_active":"2024-04-25T05:40:18.124757+0000","last_peered":"2024-04-25T05:40:18.124757+0000","last_clean":"2024-04-25T05:40:18.124757+0000","last_became_active":"2024-04-25T05:39:19.232224+0000","last_became_peered":"2024-04-25T05:39:19.232224+0000","last_unstale":"2024-04-25T05:40:18.124757+0000","last_undegraded":"2024-04-25T05:40:18.124757+0000","last_fullsized":"2024-04-25T05:40:18.124757+0000","mapping_epoch":198,"log_start":"210'1200","ondisk_log_start":"210'1200","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4034,"log_dups_size":0,"ondisk_log_size":4034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:45.474301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343015424,"num_objects":5234,"num_object_clones":0,"num_object_copies":15702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":334976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":521,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.768704+0000","last_change":"2024-04-25T05:39:32.795454+0000","last_active":"2024-04-25T05:40:14.768704+0000","last_peered":"2024-04-25T05:40:14.768704+0000","last_clean":"2024-04-25T05:40:14.768704+0000","last_became_active":"2024-04-25T05:39:32.795085+0000","last_became_peered":"2024-04-25T05:39:32.795085+0000","last_unstale":"2024-04-25T05:40:14.768704+0000","last_undegraded":"2024-04-25T05:40:14.768704+0000","last_fullsized":"2024-04-25T05:40:14.768704+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.548529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.548529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765792+0000","last_change":"2024-04-25T05:40:09.736162+0000","last_active":"2024-04-25T05:40:14.765792+0000","last_peered":"2024-04-25T05:40:14.765792+0000","last_clean":"2024-04-25T05:40:14.765792+0000","last_became_active":"2024-04-25T05:40:09.734527+0000","last_became_peered":"2024-04-25T05:40:09.734527+0000","last_unstale":"2024-04-25T05:40:14.765792+0000","last_undegraded":"2024-04-25T05:40:14.765792+0000","last_fullsized":"2024-04-25T05:40:14.765792+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.719427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.719427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:27.781697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765487+0000","last_change":"2024-04-25T05:38:13.247469+0000","last_active":"2024-04-25T05:40:14.765487+0000","last_peered":"2024-04-25T05:40:14.765487+0000","last_clean":"2024-04-25T05:40:14.765487+0000","last_became_active":"2024-04-25T05:38:13.246750+0000","last_became_peered":"2024-04-25T05:38:13.246750+0000","last_unstale":"2024-04-25T05:40:14.765487+0000","last_undegraded":"2024-04-25T05:40:14.765487+0000","last_fullsized":"2024-04-25T05:40:14.765487+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:06.377412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:06.377412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":418,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767932+0000","last_change":"2024-04-25T05:38:13.326846+0000","last_active":"2024-04-25T05:40:14.767932+0000","last_peered":"2024-04-25T05:40:14.767932+0000","last_clean":"2024-04-25T05:40:14.767932+0000","last_became_active":"2024-04-25T05:38:13.305935+0000","last_became_peered":"2024-04-25T05:38:13.305935+0000","last_unstale":"2024-04-25T05:40:14.767932+0000","last_undegraded":"2024-04-25T05:40:14.767932+0000","last_fullsized":"2024-04-25T05:40:14.767932+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:35:30.243527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:35:30.243527+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769388+0000","last_change":"2024-04-25T05:40:08.742282+0000","last_active":"2024-04-25T05:40:14.769388+0000","last_peered":"2024-04-25T05:40:14.769388+0000","last_clean":"2024-04-25T05:40:14.769388+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:40:14.769388+0000","last_undegraded":"2024-04-25T05:40:14.769388+0000","last_fullsized":"2024-04-25T05:40:14.769388+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:08.742146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.742146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:33.663262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"225'5061","reported_seq":5873,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.052022+0000","last_change":"2024-04-25T05:39:34.781974+0000","last_active":"2024-04-25T05:40:18.052022+0000","last_peered":"2024-04-25T05:40:18.052022+0000","last_clean":"2024-04-25T05:40:18.052022+0000","last_became_active":"2024-04-25T05:39:34.780422+0000","last_became_peered":"2024-04-25T05:39:34.780422+0000","last_unstale":"2024-04-25T05:40:18.052022+0000","last_undegraded":"2024-04-25T05:40:18.052022+0000","last_fullsized":"2024-04-25T05:40:18.052022+0000","mapping_epoch":208,"log_start":"207'700","ondisk_log_start":"207'700","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:38.827751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":323904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":24117248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":521,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.775431+0000","last_change":"2024-04-25T05:39:32.794193+0000","last_active":"2024-04-25T05:40:14.775431+0000","last_peered":"2024-04-25T05:40:14.775431+0000","last_clean":"2024-04-25T05:40:14.775431+0000","last_became_active":"2024-04-25T05:39:32.793746+0000","last_became_peered":"2024-04-25T05:39:32.793746+0000","last_unstale":"2024-04-25T05:40:14.775431+0000","last_undegraded":"2024-04-25T05:40:14.775431+0000","last_fullsized":"2024-04-25T05:40:14.775431+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":339,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776698+0000","last_change":"2024-04-25T05:38:13.284810+0000","last_active":"2024-04-25T05:40:14.776698+0000","last_peered":"2024-04-25T05:40:14.776698+0000","last_clean":"2024-04-25T05:40:14.776698+0000","last_became_active":"2024-04-25T05:38:13.284451+0000","last_became_peered":"2024-04-25T05:38:13.284451+0000","last_unstale":"2024-04-25T05:40:14.776698+0000","last_undegraded":"2024-04-25T05:40:14.776698+0000","last_fullsized":"2024-04-25T05:40:14.776698+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.001623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.001623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"223'10","reported_seq":43,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.776761+0000","last_change":"2024-04-25T05:40:09.728144+0000","last_active":"2024-04-25T05:40:14.776761+0000","last_peered":"2024-04-25T05:40:14.776761+0000","last_clean":"2024-04-25T05:40:14.776761+0000","last_became_active":"2024-04-25T05:40:09.727755+0000","last_became_peered":"2024-04-25T05:40:09.727755+0000","last_unstale":"2024-04-25T05:40:14.776761+0000","last_undegraded":"2024-04-25T05:40:14.776761+0000","last_fullsized":"2024-04-25T05:40:14.776761+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:37:48.604653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:48.604653+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767942+0000","last_change":"2024-04-25T05:35:43.429944+0000","last_active":"2024-04-25T05:40:14.767942+0000","last_peered":"2024-04-25T05:40:14.767942+0000","last_clean":"2024-04-25T05:40:14.767942+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:40:14.767942+0000","last_undegraded":"2024-04-25T05:40:14.767942+0000","last_fullsized":"2024-04-25T05:40:14.767942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:43.429814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:35:43.429814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":373,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.773055+0000","last_change":"2024-04-25T05:39:39.713342+0000","last_active":"2024-04-25T05:40:14.773055+0000","last_peered":"2024-04-25T05:40:14.773055+0000","last_clean":"2024-04-25T05:40:14.773055+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:40:14.773055+0000","last_undegraded":"2024-04-25T05:40:14.773055+0000","last_fullsized":"2024-04-25T05:40:14.773055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.713165+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:56.240786+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"225'5081","reported_seq":5145,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:17.734729+0000","last_change":"2024-04-25T05:39:19.230917+0000","last_active":"2024-04-25T05:40:17.734729+0000","last_peered":"2024-04-25T05:40:17.734729+0000","last_clean":"2024-04-25T05:40:17.734729+0000","last_became_active":"2024-04-25T05:39:19.230541+0000","last_became_peered":"2024-04-25T05:39:19.230541+0000","last_unstale":"2024-04-25T05:40:17.734729+0000","last_undegraded":"2024-04-25T05:40:17.734729+0000","last_fullsized":"2024-04-25T05:40:17.734729+0000","mapping_epoch":198,"log_start":"210'800","ondisk_log_start":"210'800","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:29.619639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332988416,"num_objects":5081,"num_object_clones":0,"num_object_copies":15243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":325184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":337,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767154+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:40:14.767154+0000","last_peered":"2024-04-25T05:40:14.767154+0000","last_clean":"2024-04-25T05:40:14.767154+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:40:14.767154+0000","last_undegraded":"2024-04-25T05:40:14.767154+0000","last_fullsized":"2024-04-25T05:40:14.767154+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:01.546178+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":349,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777240+0000","last_change":"2024-04-25T05:39:29.794362+0000","last_active":"2024-04-25T05:40:14.777240+0000","last_peered":"2024-04-25T05:40:14.777240+0000","last_clean":"2024-04-25T05:40:14.777240+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:40:14.777240+0000","last_undegraded":"2024-04-25T05:40:14.777240+0000","last_fullsized":"2024-04-25T05:40:14.777240+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:37:08.197114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:37:08.197114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":344,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769384+0000","last_change":"2024-04-25T05:39:28.539337+0000","last_active":"2024-04-25T05:40:14.769384+0000","last_peered":"2024-04-25T05:40:14.769384+0000","last_clean":"2024-04-25T05:40:14.769384+0000","last_became_active":"2024-04-25T05:33:58.674700+0000","last_became_peered":"2024-04-25T05:33:58.674700+0000","last_unstale":"2024-04-25T05:40:14.769384+0000","last_undegraded":"2024-04-25T05:40:14.769384+0000","last_fullsized":"2024-04-25T05:40:14.769384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:28.539190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:28.539190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:35.676380+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":362,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769668+0000","last_change":"2024-04-25T05:40:09.839273+0000","last_active":"2024-04-25T05:40:14.769668+0000","last_peered":"2024-04-25T05:40:14.769668+0000","last_clean":"2024-04-25T05:40:14.769668+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:40:14.769668+0000","last_undegraded":"2024-04-25T05:40:14.769668+0000","last_fullsized":"2024-04-25T05:40:14.769668+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:09.838661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.838661+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:11.113400+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"225'5262","reported_seq":5326,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.051935+0000","last_change":"2024-04-25T05:39:19.229731+0000","last_active":"2024-04-25T05:40:18.051935+0000","last_peered":"2024-04-25T05:40:18.051935+0000","last_clean":"2024-04-25T05:40:18.051935+0000","last_became_active":"2024-04-25T05:39:19.229438+0000","last_became_peered":"2024-04-25T05:39:19.229438+0000","last_unstale":"2024-04-25T05:40:18.051935+0000","last_undegraded":"2024-04-25T05:40:18.051935+0000","last_fullsized":"2024-04-25T05:40:18.051935+0000","mapping_epoch":198,"log_start":"210'900","ondisk_log_start":"210'900","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:32.701047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344850432,"num_objects":5262,"num_object_clones":0,"num_object_copies":15786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":336768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":512,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.777491+0000","last_change":"2024-04-25T05:38:59.374948+0000","last_active":"2024-04-25T05:40:14.777491+0000","last_peered":"2024-04-25T05:40:14.777491+0000","last_clean":"2024-04-25T05:40:14.777491+0000","last_became_active":"2024-04-25T05:38:34.759755+0000","last_became_peered":"2024-04-25T05:38:34.759755+0000","last_unstale":"2024-04-25T05:40:14.777491+0000","last_undegraded":"2024-04-25T05:40:14.777491+0000","last_fullsized":"2024-04-25T05:40:14.777491+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":332,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765976+0000","last_change":"2024-04-25T05:39:00.794514+0000","last_active":"2024-04-25T05:40:14.765976+0000","last_peered":"2024-04-25T05:40:14.765976+0000","last_clean":"2024-04-25T05:40:14.765976+0000","last_became_active":"2024-04-25T05:34:04.722753+0000","last_became_peered":"2024-04-25T05:34:04.722753+0000","last_unstale":"2024-04-25T05:40:14.765976+0000","last_undegraded":"2024-04-25T05:40:14.765976+0000","last_fullsized":"2024-04-25T05:40:14.765976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:00.794388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:00.794388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:22.097831+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":349,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.771762+0000","last_change":"2024-04-25T05:39:29.798267+0000","last_active":"2024-04-25T05:40:14.771762+0000","last_peered":"2024-04-25T05:40:14.771762+0000","last_clean":"2024-04-25T05:40:14.771762+0000","last_became_active":"2024-04-25T05:39:29.797924+0000","last_became_peered":"2024-04-25T05:39:29.797924+0000","last_unstale":"2024-04-25T05:40:14.771762+0000","last_undegraded":"2024-04-25T05:40:14.771762+0000","last_fullsized":"2024-04-25T05:40:14.771762+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:49.750361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.750361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":61,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.769882+0000","last_change":"2024-04-25T05:39:37.518793+0000","last_active":"2024-04-25T05:40:14.769882+0000","last_peered":"2024-04-25T05:40:14.769882+0000","last_clean":"2024-04-25T05:40:14.769882+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:40:14.769882+0000","last_undegraded":"2024-04-25T05:40:14.769882+0000","last_fullsized":"2024-04-25T05:40:14.769882+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.518653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.518653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:46.935057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":63,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765831+0000","last_change":"2024-04-25T05:40:09.734564+0000","last_active":"2024-04-25T05:40:14.765831+0000","last_peered":"2024-04-25T05:40:14.765831+0000","last_clean":"2024-04-25T05:40:14.765831+0000","last_became_active":"2024-04-25T05:40:09.733230+0000","last_became_peered":"2024-04-25T05:40:09.733230+0000","last_unstale":"2024-04-25T05:40:14.765831+0000","last_undegraded":"2024-04-25T05:40:14.765831+0000","last_fullsized":"2024-04-25T05:40:14.765831+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:34.605704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.605704+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:59.766260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"225'5135","reported_seq":5199,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:18.141718+0000","last_change":"2024-04-25T05:39:19.231808+0000","last_active":"2024-04-25T05:40:18.141718+0000","last_peered":"2024-04-25T05:40:18.141718+0000","last_clean":"2024-04-25T05:40:18.141718+0000","last_became_active":"2024-04-25T05:39:19.231525+0000","last_became_peered":"2024-04-25T05:39:19.231525+0000","last_unstale":"2024-04-25T05:40:18.141718+0000","last_undegraded":"2024-04-25T05:40:18.141718+0000","last_fullsized":"2024-04-25T05:40:18.141718+0000","mapping_epoch":198,"log_start":"210'1100","ondisk_log_start":"210'1100","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:23.331924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336527360,"num_objects":5135,"num_object_clones":0,"num_object_copies":15405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5135,"num_write_kb":328640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":643,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.778727+0000","last_change":"2024-04-25T05:39:01.087263+0000","last_active":"2024-04-25T05:40:14.778727+0000","last_peered":"2024-04-25T05:40:14.778727+0000","last_clean":"2024-04-25T05:40:14.778727+0000","last_became_active":"2024-04-25T05:38:34.761642+0000","last_became_peered":"2024-04-25T05:38:34.761642+0000","last_unstale":"2024-04-25T05:40:14.778727+0000","last_undegraded":"2024-04-25T05:40:14.778727+0000","last_fullsized":"2024-04-25T05:40:14.778727+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:01.087145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:01.087145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":16,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.772081+0000","last_change":"2024-04-25T05:40:09.719667+0000","last_active":"2024-04-25T05:40:14.772081+0000","last_peered":"2024-04-25T05:40:14.772081+0000","last_clean":"2024-04-25T05:40:14.772081+0000","last_became_active":"2024-04-25T05:40:09.719153+0000","last_became_peered":"2024-04-25T05:40:09.719153+0000","last_unstale":"2024-04-25T05:40:14.772081+0000","last_undegraded":"2024-04-25T05:40:14.772081+0000","last_fullsized":"2024-04-25T05:40:14.772081+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:02.813078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.813078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:41:27.893488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"98'9","reported_seq":424,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.765993+0000","last_change":"2024-04-25T05:39:33.738444+0000","last_active":"2024-04-25T05:40:14.765993+0000","last_peered":"2024-04-25T05:40:14.765993+0000","last_clean":"2024-04-25T05:40:14.765993+0000","last_became_active":"2024-04-25T05:34:02.704719+0000","last_became_peered":"2024-04-25T05:34:02.704719+0000","last_unstale":"2024-04-25T05:40:14.765993+0000","last_undegraded":"2024-04-25T05:40:14.765993+0000","last_fullsized":"2024-04-25T05:40:14.765993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:39:33.738307+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.738307+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:40:59.211618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":343,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.766984+0000","last_change":"2024-04-25T05:38:13.286297+0000","last_active":"2024-04-25T05:40:14.766984+0000","last_peered":"2024-04-25T05:40:14.766984+0000","last_clean":"2024-04-25T05:40:14.766984+0000","last_became_active":"2024-04-25T05:38:13.285812+0000","last_became_peered":"2024-04-25T05:38:13.285812+0000","last_unstale":"2024-04-25T05:40:14.766984+0000","last_undegraded":"2024-04-25T05:40:14.766984+0000","last_fullsized":"2024-04-25T05:40:14.766984+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":50,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-25T05:40:14.767309+0000","last_change":"2024-04-25T05:40:09.731382+0000","last_active":"2024-04-25T05:40:14.767309+0000","last_peered":"2024-04-25T05:40:14.767309+0000","last_clean":"2024-04-25T05:40:14.767309+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:40:14.767309+0000","last_undegraded":"2024-04-25T05:40:14.767309+0000","last_fullsized":"2024-04-25T05:40:14.767309+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5393350656,"num_objects":82296,"num_object_clones":0,"num_object_copies":246888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82296,"num_write_kb":5266944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2058,"num_bytes_recovered":134873088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16167862272,"data_stored":16167862272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71096,"ondisk_log_size":71096,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":369,"num_read_kb":364,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":4977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":974,"num_read_kb":699,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":3278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":12800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":647,"ondisk_log_size":647,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070545,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2150188,"kb_used_data":1934372,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91594964,"statfs":{"total":95995035648,"available":93793243136,"internally_reserved":0,"allocated":1980796928,"data_stored":1980025087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364885,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759304,"kb_used_data":1615676,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91985848,"statfs":{"total":95995035648,"available":94193508352,"internally_reserved":0,"allocated":1654452224,"data_stored":1653719869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2601476,"kb_used_data":2303740,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91143676,"statfs":{"total":95995035648,"available":93331124224,"internally_reserved":0,"allocated":2359029760,"data_stored":2358232317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2605592,"kb_used_data":2317584,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91139560,"statfs":{"total":95995035648,"available":93326909440,"internally_reserved":0,"allocated":2373206016,"data_stored":2372453117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2965036,"kb_used_data":2618724,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90780116,"statfs":{"total":95995035648,"available":92958838784,"internally_reserved":0,"allocated":2681573376,"data_stored":2680800566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354613703},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986270,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771680,"kb_used_data":1605848,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91973472,"statfs":{"total":95995035648,"available":94180835328,"internally_reserved":0,"allocated":1644388352,"data_stored":1643627507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691930,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3457488,"kb_used_data":3268040,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":90287664,"statfs":{"total":95995035648,"available":92454567936,"internally_reserved":0,"allocated":3346472960,"data_stored":3345728447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2373124096,"data_stored":2373124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2372468736,"data_stored":2372468736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2707226624,"data_stored":2707226624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1665662976,"data_stored":1665662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3363766272,"data_stored":3363766272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1676279808,"data_stored":1676279808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2009333760,"data_stored":2009333760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:40:23.872 INFO:tasks.thrashosds.thrasher:Setting 6.3 to [3, 2, 7, 1, 6, 0] 2024-04-25T05:40:23.872 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.3', '3', '2', '7', '1', '6', '0'] 2024-04-25T05:40:23.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap-items 6.3 3 2 7 1 6 0 2024-04-25T05:40:23.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:23.926 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:23.936 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:23.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:23.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:23 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:24.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:24.071 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:23 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:24.071 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:23 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:24.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:24.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:24.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:24.247 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61441 7.5 7:a916e294:::benchmark_data_smithi043_74356_object61440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 272.06960546800002, 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098210427000000003, 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61441 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174498+0000", 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6130000000000001e-06 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174543+0000", 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5485999999999998e-05 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174552+0000", 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174557+0000", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2220000000000001e-06 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175099+0000", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000542155 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175220+0000", 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121154 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.251 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175330+0000", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109385 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176611+0000", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001281199 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.182295+0000", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056836250000000003 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185068+0000", 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027733480000000001 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185119+0000", 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1099e-05 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272707+0000", 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087587571000000003 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61442 7.5 7:af9252c3:::benchmark_data_smithi043_74356_object61441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 272.06728918800002, 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096090660999999994, 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.254 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61442 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:40:24.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176814+0000", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.533e-06 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176873+0000", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8678000000000003e-05 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176883+0000", 2024-04-25T05:40:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0701e-05 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176892+0000", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4479999999999994e-06 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176940+0000", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8111000000000001e-05 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177024+0000", 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4243e-05 2024-04-25T05:40:24.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177104+0000", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9800000000000002e-05 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.178395+0000", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001291489 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184162+0000", 2024-04-25T05:40:24.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005766658 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272787+0000", 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088625132999999995 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272844+0000", 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209000000000001e-05 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272903+0000", 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8658e-05 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130500 7.5 7:a4f80177:::benchmark_data_smithi043_74356_object13631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 200.11689887899999, 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10185490899999999, 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.260 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.261 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.261 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.261 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130500 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:40:24.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127204+0000", 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4050000000000001e-06 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127211+0000", 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6930000000000001e-06 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127225+0000", 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3815000000000001e-05 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127234+0000", 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1719999999999996e-06 2024-04-25T05:40:24.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127294+0000", 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9904000000000001e-05 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127421+0000", 2024-04-25T05:40:24.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012758499999999999 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127543+0000", 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012183199999999999 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.267 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.129963+0000", 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024198269999999998 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.130082+0000", 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.228908+0000", 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098825536000000005 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229009+0000", 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101307 2024-04-25T05:40:24.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229058+0000", 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8513000000000001e-05 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131177 7.5 7:a3c24791:::benchmark_data_smithi043_74356_object14308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 199.186674619, 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1201996, 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.271 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131177 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057428+0000", 2024-04-25T05:40:24.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5799999999999998e-07 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057435+0000", 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3420000000000002e-06 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057451+0000", 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6067999999999999e-05 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057460+0000", 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8990000000000007e-06 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058761+0000", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013012029999999999 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058890+0000", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012905600000000001 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.059004+0000", 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011395699999999999 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.061474+0000", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024696000000000002 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.063725+0000", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022509140000000001 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177555+0000", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11383001700000001 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177596+0000", 2024-04-25T05:40:24.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0970000000000002e-05 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177627+0000", 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0815999999999998e-05 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131521 7.5 7:a454f61c:::benchmark_data_smithi043_74356_object14652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 198.67104039099999, 2024-04-25T05:40:24.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14413125099999999, 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131521 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573063+0000", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.449e-06 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573069+0000", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1099999999999999e-06 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573082+0000", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3181e-05 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573089+0000", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2300000000000002e-06 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573149+0000", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9386999999999999e-05 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573243+0000", 2024-04-25T05:40:24.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4758000000000003e-05 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573328+0000", 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4320999999999994e-05 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.575369+0000", 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00204163 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.577122+0000", 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017531059999999999 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717083+0000", 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13996059199999999 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717144+0000", 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1328000000000006e-05 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717193+0000", 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8158999999999997e-05 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.285 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131530 7.5 7:a6cf53ae:::benchmark_data_smithi043_74356_object14661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 198.62943135699999, 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10267322399999999, 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131530 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614672+0000", 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2419999999999999e-06 2024-04-25T05:40:24.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614677+0000", 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5010000000000004e-06 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614688+0000", 2024-04-25T05:40:24.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1085e-05 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614697+0000", 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4749999999999993e-06 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614749+0000", 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2003000000000003e-05 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614882+0000", 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013371999999999999 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.615037+0000", 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015419300000000001 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.617721+0000", 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026844970000000001 2024-04-25T05:40:24.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.621195+0000", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034735780000000002 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717252+0000", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096057038999999997 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717303+0000", 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1363000000000003e-05 2024-04-25T05:40:24.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717344+0000", 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0528000000000002e-05 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:188543 7.5 7:ac5985ff:::benchmark_data_smithi043_74356_object71674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 122.925770094, 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10455339800000001, 2024-04-25T05:40:24.294 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 188543 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318333+0000", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3009999999999999e-06 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318338+0000", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9389999999999996e-06 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318349+0000", 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1216999999999999e-05 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318357+0000", 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6359999999999994e-06 2024-04-25T05:40:24.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318398+0000", 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1177999999999998e-05 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318596+0000", 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019781900000000001 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:40:24.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318760+0000", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000164198 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321782+0000", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003021714 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422025+0000", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10024319900000001 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422069+0000", 2024-04-25T05:40:24.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3764999999999999e-05 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422680+0000", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061150799999999997 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422770+0000", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0245000000000001e-05 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422885+0000", 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114679 2024-04-25T05:40:24.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:188544 7.8 e165/88 7:16da1a97:::benchmark_data_smithi043_74356_object71675:head v 165'11663, mlcod=165'11660)", 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 122.922820566, 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100748163, 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321282+0000", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5000000000000002e-07 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321292+0000", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0033e-05 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321300+0000", 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4540000000000007e-06 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321307+0000", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.742e-06 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321365+0000", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7552000000000001e-05 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321377+0000", 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2500999999999999e-05 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421970+0000", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100592746 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422029+0000", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9484999999999997e-05 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPushReply(7.6 165/160 [PushReplyOp(7:6e80069b:::benchmark_data_smithi043_74356_object89093:head)])", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:40:24.305 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 122.921335101, 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100303718, 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:40:24.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6100000000000002e-07 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322770+0000", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0000000000000001e-06 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322775+0000", 2024-04-25T05:40:24.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6929999999999998e-06 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421939+0000", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099164609000000001 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422361+0000", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00042228799999999999 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.423070+0000", 2024-04-25T05:40:24.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00070886699999999996 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:188543 7.5 e165/160 ondisk, result = 0)", 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 122.920041106, 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097973324000000001, 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5799999999999999e-07 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324064+0000", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8289999999999999e-06 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421928+0000", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097864313999999994 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421944+0000", 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5991e-05 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422007+0000", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2378000000000005e-05 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422017+0000", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0069e-05 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422034+0000", 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7184999999999999e-05 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:188543 7.5 e165/160 ondisk, result = 0)", 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 122.91858845500001, 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096562752000000002, 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325514+0000", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2300000000000001e-07 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325518+0000", 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2030000000000002e-06 2024-04-25T05:40:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421938+0000", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096420230999999995 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421950+0000", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1931e-05 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422056+0000", 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105924 2024-04-25T05:40:24.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422063+0000", 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8070000000000002e-06 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422076+0000", 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3033e-05 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.316 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195837 7.5 7:ad811058:::benchmark_data_smithi043_74356_object78968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.988039794, 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096179872999999999, 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195837 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256063+0000", 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5999999999999999e-06 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256068+0000", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8849999999999998e-06 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256080+0000", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1585e-05 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256087+0000", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3289999999999996e-06 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256191+0000", 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103528 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256285+0000", 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3924000000000004e-05 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256361+0000", 2024-04-25T05:40:24.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6687999999999995e-05 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.260072+0000", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037107910000000002 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.262721+0000", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026484740000000001 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.263704+0000", 2024-04-25T05:40:24.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0009836109999999999 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.351998+0000", 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088294124000000002 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.352074+0000", 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5123000000000006e-05 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.352242+0000", 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016821100000000001 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196430 7.6 7:67a0b441:::benchmark_data_smithi043_74356_object79561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.345609376, 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17769133400000001, 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196430 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:40:24.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898496+0000", 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5209999999999998e-06 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898500+0000", 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9099999999999998e-06 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898509+0000", 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0329999999999997e-06 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898514+0000", 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5659999999999997e-06 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898569+0000", 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4317999999999997e-05 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898692+0000", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012370400000000001 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898805+0000", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011234300000000001 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900717+0000", 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001912569 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900846+0000", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128825 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076080+0000", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17523397700000001 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076138+0000", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7815000000000001e-05 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076184+0000", 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5753000000000001e-05 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196431 7.6 7:64208f17:::benchmark_data_smithi043_74356_object79562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.34259452400001, 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17835189000000001, 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.329 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196431 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901509+0000", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7439999999999999e-06 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901517+0000", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8520000000000004e-06 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901526+0000", 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4679999999999998e-06 2024-04-25T05:40:24.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901534+0000", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8720000000000002e-06 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901588+0000", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3638999999999999e-05 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901700+0000", 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112355 2024-04-25T05:40:24.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901816+0000", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116359 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.903841+0000", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020240940000000002 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907001+0000", 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031600640000000002 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079789+0000", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17278797700000001 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079821+0000", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2480000000000001e-05 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079859+0000", 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7985999999999998e-05 2024-04-25T05:40:24.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196437 7.5 7:a3615d17:::benchmark_data_smithi043_74356_object79568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.33955516899999, 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16688376599999999, 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.335 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196437 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904548+0000", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904553+0000", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2429999999999996e-06 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904563+0000", 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0046e-05 2024-04-25T05:40:24.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:24.339 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 90391 90375 99.0475 81.6875 0.0104065 0.0100913 2024-04-25T05:40:24.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904571+0000", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0169999999999996e-06 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904710+0000", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013878799999999999 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904838+0000", 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128014 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904935+0000", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7627999999999994e-05 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.906988+0000", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020523189999999999 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908135+0000", 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00114685 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908341+0000", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020669199999999999 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071248+0000", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.162907146 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071322+0000", 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3274000000000004e-05 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071430+0000", 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010847400000000001 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196438 7.6 7:6894f35c:::benchmark_data_smithi043_74356_object79569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.339450441, 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17980945200000001, 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.343 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196438 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904652+0000", 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2099999999999996e-07 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904655+0000", 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4249999999999998e-06 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904666+0000", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185999999999999e-05 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904673+0000", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8009999999999998e-06 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904726+0000", 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2479999999999999e-05 2024-04-25T05:40:24.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904836+0000", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110016 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904947+0000", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111721 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907188+0000", 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022400810000000001 2024-04-25T05:40:24.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907463+0000", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027584899999999999 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084360+0000", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17689687000000001 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084421+0000", 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1032000000000001e-05 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084461+0000", 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9369999999999997e-05 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198457 7.5 7:aeb56927:::benchmark_data_smithi043_74356_object81588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 108.439906169, 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97568060099999998, 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.349 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198457 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:40:24.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804197+0000", 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.544e-06 2024-04-25T05:40:24.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804204+0000", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7569999999999998e-06 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804216+0000", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1841000000000001e-05 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804224+0000", 2024-04-25T05:40:24.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7099999999999996e-06 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804278+0000", 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3275000000000001e-05 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804284+0000", 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6030000000000001e-06 2024-04-25T05:40:24.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773320+0000", 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96903589899999998 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773471+0000", 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015128900000000001 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:40:24.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773614+0000", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014288699999999999 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777300+0000", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036858519999999999 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778657+0000", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013569280000000001 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779814+0000", 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001156481 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779846+0000", 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2005999999999998e-05 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779876+0000", 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0528999999999999e-05 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198459 7.5 7:a46444d1:::benchmark_data_smithi043_74356_object81590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 108.439563073, 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.977629847, 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.357 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198459 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3499999999999996e-07 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804544+0000", 2024-04-25T05:40:24.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3390000000000003e-06 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804553+0000", 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.969e-06 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804559+0000", 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.55e-06 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804600+0000", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1149999999999997e-05 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804606+0000", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4179999999999996e-06 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775688+0000", 2024-04-25T05:40:24.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97108227300000005 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775807+0000", 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011906899999999999 2024-04-25T05:40:24.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775929+0000", 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012165199999999999 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779949+0000", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040201059999999999 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779988+0000", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8943000000000003e-05 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782076+0000", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020885729999999998 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782128+0000", 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2080999999999998e-05 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782168+0000", 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0089000000000002e-05 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198466 7.5 7:a9a716c6:::benchmark_data_smithi043_74356_object81597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 108.43464364499999, 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97804308600000001, 2024-04-25T05:40:24.365 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198466 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809459+0000", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4529999999999999e-06 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809465+0000", 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8669999999999999e-06 2024-04-25T05:40:24.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809479+0000", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.324e-05 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809487+0000", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2020000000000004e-06 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809543+0000", 2024-04-25T05:40:24.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6376999999999999e-05 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809555+0000", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1435e-05 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777347+0000", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96779198399999999 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777465+0000", 2024-04-25T05:40:24.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011815999999999999 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777591+0000", 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012601299999999999 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779953+0000", 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023620949999999998 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782689+0000", 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027361059999999999 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787431+0000", 2024-04-25T05:40:24.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047419230000000003 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787467+0000", 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6081000000000002e-05 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787501+0000", 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4150000000000003e-05 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198479 7.5 7:a8eb3f00:::benchmark_data_smithi043_74356_object81610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 108.420948697, 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96450478500000003, 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198479 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:40:24.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:40:24.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823155+0000", 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.717e-06 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823172+0000", 2024-04-25T05:40:24.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7611000000000001e-05 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823191+0000", 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8842e-05 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:40:24.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823202+0000", 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0407e-05 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823258+0000", 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6428999999999998e-05 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:40:24.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823265+0000", 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4699999999999996e-06 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778700+0000", 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95543422700000002 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778769+0000", 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9831000000000003e-05 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778858+0000", 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8744000000000003e-05 2024-04-25T05:40:24.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779956+0000", 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010981400000000001 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782730+0000", 2024-04-25T05:40:24.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002773546 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787552+0000", 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048221590000000003 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787612+0000", 2024-04-25T05:40:24.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9880999999999998e-05 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787658+0000", 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5781000000000001e-05 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:40:24.383 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:40:24.383 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:40:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:24.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:24.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:24.520 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:24.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:24.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:24.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:24.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:24.800 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:24.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:24.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:24.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:25.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:25.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:25.106 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:24 smithi043 ceph-mon[27010]: pgmap v328: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:40:25.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:24 smithi043 ceph-mon[37492]: pgmap v328: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:40:25.135 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:24 smithi060 ceph-mon[30898]: pgmap v328: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:40:25.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:25.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:40:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:25.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:25.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:40:25.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 91675 91659 98.7218 80.25 0.00952163 0.0101244 2024-04-25T05:40:25.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.348 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:25.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:25.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:25.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:25.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:25.639 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:25.727 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:25.731 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:25.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:25.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:40:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:set 6.3 pg_upmap_items mapping to [3->2,7->1,6->0] 2024-04-25T05:40:25.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:25.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:25.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:26.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:26.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:26.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:26.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2391560875' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.110 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:26.110 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2391560875' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.110 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:25 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:26.219 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:26.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2391560875' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:25 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]: dispatch 2024-04-25T05:40:26.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 15 92859 92844 98.3039 74.0625 0.01141 0.0101677 2024-04-25T05:40:26.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:26.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:26.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:26.484 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:26.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:26.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:26.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:26.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:26.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:26.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:27.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:27.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:27.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[27010]: pgmap v329: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-04-25T05:40:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]': finished 2024-04-25T05:40:27.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[27010]: osdmap e228: 8 total, 8 up, 7 in 2024-04-25T05:40:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[37492]: pgmap v329: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-04-25T05:40:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]': finished 2024-04-25T05:40:27.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:26 smithi043 ceph-mon[37492]: osdmap e228: 8 total, 8 up, 7 in 2024-04-25T05:40:27.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:27.211 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:26 smithi060 ceph-mon[30898]: pgmap v329: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-04-25T05:40:27.211 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:26 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:27.211 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1184798656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:27.211 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.3", "id": [3, 2, 7, 1, 6, 0]}]': finished 2024-04-25T05:40:27.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:26 smithi060 ceph-mon[30898]: osdmap e228: 8 total, 8 up, 7 in 2024-04-25T05:40:27.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:27.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:40:27.300216+0000 min lat: 0.00119975 max lat: 1.57603 avg lat: 0.0101963 2024-04-25T05:40:27.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:40:27.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 94171 94155 98.0311 81.9375 0.00665685 0.0101963 2024-04-25T05:40:27.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:27.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:27.426 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:40:27.426 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:40:27.426 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:40:27.427 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:40:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:27.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:27.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:27.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:27.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:27.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:27.852 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:27.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:27.987 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:27.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:28.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:28.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:28.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:27 smithi043 ceph-mon[27010]: osdmap e229: 8 total, 8 up, 7 in 2024-04-25T05:40:28.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:27 smithi043 ceph-mon[37492]: osdmap e229: 8 total, 8 up, 7 in 2024-04-25T05:40:28.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:28.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:27 smithi060 ceph-mon[30898]: osdmap e229: 8 total, 8 up, 7 in 2024-04-25T05:40:28.260 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:28.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 95534 95518 97.8205 85.1875 0.00928285 0.0102179 2024-04-25T05:40:28.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:28.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:28.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:28.536 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:28.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:28.672 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:28.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:28.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:28.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:29.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:29.082 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:28 smithi043 ceph-mon[27010]: pgmap v332: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-04-25T05:40:29.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:28 smithi043 ceph-mon[37492]: pgmap v332: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-04-25T05:40:29.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:29.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:29.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:28 smithi060 ceph-mon[30898]: pgmap v332: 158 pgs: 158 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 62 MiB/s wr, 990 op/s 2024-04-25T05:40:29.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 96910 96894 97.6298 86 0.0124289 0.010238 2024-04-25T05:40:29.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:29.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:29.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:29.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:29.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:29.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:29.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:29.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:29.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:29.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:29.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.026 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:30.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:30.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:30.160 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:30.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:30.292 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:30.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 98218 98202 97.3776 81.75 0.00431851 0.0102624 2024-04-25T05:40:30.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:30.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:30.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.565 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:30.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:30.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:30.700 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:30.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:40:30.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:30.884 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:30.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:30.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:31.009 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:30 smithi043 ceph-mon[27010]: pgmap v333: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-04-25T05:40:31.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:30 smithi043 ceph-mon[37492]: pgmap v333: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-04-25T05:40:31.018 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:31.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:31.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:30 smithi060 ceph-mon[30898]: pgmap v333: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-04-25T05:40:31.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:31.299 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 99521 99505 97.1286 81.4375 0.00806278 0.0102912 2024-04-25T05:40:31.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:31.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:31.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:31.567 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:31.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:31.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:31.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:31.842 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:31.848 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:31.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:31.883 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:31.929 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:32.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:32.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:32.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:32.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:32.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:32.151 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2306152039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:32.151 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2306152039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:32.151 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:32.151 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:32.152 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:32.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:32.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2306152039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:32.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:32.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 100908 100892 96.9679 86.6875 0.00879995 0.0103081 2024-04-25T05:40:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:32.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:32.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:32.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:32.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:32.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:32.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.774 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:32.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:32.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:32.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:32.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:33.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:33.047 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:33.060 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:33.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:33.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:33.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:33.301 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 102315 102299 96.8283 87.9375 0.00795537 0.0103226 2024-04-25T05:40:33.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:33.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:33.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:33.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:33.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:33.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[27010]: pgmap v334: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-25T05:40:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:33.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:33.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[37492]: pgmap v334: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-25T05:40:33.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:33.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:33.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:33.721 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:33 smithi060 ceph-mon[30898]: pgmap v334: 158 pgs: 158 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-25T05:40:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:33.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:33.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:33.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:33.992 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:33.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:34.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:34.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:34.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:34.272 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:34.272 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:34.302 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 103681 103665 96.6556 85.375 0.0121621 0.0103415 2024-04-25T05:40:34.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:40:34.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:34.396 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:34.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:34.531 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:34.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:34.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.667 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:34.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:34.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:34.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:34.803 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:34.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:34.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:34.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:35.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:40:35.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:35.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:40:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:35.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:35.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:35.230 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:35.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 104990 104974 96.4373 81.8125 0.0174073 0.0103642 2024-04-25T05:40:35.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:35.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:35.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:35.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:35 smithi043 ceph-mon[27010]: pgmap v335: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:40:35.607 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:35 smithi043 ceph-mon[37492]: pgmap v335: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:40:35.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:35.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:35.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:35 smithi060 ceph-mon[30898]: pgmap v335: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:40:35.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:35.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:35.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:35.969 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:35.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:35.997 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:36.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:36.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:36.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:36.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:36.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 106262 106246 96.1918 79.5 0.0164973 0.0103913 2024-04-25T05:40:36.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:36.364 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:40:36.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:40:36.365 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-25T05:40:36.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd in 0 2024-04-25T05:40:36.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:36.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:36.514 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:36 smithi043 ceph-mon[27010]: pgmap v336: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:40:36.515 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:36 smithi043 ceph-mon[37492]: pgmap v336: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:40:36.519 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:36.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:36.653 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:36.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:36 smithi060 ceph-mon[30898]: pgmap v336: 158 pgs: 158 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:40:36.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:36.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:36.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:36.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:36.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:37.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:37.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.081 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:37.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:37.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:37.185 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:40:37.185 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:129499 7.9 7:92be4db5:::benchmark_data_smithi043_74356_object12630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:02.674324+0000", 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 214.507125409, 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.158158674, 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.186 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 129499 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674324+0000", 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.187 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674324+0000", 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674325+0000", 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4389999999999999e-06 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674330+0000", 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0760000000000002e-06 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674340+0000", 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9969999999999993e-06 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674348+0000", 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6410000000000002e-06 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674405+0000", 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7160000000000002e-05 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674549+0000", 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000144318 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.674665+0000", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115666 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.677141+0000", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024760250000000002 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.677161+0000", 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0290999999999999e-05 2024-04-25T05:40:37.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.832289+0000", 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15512769400000001 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.832438+0000", 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014882400000000001 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.832482+0000", 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4542999999999997e-05 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:129504 7.4 7:2834db13:::benchmark_data_smithi043_74356_object12635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:02.679801+0000", 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 214.501647617, 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15213449300000001, 2024-04-25T05:40:37.193 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 129504 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679801+0000", 2024-04-25T05:40:37.194 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679801+0000", 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679803+0000", 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2270000000000001e-06 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679808+0000", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1830000000000002e-06 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679817+0000", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5130000000000002e-06 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679823+0000", 2024-04-25T05:40:37.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6080000000000003e-06 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.679873+0000", 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0500000000000001e-05 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.737505+0000", 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.057631231999999998 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.737653+0000", 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000148646 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.740519+0000", 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028656649999999999 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.741286+0000", 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00076744499999999995 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.831823+0000", 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.090537049999999994 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.831887+0000", 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3448000000000006e-05 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:02.831936+0000", 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8976e-05 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:131509 7.6 7:6c73bdbd:::benchmark_data_smithi043_74356_object14640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.558735+0000", 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 211.622713984, 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16192921699999999, 2024-04-25T05:40:37.200 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 131509 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558735+0000", 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558735+0000", 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.202 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558735+0000", 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7399999999999999e-07 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558738+0000", 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5959999999999998e-06 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558743+0000", 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2780000000000001e-06 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.558748+0000", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1120000000000004e-06 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561546+0000", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027975019999999999 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561657+0000", 2024-04-25T05:40:37.204 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111351 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561763+0000", 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105964 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.616778+0000", 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.055014533999999997 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.205 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720132+0000", 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.103354238 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720573+0000", 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00044061099999999999 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.206 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720616+0000", 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3312000000000001e-05 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720664+0000", 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8344999999999999e-05 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.207 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:131512 7.6 7:6a76a03c:::benchmark_data_smithi043_74356_object14643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.561289+0000", 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 211.620159884, 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.159463458, 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 131512 2024-04-25T05:40:37.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561289+0000", 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561289+0000", 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561290+0000", 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.077e-06 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561296+0000", 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5969999999999999e-06 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.210 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561306+0000", 2024-04-25T05:40:37.211 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.063e-05 2024-04-25T05:40:37.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.561314+0000", 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0600000000000008e-06 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.612890+0000", 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.051575910000000003 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.214 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.612994+0000", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000103954 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.613099+0000", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010438999999999999 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720092+0000", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10699288899999999 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.215 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720202+0000", 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011005699999999999 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720674+0000", 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00047191600000000001 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720715+0000", 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1205999999999998e-05 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.216 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720753+0000", 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7771999999999997e-05 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:131514 7.4 7:2f2ae373:::benchmark_data_smithi043_74356_object14645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.562108+0000", 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 211.61934051200001, 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15253266200000001, 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.217 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 131514 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.562108+0000", 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.219 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.219 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.562108+0000", 2024-04-25T05:40:37.219 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.562109+0000", 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.006e-06 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.220 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.221 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.562116+0000", 2024-04-25T05:40:37.221 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0439999999999998e-06 2024-04-25T05:40:37.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.609927+0000", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.047810770000000002 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.609939+0000", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1902000000000001e-05 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.609989+0000", 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0311e-05 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610120+0000", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013101500000000001 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610249+0000", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012814799999999999 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.616080+0000", 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058317120000000002 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.714515+0000", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.098434835999999998 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.714552+0000", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6958000000000001e-05 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.714600+0000", 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8075000000000003e-05 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.224 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.714641+0000", 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0884999999999999e-05 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:131527 7.6 7:680030d7:::benchmark_data_smithi043_74356_object14658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:05.610684+0000", 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 211.57076457700001, 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.176845734, 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 131527 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610684+0000", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610684+0000", 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610685+0000", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6600000000000005e-07 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610690+0000", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.656e-06 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610700+0000", 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0285000000000001e-05 2024-04-25T05:40:37.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.610707+0000", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3250000000000002e-06 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.616810+0000", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0061027069999999997 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.616938+0000", 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128255 2024-04-25T05:40:37.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.715409+0000", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.098470797999999998 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720166+0000", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047567599999999996 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.720759+0000", 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00059284099999999998 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.787431+0000", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.066671803000000002 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.787484+0000", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3662000000000002e-05 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:05.787530+0000", 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5775999999999997e-05 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146292 7.4 7:27c079d3:::benchmark_data_smithi043_74356_object29423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 193.10260521800001, 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.830877023, 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146292 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.231 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078845+0000", 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.22e-06 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078850+0000", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8199999999999996e-06 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078865+0000", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5109000000000001e-05 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078872+0000", 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0040000000000002e-06 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079233+0000", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036102900000000002 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079358+0000", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012519399999999999 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079469+0000", 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110436 2024-04-25T05:40:37.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087939+0000", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0084700910000000008 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087953+0000", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4079e-05 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734514+0000", 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.646560994 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734529+0000", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4897e-05 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869085+0000", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1345559490000001 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869249+0000", 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016453999999999999 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909625+0000", 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.040376196000000003 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909721+0000", 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5464999999999995e-05 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146325 7.4 7:287eff09:::benchmark_data_smithi043_74356_object29456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.237 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 193.059063084, 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.78676667, 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146325 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:40:37.238 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122388+0000", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8509999999999999e-06 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122394+0000", 2024-04-25T05:40:37.239 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4629999999999998e-06 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122407+0000", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2965e-05 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122416+0000", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1439999999999992e-06 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122487+0000", 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0815000000000006e-05 2024-04-25T05:40:37.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122500+0000", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2641e-05 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734558+0000", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.61205790800000004 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734567+0000", 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869583+0000", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.135015624 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869717+0000", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013412499999999999 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869897+0000", 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018037399999999999 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876526+0000", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066293460000000004 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882816+0000", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062898030000000001 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889558+0000", 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067413789999999996 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.908921+0000", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.01936336 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909004+0000", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3045000000000002e-05 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909153+0000", 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014846799999999999 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146329 7.4 7:2d613d15:::benchmark_data_smithi043_74356_object29460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 193.05648918200001, 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.784987538, 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146329 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.245 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124962+0000", 2024-04-25T05:40:37.246 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.835e-06 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124968+0000", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5849999999999996e-06 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124986+0000", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.823e-05 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124996+0000", 2024-04-25T05:40:37.247 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3719999999999994e-06 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125091+0000", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5061000000000003e-05 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125102+0000", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.094e-05 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734591+0000", 2024-04-25T05:40:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.60948939099999999 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734600+0000", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4509999999999992e-06 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872375+0000", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1377753349999999 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872499+0000", 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124499 2024-04-25T05:40:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872641+0000", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141438 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876579+0000", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039384210000000001 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882926+0000", 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0063468170000000003 2024-04-25T05:40:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889565+0000", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066388469999999998 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909760+0000", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020195310000000001 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909819+0000", 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8798000000000002e-05 2024-04-25T05:40:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909947+0000", 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012820799999999999 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146354 7.4 7:2311a945:::benchmark_data_smithi043_74356_object29485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 193.016267849, 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.745096427, 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146354 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165182+0000", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.32e-07 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165184+0000", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5189999999999999e-06 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165189+0000", 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3580000000000002e-06 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165195+0000", 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.356e-06 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165284+0000", 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8756000000000002e-05 2024-04-25T05:40:37.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165295+0000", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1338000000000001e-05 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734624+0000", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.56932913799999996 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734632+0000", 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4710000000000007e-06 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876617+0000", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.141984085 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876755+0000", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138923 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876917+0000", 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016150299999999999 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882874+0000", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059565750000000004 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883014+0000", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013997799999999999 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889572+0000", 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065581820000000001 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910091+0000", 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.02051905 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910155+0000", 2024-04-25T05:40:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:37.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4060000000000007e-05 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910278+0000", 2024-04-25T05:40:37.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012270299999999999 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146371 7.4 7:20dc29ab:::benchmark_data_smithi043_74356_object29502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.99821578199999, 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7272659080000001, 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.262 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146371 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183234+0000", 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.46e-07 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183236+0000", 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4040000000000002e-06 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183240+0000", 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4000000000000002e-06 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.264 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183245+0000", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6750000000000001e-06 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183308+0000", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2946000000000003e-05 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183320+0000", 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1895e-05 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734658+0000", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.55133849499999998 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734668+0000", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0145e-05 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879759+0000", 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.145090291 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879914+0000", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154835 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.880070+0000", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015658800000000001 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882969+0000", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028987420000000002 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889540+0000", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065711279999999999 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889579+0000", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8476999999999998e-05 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910326+0000", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020747422000000001 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910386+0000", 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0270999999999999e-05 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910499+0000", 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112848 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146375 7.4 7:20e96b3d:::benchmark_data_smithi043_74356_object29506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.993430848, 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7227024800000001, 2024-04-25T05:40:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146375 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188019+0000", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.333e-06 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188025+0000", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7180000000000001e-06 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188038+0000", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3264999999999999e-05 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188047+0000", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0450000000000006e-06 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188122+0000", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4140000000000005e-05 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188132+0000", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0737000000000001e-05 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734692+0000", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.54655990700000001 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734701+0000", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2749999999999995e-06 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883051+0000", 2024-04-25T05:40:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1483502480000001 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883195+0000", 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014417400000000001 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883347+0000", 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015209500000000001 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889584+0000", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062372699999999996 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889631+0000", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7012999999999999e-05 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889684+0000", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3078999999999998e-05 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910550+0000", 2024-04-25T05:40:37.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020865748999999999 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910610+0000", 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9883999999999998e-05 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910721+0000", 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110548 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146378 7.4 7:2089bf57:::benchmark_data_smithi043_74356_object29509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.98475516799999, 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.714255568, 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146378 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.277 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196695+0000", 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.491e-06 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196703+0000", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4789999999999999e-06 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196715+0000", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1938e-05 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196724+0000", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8410000000000006e-06 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196789+0000", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5946999999999997e-05 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196801+0000", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1279e-05 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734725+0000", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53792390300000004 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734733+0000", 2024-04-25T05:40:37.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3599999999999996e-06 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886492+0000", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1517590280000001 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886663+0000", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000171094 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886838+0000", 2024-04-25T05:40:37.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017525099999999999 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889701+0000", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028622180000000001 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889754+0000", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3440000000000003e-05 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899237+0000", 2024-04-25T05:40:37.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0094831459999999996 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910763+0000", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011526290999999999 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910824+0000", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0829999999999999e-05 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910949+0000", 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012503199999999999 2024-04-25T05:40:37.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146380 7.4 7:2f11d6b2:::benchmark_data_smithi043_74356_object29511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.984030397, 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7323310589999998, 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146380 2024-04-25T05:40:37.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6000000000000003e-07 2024-04-25T05:40:37.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197424+0000", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.369e-06 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197441+0000", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7419999999999999e-05 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197449+0000", 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4910000000000005e-06 2024-04-25T05:40:37.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197538+0000", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8030999999999998e-05 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197548+0000", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0334e-05 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734761+0000", 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721291999999998 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734769+0000", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4290000000000001e-06 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889796+0000", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1550264050000001 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890050+0000", 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025425699999999997 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890229+0000", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017900300000000001 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899288+0000", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090589869999999992 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899330+0000", 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2453999999999997e-05 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911000+0000", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011669538 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929593+0000", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018593014000000001 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929639+0000", 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6319999999999997e-05 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929750+0000", 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011042700000000001 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146381 7.4 7:2ef47468:::benchmark_data_smithi043_74356_object29512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.983978987, 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.732786876, 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.291 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146381 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6800000000000001e-07 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197474+0000", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3900000000000002e-06 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197481+0000", 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7730000000000002e-06 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197487+0000", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.421e-06 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197574+0000", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7058000000000006e-05 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197582+0000", 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3489999999999992e-06 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734793+0000", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721096300000004 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734802+0000", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5159999999999994e-06 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895295+0000", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1604930549999999 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895455+0000", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015972800000000001 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895617+0000", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016215500000000001 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899375+0000", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037581239999999998 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909210+0000", 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098352049999999996 2024-04-25T05:40:37.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911053+0000", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018428959999999999 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930101+0000", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019048197999999999 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930150+0000", 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8424000000000003e-05 2024-04-25T05:40:37.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930257+0000", 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107153 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146407 7.4 7:23e67bf6:::benchmark_data_smithi043_74356_object29538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.94437139600001, 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.693338502, 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146407 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237079+0000", 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.516e-06 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237089+0000", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6940000000000007e-06 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237100+0000", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1114000000000001e-05 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237108+0000", 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1040000000000007e-06 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237170+0000", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1851000000000001e-05 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237180+0000", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.063e-05 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734826+0000", 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.497645477 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734834+0000", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1210000000000007e-06 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899407+0000", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.164572851 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899548+0000", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141142 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899702+0000", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015383899999999999 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909262+0000", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0095605629999999994 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909994+0000", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00073198100000000004 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926123+0000", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016128397999999999 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930265+0000", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041419300000000003 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930315+0000", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0584000000000002e-05 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930416+0000", 2024-04-25T05:40:37.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100707 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146411 7.4 7:2d6ecc61:::benchmark_data_smithi043_74356_object29542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.94212085000001, 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6912330820000001, 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.306 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146411 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.307 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239329+0000", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2399999999999997e-07 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239333+0000", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5249999999999999e-06 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239343+0000", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7049999999999994e-06 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239351+0000", 2024-04-25T05:40:37.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1499999999999999e-06 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239474+0000", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012281099999999999 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239484+0000", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9709999999999999e-06 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735030+0000", 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.49554585800000001 2024-04-25T05:40:37.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735043+0000", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3399e-05 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905732+0000", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.170688524 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905903+0000", 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017154299999999999 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.906084+0000", 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:37.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:37.312 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000180502 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910045+0000", 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039608480000000003 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916884+0000", 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0068390710000000004 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926215+0000", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.009331331 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930423+0000", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042081159999999996 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930465+0000", 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1239000000000003e-05 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930561+0000", 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6664999999999997e-05 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146416 7.4 7:20f93e30:::benchmark_data_smithi043_74356_object29547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 192.93435291899999, 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6920498500000001, 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146416 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247097+0000", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.268e-06 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247103+0000", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6799999999999998e-06 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247113+0000", 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5869999999999999e-06 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247120+0000", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8630000000000008e-06 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247187+0000", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6701999999999999e-05 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247197+0000", 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0013000000000001e-05 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736134+0000", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.48893706999999997 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736148+0000", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3709999999999999e-05 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911091+0000", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1749430030000001 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911228+0000", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013674 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911385+0000", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015695600000000001 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916945+0000", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055600700000000003 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930568+0000", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013623273999999999 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.931318+0000", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075026300000000004 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.935451+0000", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041326820000000004 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939020+0000", 2024-04-25T05:40:37.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035686450000000001 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939146+0000", 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012632400000000001 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175158 7.f 7:f679f422:::benchmark_data_smithi043_74356_object58289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.466671+0000", 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 153.71477818100001, 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15092359899999999, 2024-04-25T05:40:37.322 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175158 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466671+0000", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466671+0000", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466672+0000", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.16e-07 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466676+0000", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0890000000000002e-06 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466683+0000", 2024-04-25T05:40:37.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0990000000000001e-06 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.466690+0000", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.311e-06 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.474777+0000", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080866989999999993 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476682+0000", 2024-04-25T05:40:37.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019055020000000001 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.476794+0000", 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011171 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608520+0000", 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13172626700000001 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608564+0000", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3832999999999999e-05 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617490+0000", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0089256040000000002 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617553+0000", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3287999999999994e-05 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617594+0000", 2024-04-25T05:40:37.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1281e-05 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:175168 7.7 7:eacbd0f4:::benchmark_data_smithi043_74356_object58299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.480047+0000", 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 153.70140181400001, 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.149101698, 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:37.328 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 175168 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480047+0000", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480047+0000", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480048+0000", 2024-04-25T05:40:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.299e-06 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.480054+0000", 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8900000000000004e-06 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608326+0000", 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.128271837 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:37.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608335+0000", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8359999999999998e-06 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608625+0000", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00029046200000000001 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.608844+0000", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021829799999999999 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.609044+0000", 2024-04-25T05:40:37.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000199953 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617057+0000", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0080130340000000005 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.617091+0000", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4597000000000001e-05 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.629001+0000", 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011910041 2024-04-25T05:40:37.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.629099+0000", 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7454999999999994e-05 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.629149+0000", 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9996e-05 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:37.333 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:40:37.334 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 107515 107499 95.9364 78.3125 0.010021 0.0104186 2024-04-25T05:40:37.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:37.380 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:40:37.394 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:37.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:37.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:37.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:37.612 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2155660770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:37.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:37.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2155660770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:37.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:37.668 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:37.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:40:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2155660770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:37.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:37.806 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:37.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:37.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:38.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:38.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:38.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:38.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:38.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 108917 108901 95.8192 87.625 0.0100146 0.0104318 2024-04-25T05:40:38.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:38.348 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:38.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:38.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:38.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: osdmap e230: 8 total, 8 up, 7 in 2024-04-25T05:40:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:40:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 5.14 scrub starts 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 5.14 scrub ok 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: pgmap v338: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: osdmap e230: 8 total, 8 up, 7 in 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 5.14 scrub starts 2024-04-25T05:40:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 5.14 scrub ok 2024-04-25T05:40:38.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: pgmap v338: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T05:40:38.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:40:38.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:38 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:40:38.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3247158188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: osdmap e230: 8 total, 8 up, 7 in 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 5.14 scrub starts 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 5.14 scrub ok 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: pgmap v338: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:40:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:38 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:40:38.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:38.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:38.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:38.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:38.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:39.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:39.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.171 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:39.272 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:39.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 110261 110245 95.655 84 0.0146884 0.0104492 2024-04-25T05:40:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.313 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:39.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.454 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:39.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:39.590 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:39 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:40:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:39 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:40:39.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:39 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:40:39.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:39 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:40:39.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:39.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:39.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:39 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:40:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:39 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:40:39.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:39.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:39.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:39.997 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:39.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:40.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:40.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:40.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:40:40.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:40.256 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:40.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 111436 111420 95.3507 73.4375 0.0102613 0.0104827 2024-04-25T05:40:40.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:40.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:40.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: 5.17 scrub starts 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: 5.17 scrub ok 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: 8.2 scrub starts 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: 8.9 scrub starts 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: pgmap v339: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[27010]: osdmap e231: 8 total, 8 up, 7 in 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: 5.17 scrub starts 2024-04-25T05:40:40.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: 5.17 scrub ok 2024-04-25T05:40:40.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: 8.2 scrub starts 2024-04-25T05:40:40.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: 8.9 scrub starts 2024-04-25T05:40:40.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: pgmap v339: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T05:40:40.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:40 smithi043 ceph-mon[37492]: osdmap e231: 8 total, 8 up, 7 in 2024-04-25T05:40:40.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:40:40.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:40.578 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:40.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:40.712 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: 5.17 scrub starts 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: 5.17 scrub ok 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: 8.2 scrub starts 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: 8.9 scrub starts 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: pgmap v339: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-25T05:40:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:40 smithi060 ceph-mon[30898]: osdmap e231: 8 total, 8 up, 7 in 2024-04-25T05:40:40.815 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:40.852 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:40.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:40.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:41.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:41.119 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:41.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:41.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:41.303 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 15 112562 112547 95.0141 70.4375 0.010933 0.0105204 2024-04-25T05:40:41.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:41.388 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:41.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:41.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:40:41.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:40:41.401+0000 7f173ec68700 -1 mon.a@0(leader).osd e231 definitely_dead 0 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:40:41.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:41 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:40:41.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:40:41.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:41.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:40:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:41 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:40:41.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.799 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:41.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:41.934 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:41.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:42.036 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:42.061 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:42.074 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:42.101 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:42.146 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:42.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:42.198 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:42.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:42.249 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:42.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 113857 113841 94.8239 80.875 0.0185721 0.0105409 2024-04-25T05:40:42.306 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:42.306 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:42.310 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:42.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:42.348 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:42.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:42.393 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:42.434 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.0. 2024-04-25T05:40:42.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:42.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:42.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:42.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 8.2 scrub ok 2024-04-25T05:40:42.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:40:42.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:40:42.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:40:42.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 8.9 scrub ok 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3364290450' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 4.17 scrub starts 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: 4.17 scrub ok 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: pgmap v341: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1608719230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 8.2 scrub ok 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:40:42.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 8.9 scrub ok 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3364290450' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 4.17 scrub starts 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: 4.17 scrub ok 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: pgmap v341: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-04-25T05:40:42.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1608719230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:42.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:42.726 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 8.2 scrub ok 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 8.9 scrub ok 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3364290450' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:40:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 4.17 scrub starts 2024-04-25T05:40:42.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:40:42.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: 4.17 scrub ok 2024-04-25T05:40:42.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: pgmap v341: 158 pgs: 158 active+clean; 6.6 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 824 op/s 2024-04-25T05:40:42.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1608719230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:42.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:42.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:42.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:42.975 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:42.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:43.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:43.098 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:43.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:43.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:43.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 114004 113988 93.6974 9.1875 0.00282609 0.0105492 2024-04-25T05:40:43.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:43.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:43.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:43.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:43.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: 8.4 scrub starts 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: osdmap e232: 8 total, 8 up, 8 in 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:40:43.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[27010]: 8.4 scrub ok 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: 8.4 scrub starts 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: osdmap e232: 8 total, 8 up, 8 in 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:40:43.582 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:43.583 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:43 smithi043 ceph-mon[37492]: 8.4 scrub ok 2024-04-25T05:40:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:43.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:43.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:43.724 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: 8.4 scrub starts 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: osdmap e232: 8 total, 8 up, 8 in 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:43 smithi060 ceph-mon[30898]: 8.4 scrub ok 2024-04-25T05:40:43.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.848 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:43.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:43.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:43.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:44.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:44.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:44.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:44.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:40:44.230 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:44.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 114004 113988 92.481 0 - 0.0105492 2024-04-25T05:40:44.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:44.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:44.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:44.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:44.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:44.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[27010]: osdmap e233: 8 total, 8 up, 8 in 2024-04-25T05:40:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:40:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:40:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[27010]: pgmap v344: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-25T05:40:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[27010]: osdmap e234: 8 total, 8 up, 8 in 2024-04-25T05:40:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[37492]: osdmap e233: 8 total, 8 up, 8 in 2024-04-25T05:40:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:40:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:40:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[37492]: pgmap v344: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-25T05:40:44.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:44 smithi043 ceph-mon[37492]: osdmap e234: 8 total, 8 up, 8 in 2024-04-25T05:40:44.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:44.719 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:44.748 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-25T05:40:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:44 smithi060 ceph-mon[30898]: osdmap e233: 8 total, 8 up, 8 in 2024-04-25T05:40:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:44 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:40:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:44 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:40:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:44 smithi060 ceph-mon[30898]: pgmap v344: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-25T05:40:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:44 smithi060 ceph-mon[30898]: osdmap e234: 8 total, 8 up, 8 in 2024-04-25T05:40:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:44.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:44.901 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:44.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:44.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:45.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:45.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:45.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.229 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:45.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:45.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 114800 114784 91.9333 24.875 0.00970247 0.0108724 2024-04-25T05:40:45.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:45.367 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:45.453 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:45.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:45.499 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:45.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:45.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.c starting backfill to osd.0 from (0'0,0'0] MAX to 231'7160 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.6 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7144 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.e starting backfill to osd.0 from (0'0,0'0] MAX to 231'7029 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.9 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7073 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.1 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7299 2024-04-25T05:40:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 8.2 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7026 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[37492]: osdmap e235: 8 total, 8 up, 8 in 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.c starting backfill to osd.0 from (0'0,0'0] MAX to 231'7160 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.6 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7144 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.e starting backfill to osd.0 from (0'0,0'0] MAX to 231'7029 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.9 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7073 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.1 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7299 2024-04-25T05:40:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 8.2 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7026 2024-04-25T05:40:45.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-25T05:40:45.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:40:45.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:40:45.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:45 smithi043 ceph-mon[27010]: osdmap e235: 8 total, 8 up, 8 in 2024-04-25T05:40:45.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.c starting backfill to osd.0 from (0'0,0'0] MAX to 231'7160 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.6 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7144 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.e starting backfill to osd.0 from (0'0,0'0] MAX to 231'7029 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.9 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7073 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.1 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7299 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 8.2 starting backfill to osd.0 from (0'0,0'0] MAX to 231'7026 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: Health check failed: Reduced data availability: 14 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:40:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:45 smithi060 ceph-mon[30898]: osdmap e235: 8 total, 8 up, 8 in 2024-04-25T05:40:45.774 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:45.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:45.909 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:45.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:46.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:46.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:46.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:40:46.144 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:40:46.144 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:40:46.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:46.177 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:46.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:46.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 115776 115760 91.5418 61 0.0185297 0.0109187 2024-04-25T05:40:46.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:46.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:46.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:46.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:46.618 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:46.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:46.750 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:46.851 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:40:46.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:46.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:46.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:46.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:46.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:46 smithi060 ceph-mon[30898]: pgmap v347: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:40:46.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:46 smithi060 ceph-mon[30898]: osdmap e236: 8 total, 8 up, 8 in 2024-04-25T05:40:46.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:47.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:47.103 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[27010]: pgmap v347: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:40:47.103 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[27010]: osdmap e236: 8 total, 8 up, 8 in 2024-04-25T05:40:47.103 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:47.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[37492]: pgmap v347: 158 pgs: 1 active+clean+scrubbing, 4 unknown, 50 peering, 103 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:40:47.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[37492]: osdmap e236: 8 total, 8 up, 8 in 2024-04-25T05:40:47.104 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:40:47.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:47.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:47.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:47.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:40:47.306449+0000 min lat: 0.00119975 max lat: 2.14806 avg lat: 0.0109485 2024-04-25T05:40:47.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:40:47.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 116929 116913 91.2983 72.0625 0.0114859 0.0109485 2024-04-25T05:40:47.313 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:47.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:47.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:47.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:40:47.521 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:47.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:47.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:47.689 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:47.733 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:47.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:47.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:47.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:47.968 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:48.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:48.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:48.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 5.16 scrub starts 2024-04-25T05:40:48.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 5.16 scrub ok 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: osdmap e237: 8 total, 8 up, 8 in 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:48.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:40:48.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:40:48.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 3.1e deep-scrub starts 2024-04-25T05:40:48.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[27010]: 3.1e deep-scrub ok 2024-04-25T05:40:48.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 5.16 scrub starts 2024-04-25T05:40:48.140 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 5.16 scrub ok 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: osdmap e237: 8 total, 8 up, 8 in 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:48.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:40:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:40:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 3.1e deep-scrub starts 2024-04-25T05:40:48.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:47 smithi043 ceph-mon[37492]: 3.1e deep-scrub ok 2024-04-25T05:40:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.146 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:48.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:48.248 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 5.16 scrub starts 2024-04-25T05:40:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 5.16 scrub ok 2024-04-25T05:40:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:40:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1823982544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:40:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: osdmap e237: 8 total, 8 up, 8 in 2024-04-25T05:40:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:40:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:40:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:40:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:40:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 3.1e deep-scrub starts 2024-04-25T05:40:48.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:47 smithi060 ceph-mon[30898]: 3.1e deep-scrub ok 2024-04-25T05:40:48.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:48.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 118011 117995 91.006 67.625 0.0259826 0.0109836 2024-04-25T05:40:48.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:48.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:48.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:48.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:48.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:48.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:48.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:48.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:48.899 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:48.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:48.966 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:40:48.984 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:48.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:49.012 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:40:49.057 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:40:49.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:49.111 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:40:49.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[27010]: pgmap v350: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 4 unknown, 45 peering, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 57 op/s; 6773/279342 objects misplaced (2.425%); 7.4 MiB/s, 119 objects/s recovering 2024-04-25T05:40:49.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:40:49.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:40:49.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[37492]: pgmap v350: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 4 unknown, 45 peering, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 57 op/s; 6773/279342 objects misplaced (2.425%); 7.4 MiB/s, 119 objects/s recovering 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:40:49.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:48 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:40:49.130 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:49.152 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:40:49.194 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:40:49.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:49.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:40:49.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:48 smithi060 ceph-mon[30898]: pgmap v350: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 4 unknown, 45 peering, 107 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 57 op/s; 6773/279342 objects misplaced (2.425%); 7.4 MiB/s, 119 objects/s recovering 2024-04-25T05:40:49.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:48 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:40:49.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:48 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:40:49.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:48 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:40:49.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:48 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:40:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:49.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:49.280 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:40:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:40:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:40:49.305 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 119006 118990 90.6546 62.1875 0.0184568 0.0110255 2024-04-25T05:40:49.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:49.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:49.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:49.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:49.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.685 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:49.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:49.749 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 7, 6, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:40:49.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:40:49.750 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 3, 4, 5, 7, 6, 0] 2024-04-25T05:40:49.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd out 2 2024-04-25T05:40:49.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:49.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:49.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:49.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:49.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:50.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: osdmap e238: 8 total, 8 up, 8 in 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2337092179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: 8.a scrub starts 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: 4.11 scrub starts 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[27010]: osdmap e239: 8 total, 8 up, 8 in 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:40:50.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:40:50.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: osdmap e238: 8 total, 8 up, 8 in 2024-04-25T05:40:50.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2337092179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:50.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: 8.a scrub starts 2024-04-25T05:40:50.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: 4.11 scrub starts 2024-04-25T05:40:50.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:49 smithi043 ceph-mon[37492]: osdmap e239: 8 total, 8 up, 8 in 2024-04-25T05:40:50.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:50.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:50.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: osdmap e238: 8 total, 8 up, 8 in 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2337092179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: 8.a scrub starts 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: 4.11 scrub starts 2024-04-25T05:40:50.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:49 smithi060 ceph-mon[30898]: osdmap e239: 8 total, 8 up, 8 in 2024-04-25T05:40:50.250 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:40:50.250 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:40:50.250 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:40:50.250 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:40:50.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164389 7.b 7:d5a611de:::benchmark_data_smithi043_74356_object47520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.099288+0000", 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.145644336, 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0330987490000001, 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164389 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099288+0000", 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099288+0000", 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.252 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099290+0000", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.762e-06 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099295+0000", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2680000000000002e-06 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099311+0000", 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5126999999999999e-05 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.253 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099319+0000", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0609999999999996e-06 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099386+0000", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7432000000000001e-05 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.099397+0000", 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1415e-05 2024-04-25T05:40:50.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.112283+0000", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.012885993 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.112414+0000", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013056499999999999 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.112775+0000", 2024-04-25T05:40:50.255 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036104699999999997 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.116045+0000", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032698430000000001 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.120627+0000", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045823770000000003 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132286+0000", 2024-04-25T05:40:50.256 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011659213 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132339+0000", 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2290000000000002e-05 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132387+0000", 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8356000000000003e-05 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164394 7.b 7:d4d52732:::benchmark_data_smithi043_74356_object47525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.102318+0000", 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.142614928, 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.030278539, 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164394 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102318+0000", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102318+0000", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102319+0000", 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.471e-06 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102324+0000", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1170000000000003e-06 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102335+0000", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0224000000000001e-05 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102343+0000", 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5290000000000008e-06 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.260 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102414+0000", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0419000000000005e-05 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.102423+0000", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2459999999999992e-06 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.116080+0000", 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0136572770000001 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.261 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.116209+0000", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012935900000000001 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.116349+0000", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000139617 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.120675+0000", 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043257879999999997 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.262 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.125790+0000", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051147060000000001 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132494+0000", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067047829999999997 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132545+0000", 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0997e-05 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.263 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132596+0000", 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1005999999999999e-05 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164402 7.b 7:d5005ec9:::benchmark_data_smithi043_74356_object47533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.113140+0000", 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.13179262700001, 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.019554421, 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.264 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164402 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113140+0000", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113140+0000", 2024-04-25T05:40:50.265 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113142+0000", 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5120000000000001e-06 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113147+0000", 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.665e-06 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113158+0000", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0883999999999999e-05 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113166+0000", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9680000000000006e-06 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113227+0000", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1258000000000002e-05 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113236+0000", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6060000000000003e-06 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.118080+0000", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0048440219999999 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.118208+0000", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012772700000000001 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.118329+0000", 2024-04-25T05:40:50.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121551 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.125833+0000", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0075036410000000001 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.125882+0000", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9416000000000003e-05 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132603+0000", 2024-04-25T05:40:50.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067210780000000001 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132657+0000", 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3736999999999997e-05 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132695+0000", 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7355999999999999e-05 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164403 7.b 7:ddccdeb8:::benchmark_data_smithi043_74356_object47534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.113638+0000", 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.13129482400001, 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.019138248, 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164403 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113638+0000", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113638+0000", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113639+0000", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3300000000000001e-07 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113643+0000", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5990000000000004e-06 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113653+0000", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8770000000000005e-06 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113661+0000", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4150000000000002e-06 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113712+0000", 2024-04-25T05:40:50.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1279000000000002e-05 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.113720+0000", 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6580000000000002e-06 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.120710+0000", 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0069907 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.120816+0000", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001054 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.120942+0000", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012657599999999999 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.125923+0000", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049808750000000001 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.125962+0000", 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8933999999999997e-05 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132701+0000", 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067391739999999997 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:50.277 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132741+0000", 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9690000000000001e-05 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132776+0000", 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5237999999999997e-05 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164421 7.b 7:d3694314:::benchmark_data_smithi043_74356_object47552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.131225+0000", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.113707433, 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0016433149999999, 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164421 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131225+0000", 2024-04-25T05:40:50.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131225+0000", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131226+0000", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.178e-06 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131231+0000", 2024-04-25T05:40:50.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6380000000000003e-06 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131239+0000", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9840000000000001e-06 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131245+0000", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6779999999999997e-06 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131303+0000", 2024-04-25T05:40:50.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7907e-05 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.131313+0000", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0154e-05 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.123182+0000", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99186881599999999 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.123344+0000", 2024-04-25T05:40:50.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000162318 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.123466+0000", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012168 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.126003+0000", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025368550000000002 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.126045+0000", 2024-04-25T05:40:50.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2636000000000003e-05 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132783+0000", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067376040000000003 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132823+0000", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0021000000000002e-05 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132869+0000", 2024-04-25T05:40:50.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5846000000000001e-05 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164435 7.b 7:dfdb10cb:::benchmark_data_smithi043_74356_object47566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.147967+0000", 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.09696610899999, 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98503510400000005, 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.285 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164435 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147967+0000", 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147967+0000", 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147968+0000", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4640000000000001e-06 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147974+0000", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8309999999999997e-06 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147983+0000", 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4369999999999996e-06 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.147991+0000", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5299999999999999e-06 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.148051+0000", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9642999999999998e-05 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.148058+0000", 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7589999999999997e-06 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.126078+0000", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97802000499999997 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.126190+0000", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001114 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.126311+0000", 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121208 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.289 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132271+0000", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059605860000000004 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132876+0000", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00060410300000000004 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132916+0000", 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0534999999999997e-05 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.132964+0000", 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7435000000000002e-05 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133002+0000", 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8167999999999998e-05 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164454 7.b 7:d98aa1f5:::benchmark_data_smithi043_74356_object47585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.291 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.170366+0000", 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.07456685099999, 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96292113899999998, 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164454 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170366+0000", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170366+0000", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170367+0000", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.429e-06 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170373+0000", 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2859999999999999e-06 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170384+0000", 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1766999999999999e-05 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.170394+0000", 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2699999999999993e-06 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.171802+0000", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014088130000000001 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.171811+0000", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6799999999999999e-06 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.129310+0000", 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95749854099999998 2024-04-25T05:40:50.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.129426+0000", 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116648 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.129548+0000", 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000122111 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133011+0000", 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003462396 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133142+0000", 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013118 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133196+0000", 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4301000000000001e-05 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133249+0000", 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2510999999999999e-05 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133287+0000", 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8206e-05 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164504 7.b 7:d32fb72f:::benchmark_data_smithi043_74356_object47635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.224027+0000", 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.02090558200001, 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91800847600000002, 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164504 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224027+0000", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224027+0000", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224028+0000", 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.353e-06 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224034+0000", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2469999999999999e-06 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224044+0000", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0190999999999999e-05 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224053+0000", 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8519999999999993e-06 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224122+0000", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9109000000000005e-05 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.224131+0000", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3980000000000005e-06 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133320+0000", 2024-04-25T05:40:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.90918835899999995 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133596+0000", 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027676799999999999 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.133728+0000", 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013131800000000001 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.141851+0000", 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0081236810000000007 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.141918+0000", 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6752000000000001e-05 2024-04-25T05:40:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.141939+0000", 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1039e-05 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.141992+0000", 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3029000000000003e-05 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.142036+0000", 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3380000000000001e-05 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:164514 7.b 7:d5173e7c:::benchmark_data_smithi043_74356_object47645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:49.235886+0000", 2024-04-25T05:40:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 181.009046334, 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91719473799999995, 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 164514 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235886+0000", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235886+0000", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235888+0000", 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5290000000000001e-06 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235894+0000", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6420000000000004e-06 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235905+0000", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1549000000000001e-05 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.235914+0000", 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.551e-06 2024-04-25T05:40:50.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.236098+0000", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018454100000000001 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:49.236107+0000", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2050000000000008e-06 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.139210+0000", 2024-04-25T05:40:50.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.90310251900000005 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.139318+0000", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107742 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.139408+0000", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0037999999999994e-05 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.142121+0000", 2024-04-25T05:40:50.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027132530000000001 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.147463+0000", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053418720000000001 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.152984+0000", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055208699999999998 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.153039+0000", 2024-04-25T05:40:50.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5504000000000003e-05 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:50.153081+0000", 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1922999999999998e-05 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197864 7.9 7:93d3220a:::benchmark_data_smithi043_74356_object80995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.797776+0000", 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.447156707, 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99031608599999998, 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.313 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197864 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797776+0000", 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797776+0000", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797777+0000", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5799999999999999e-07 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797779+0000", 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.774e-06 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.315 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797785+0000", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8499999999999999e-06 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.797791+0000", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.015e-06 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.811743+0000", 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013952208000000001 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.811752+0000", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6840000000000002e-06 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.780243+0000", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.968490662 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.780579+0000", 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033602099999999999 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.780712+0000", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013302999999999999 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.787800+0000", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.007088006 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.787851+0000", 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0739000000000003e-05 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788019+0000", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016872999999999999 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788051+0000", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1986999999999997e-05 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788092+0000", 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0822000000000003e-05 2024-04-25T05:40:50.319 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197879 7.9 7:961e311a:::benchmark_data_smithi043_74356_object81010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.821100+0000", 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.42383225500001, 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97167690900000003, 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197879 2024-04-25T05:40:50.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821100+0000", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821100+0000", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821102+0000", 2024-04-25T05:40:50.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6950000000000001e-06 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821108+0000", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1890000000000004e-06 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821125+0000", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6220999999999999e-05 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821133+0000", 2024-04-25T05:40:50.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6500000000000002e-06 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821234+0000", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100895 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821243+0000", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1090000000000004e-06 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.783460+0000", 2024-04-25T05:40:50.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96221683099999999 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.783577+0000", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116699 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.783697+0000", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120497 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.787907+0000", 2024-04-25T05:40:50.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042096800000000004 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788005+0000", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7774000000000003e-05 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792692+0000", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046874519999999999 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792739+0000", 2024-04-25T05:40:50.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6536000000000003e-05 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792777+0000", 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8680999999999998e-05 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197883 7.9 7:99034877:::benchmark_data_smithi043_74356_object81014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.821157+0000", 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.42377573300001, 2024-04-25T05:40:50.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97178028400000005, 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197883 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821157+0000", 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821157+0000", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821158+0000", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4199999999999996e-07 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821161+0000", 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.128e-06 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821167+0000", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.302e-06 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821174+0000", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9070000000000001e-06 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821269+0000", 2024-04-25T05:40:50.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4690000000000003e-05 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.821275+0000", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9479999999999996e-06 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.785655+0000", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96438041100000005 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.785941+0000", 2024-04-25T05:40:50.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00028561500000000002 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.786063+0000", 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000122015 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.332 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 119794 119778 90.1555 49.25 0.0204707 0.0110873 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790338+0000", 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004275933 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790383+0000", 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4492e-05 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792838+0000", 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002454871 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792890+0000", 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.253e-05 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792937+0000", 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6900000000000002e-05 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197890 7.9 7:9afcd7bd:::benchmark_data_smithi043_74356_object81021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.826817+0000", 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.41811534199999, 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96620932000000004, 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197890 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.335 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826817+0000", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826817+0000", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826819+0000", 2024-04-25T05:40:50.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.246e-06 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826824+0000", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2279999999999998e-06 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826834+0000", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0192e-05 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826842+0000", 2024-04-25T05:40:50.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0430000000000008e-06 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826910+0000", 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.792e-05 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.826920+0000", 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0000000000000001e-05 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788122+0000", 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96120230100000004 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788239+0000", 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116859 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.788365+0000", 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012557200000000001 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790421+0000", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020561490000000002 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790456+0000", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4830999999999999e-05 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792945+0000", 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024895239999999999 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792985+0000", 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0247000000000003e-05 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.793027+0000", 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1208000000000003e-05 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197891 7.9 7:94b1b3a0:::benchmark_data_smithi043_74356_object81022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.827474+0000", 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.41745843199999, 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97093975499999996, 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197891 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.342 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827474+0000", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827474+0000", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827475+0000", 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9699999999999994e-07 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827481+0000", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3399999999999997e-06 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827528+0000", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7679000000000002e-05 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827538+0000", 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5859999999999994e-06 2024-04-25T05:40:50.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827596+0000", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7865999999999998e-05 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.827604+0000", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.303e-06 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790488+0000", 2024-04-25T05:40:50.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.962883763 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790603+0000", 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115165 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.790720+0000", 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117503 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.792826+0000", 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00210523 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.793033+0000", 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000207672 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798306+0000", 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0052721240000000004 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798367+0000", 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1136999999999994e-05 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798414+0000", 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7389999999999999e-05 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197894 7.9 7:90faa7af:::benchmark_data_smithi043_74356_object81025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.348 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.829499+0000", 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.41543348499999, 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96912824399999997, 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197894 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829499+0000", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829499+0000", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829500+0000", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0139999999999999e-06 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829505+0000", 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7110000000000003e-06 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829514+0000", 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0489999999999993e-06 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829522+0000", 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5859999999999999e-06 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.831630+0000", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021079710000000002 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.831638+0000", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5439999999999999e-06 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.793064+0000", 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96142607000000002 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.793163+0000", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9127000000000001e-05 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.793263+0000", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9381000000000003e-05 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798492+0000", 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0052292529999999997 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798504+0000", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2464e-05 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798540+0000", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5525000000000003e-05 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798590+0000", 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9719000000000003e-05 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798627+0000", 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7830000000000002e-05 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197895 7.9 7:93103050:::benchmark_data_smithi043_74356_object81026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.829542+0000", 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.41539103299999, 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97835268399999997, 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.355 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197895 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829542+0000", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829542+0000", 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829542+0000", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.01e-07 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829545+0000", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7089999999999998e-06 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.829551+0000", 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7969999999999997e-06 2024-04-25T05:40:50.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.831341+0000", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0017906840000000001 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.831670+0000", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00032910099999999999 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.831679+0000", 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6859999999999995e-06 2024-04-25T05:40:50.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.796071+0000", 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96439179100000005 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.796171+0000", 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9714000000000001e-05 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.360 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:50.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.796273+0000", 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010265299999999999 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801109+0000", 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0048357399999999998 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.803013+0000", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001904483 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.807769+0000", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047558440000000004 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.807849+0000", 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9870000000000006e-05 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.807894+0000", 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5210999999999998e-05 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:197932 7.9 7:9dd0eeb3:::benchmark_data_smithi043_74356_object81063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:33.867912+0000", 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 136.37702073299999, 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94013196300000001, 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.363 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 197932 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867912+0000", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867912+0000", 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867913+0000", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.469e-06 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867918+0000", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2719999999999999e-06 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867926+0000", 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2749999999999995e-06 2024-04-25T05:40:50.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867932+0000", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.489e-06 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.867995+0000", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2205999999999994e-05 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:33.868004+0000", 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3999999999999998e-06 2024-04-25T05:40:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798659+0000", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93065498099999999 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798780+0000", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120432 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.798969+0000", 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018968899999999999 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.803050+0000", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040810450000000002 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.805090+0000", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020396149999999998 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.807941+0000", 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028507950000000002 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.808010+0000", 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9555999999999996e-05 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.808044+0000", 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3738999999999997e-05 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198209 7.d 7:b85f0881:::benchmark_data_smithi043_74356_object81340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-04-25T05:40:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:34.801764+0000", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 135.44316877399999, 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97162393700000005, 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198209 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801764+0000", 2024-04-25T05:40:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801764+0000", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801765+0000", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2529999999999999e-06 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801771+0000", 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6829999999999996e-06 2024-04-25T05:40:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801784+0000", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2707e-05 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.801792+0000", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5390000000000007e-06 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.802124+0000", 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033229100000000002 2024-04-25T05:40:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.802134+0000", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4239999999999999e-06 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.768715+0000", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.96658125100000003 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.768835+0000", 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120338 2024-04-25T05:40:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.768972+0000", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000136947 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.770616+0000", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016439849999999999 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.770633+0000", 2024-04-25T05:40:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6379e-05 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773260+0000", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002627065 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773340+0000", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0446999999999997e-05 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773388+0000", 2024-04-25T05:40:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7627999999999999e-05 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198226 7.d 7:ba727b37:::benchmark_data_smithi043_74356_object81357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:34.816253+0000", 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 135.42867939000001, 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95925917999999999, 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.376 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198226 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816253+0000", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816253+0000", 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816255+0000", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.384e-06 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816260+0000", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3920000000000002e-06 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816272+0000", 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1508000000000001e-05 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816280+0000", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.49e-06 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816347+0000", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6470999999999994e-05 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.816356+0000", 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.842e-06 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.770677+0000", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95432083599999995 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.770804+0000", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012675499999999999 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.770922+0000", 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011780199999999999 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773460+0000", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002538355 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.773565+0000", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104887 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775412+0000", 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018467220000000001 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775468+0000", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6150000000000003e-05 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775513+0000", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4586000000000003e-05 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198264 7.d 7:b7aea299:::benchmark_data_smithi043_74356_object81395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:34.852806+0000", 2024-04-25T05:40:50.382 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 135.39212673399999, 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92286663300000005, 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198264 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852806+0000", 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852806+0000", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852807+0000", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6500000000000008e-07 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852811+0000", 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3850000000000004e-06 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852825+0000", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3713999999999999e-05 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852834+0000", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5550000000000003e-06 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852901+0000", 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7003999999999997e-05 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:34.852910+0000", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1649999999999995e-06 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.772139+0000", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91922874600000004 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.772208+0000", 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9568999999999997e-05 2024-04-25T05:40:50.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.772304+0000", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6135e-05 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775371+0000", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030663299999999999 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775553+0000", 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000182907 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775588+0000", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4980000000000001e-05 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775636+0000", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7318000000000003e-05 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.775673+0000", 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6860000000000003e-05 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:50.389 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:50.389 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:40:50.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:50.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:50.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:50.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:50.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:50.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:50.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:50.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:50.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:51.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:51.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:40:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: 4.11 scrub ok 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: pgmap v352: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 313 MiB/s wr, 5.01k op/s; 42941/353640 objects misplaced (12.143%); 20 MiB/s, 331 objects/s recovering 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: 8.7 scrub starts 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[27010]: 3.18 scrub starts 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: 4.11 scrub ok 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: pgmap v352: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 313 MiB/s wr, 5.01k op/s; 42941/353640 objects misplaced (12.143%); 20 MiB/s, 331 objects/s recovering 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:40:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-25T05:40:51.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: 8.7 scrub starts 2024-04-25T05:40:51.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:50 smithi043 ceph-mon[37492]: 3.18 scrub starts 2024-04-25T05:40:51.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:51.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: 4.11 scrub ok 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: pgmap v352: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 313 MiB/s wr, 5.01k op/s; 42941/353640 objects misplaced (12.143%); 20 MiB/s, 331 objects/s recovering 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 15 pgs peering) 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: 8.7 scrub starts 2024-04-25T05:40:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:50 smithi060 ceph-mon[30898]: 3.18 scrub starts 2024-04-25T05:40:51.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:51.306 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 120662 120646 89.7281 54.25 0.0178419 0.0111399 2024-04-25T05:40:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.321 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:51.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:51.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:51.454 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:51.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:51.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:51.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.722 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:51.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:51.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:40:51.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:51.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:51.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:51.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:52.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:52.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:40:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:40:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[27010]: 3.18 scrub ok 2024-04-25T05:40:52.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[37492]: 3.18 scrub ok 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:40:52.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:51 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:40:52.156 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:51 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:40:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:51 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:40:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:51 smithi060 ceph-mon[30898]: 3.18 scrub ok 2024-04-25T05:40:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:51 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:40:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:51 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:40:52.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:52.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:52.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 121555 121539 89.3261 55.8125 0.0528268 0.01119 2024-04-25T05:40:52.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:52.460 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:52.471 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:52.504 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:40:52.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:52.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:52.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:52.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:52.822 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:40:52.882 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:40:52.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:40:52.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:40:52.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: pgmap v354: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 857 B/s rd, 262 MiB/s wr, 4.19k op/s; 42941/353640 objects misplaced (12.143%); 17 MiB/s, 276 objects/s recovering 2024-04-25T05:40:52.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[37492]: 8.a scrub ok 2024-04-25T05:40:52.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:52.967 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:52.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:53.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:53.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: pgmap v354: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 857 B/s rd, 262 MiB/s wr, 4.19k op/s; 42941/353640 objects misplaced (12.143%); 17 MiB/s, 276 objects/s recovering 2024-04-25T05:40:53.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:52 smithi043 ceph-mon[27010]: 8.a scrub ok 2024-04-25T05:40:53.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:53.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:53.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:40:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:40:53.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:40:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:40:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:40:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:40:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: pgmap v354: 158 pgs: 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 857 B/s rd, 262 MiB/s wr, 4.19k op/s; 42941/353640 objects misplaced (12.143%); 17 MiB/s, 276 objects/s recovering 2024-04-25T05:40:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:52 smithi060 ceph-mon[30898]: 8.a scrub ok 2024-04-25T05:40:53.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 15 122468 122453 88.9517 57.125 0.0188468 0.0112372 2024-04-25T05:40:53.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:53.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:53.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:53.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:53.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:53.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:53.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:53.793 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:122907 8.1 e239/233 8:8633bd3d:::benchmark_data_smithi043_104001_object122906:head v 239'7860, mlcod=239'7854)", 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.816578+0000", 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.011209505999999999, 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011246018999999999, 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.829 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816578+0000", 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816578+0000", 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.830 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816579+0000", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1900000000000002e-07 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816607+0000", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8642000000000001e-05 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816623+0000", 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5515999999999998e-05 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.831 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.816631+0000", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6629999999999993e-06 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.817820+0000", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011889979999999999 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.817846+0000", 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6208e-05 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(8.6 239/233 [PushReplyOp(8:6193a4d4:::benchmark_data_smithi043_104001_object59661:head)])", 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.824065+0000", 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0037222319999999998, 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038447210000000002, 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824065+0000", 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824065+0000", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824066+0000", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9900000000000002e-07 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824069+0000", 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5370000000000002e-06 2024-04-25T05:40:53.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824076+0000", 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2529999999999998e-06 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824084+0000", 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6550000000000004e-06 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:122918 8.d e239/198 8:be5b9bb2:::benchmark_data_smithi043_104001_object122917:head v 239'7714, mlcod=239'7712)", 2024-04-25T05:40:53.835 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.824107+0000", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0036803080000000002, 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003863137, 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824107+0000", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824107+0000", 2024-04-25T05:40:53.836 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824108+0000", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6199999999999998e-07 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824140+0000", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2214e-05 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824149+0000", 2024-04-25T05:40:53.837 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7490000000000004e-06 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824155+0000", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4949999999999996e-06 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824455+0000", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030041399999999999 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824562+0000", 2024-04-25T05:40:53.838 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106097 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:122922 8.a e239/198 8:58f36d8e:::benchmark_data_smithi043_104001_object122921:head v 239'7757, mlcod=239'7755)", 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.824886+0000", 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0029010849999999999, 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031559090000000001, 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:40:53.839 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824886+0000", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824886+0000", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824887+0000", 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4700000000000001e-07 2024-04-25T05:40:53.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824914+0000", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6580999999999998e-05 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824923+0000", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2620000000000004e-06 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824930+0000", 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7139999999999996e-06 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.825048+0000", 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011879000000000001 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.825065+0000", 2024-04-25T05:40:53.842 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6939000000000002e-05 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:122925 8.1 e239/233 8:8969e249:::benchmark_data_smithi043_104001_object122924:head v 239'7861, mlcod=239'7858)", 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.827887+0000", 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9999003, 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000230447, 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827887+0000", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827887+0000", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827888+0000", 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7900000000000007e-07 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.844 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827914+0000", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5800999999999999e-05 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827924+0000", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0277e-05 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.827932+0000", 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5630000000000003e-06 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.845 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828016+0000", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3708000000000006e-05 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828041+0000", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5281000000000001e-05 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:122923 8.4 e239/198)", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.824169+0000", 2024-04-25T05:40:53.846 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0036181189999999999, 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040152030000000002, 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824169+0000", 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824169+0000", 2024-04-25T05:40:53.847 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.824170+0000", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0999999999999999e-07 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828051+0000", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038816749999999998 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828062+0000", 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0122999999999999e-05 2024-04-25T05:40:53.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828068+0000", 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7870000000000004e-06 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:122921 8.e 8.ec6c39fe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.828097+0000", 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.999691, 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000147107, 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 122921 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828097+0000", 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.850 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828097+0000", 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.851 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828098+0000", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4699999999999999e-06 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828123+0000", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4408000000000001e-05 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828132+0000", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1859999999999998e-06 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.852 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828139+0000", 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0040000000000002e-06 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:122930 8.6 8.6a166f16 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:53.828151+0000", 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 4294967295.9996367, 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016170099999999999, 2024-04-25T05:40:53.853 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 122930 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828151+0000", 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.854 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828151+0000", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828152+0000", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2300000000000001e-07 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828181+0000", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9904e-05 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:40:53.855 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828188+0000", 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0530000000000001e-06 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:53.828195+0000", 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1210000000000003e-06 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 8 2024-04-25T05:40:53.856 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:40:53.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:53.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:53.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:53.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:40:53.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:40:53.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:53.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:40:53.972 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:40:53.972 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:54.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:54.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:54.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:54.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:54.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:53 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:40:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:53 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:40:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:40:54.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:54.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:40:54.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 123408 123392 88.6039 58.6875 0.00925994 0.0112812 2024-04-25T05:40:54.340 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:54.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:53 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:40:53.988+0000 7f173ec68700 -1 mon.a@0(leader).osd e239 definitely_dead 0 2024-04-25T05:40:54.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:54.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:54.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:54.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:54.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:54.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:54.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:54.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:54.928 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.2. 2024-04-25T05:40:55.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:55.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:55.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: 8.7 scrub ok 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 265 MiB/s wr, 4.24k op/s; 42715/367167 objects misplaced (11.634%); 31 MiB/s, 499 objects/s recovering 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3854429870' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:54 smithi060 ceph-mon[30898]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-25T05:40:55.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:55.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:55.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 124001 123985 88.0184 37.0625 0.00424627 0.0113095 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: 8.7 scrub ok 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 265 MiB/s wr, 4.24k op/s; 42715/367167 objects misplaced (11.634%); 31 MiB/s, 499 objects/s recovering 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3854429870' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[37492]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: 8.7 scrub ok 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 265 MiB/s wr, 4.24k op/s; 42715/367167 objects misplaced (11.634%); 31 MiB/s, 499 objects/s recovering 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3854429870' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-04-25T05:40:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:54 smithi043 ceph-mon[27010]: Client client.admin marked osd.2 out, while it was still marked up 2024-04-25T05:40:55.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:55.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:55.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:55.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:55.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:55.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:55.902 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:40:55.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:55.927 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:55.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:56.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:56.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:56.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-25T05:40:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: osdmap e240: 8 total, 8 up, 7 in 2024-04-25T05:40:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/959528183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:56.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:56.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:55 smithi060 ceph-mon[30898]: osdmap e241: 8 total, 8 up, 7 in 2024-04-25T05:40:56.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:56.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 124002 123986 87.0305 0.0625 1.05462 0.0113179 2024-04-25T05:40:56.311 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:56.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-25T05:40:56.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: osdmap e240: 8 total, 8 up, 7 in 2024-04-25T05:40:56.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/959528183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[27010]: osdmap e241: 8 total, 8 up, 7 in 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: osdmap e240: 8 total, 8 up, 7 in 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/959528183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:40:56.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:55 smithi043 ceph-mon[37492]: osdmap e241: 8 total, 8 up, 7 in 2024-04-25T05:40:56.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:56.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.561 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:56.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:56.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:56.809 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:56.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:56.942 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:56.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: pgmap v357: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 735 B/s rd, 40 MiB/s wr, 648 op/s; 42715/367167 objects misplaced (11.634%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: 8.e continuing backfill to osd.0 from (213'2200,239'7649] MIN to 239'7649 2024-04-25T05:40:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:56 smithi060 ceph-mon[30898]: osdmap e242: 8 total, 8 up, 7 in 2024-04-25T05:40:57.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:57.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:57.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:57.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:57.309 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 90 2 124121 124119 86.1561 8.3125 0.170404 0.0115969 2024-04-25T05:40:57.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:57.336 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:40:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:40:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:40:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: pgmap v357: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 735 B/s rd, 40 MiB/s wr, 648 op/s; 42715/367167 objects misplaced (11.634%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: 8.e continuing backfill to osd.0 from (213'2200,239'7649] MIN to 239'7649 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[27010]: osdmap e242: 8 total, 8 up, 7 in 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: pgmap v357: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 151 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 735 B/s rd, 40 MiB/s wr, 648 op/s; 42715/367167 objects misplaced (11.634%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: 8.e continuing backfill to osd.0 from (213'2200,239'7649] MIN to 239'7649 2024-04-25T05:40:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:56 smithi043 ceph-mon[37492]: osdmap e242: 8 total, 8 up, 7 in 2024-04-25T05:40:57.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:57.468 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:57.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:57.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:57.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:40:57.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:40:57.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:57.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:57.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:40:57.996 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:57.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:40:58.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:58.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:58.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:58.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 239'7884 2024-04-25T05:40:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 239'7776 2024-04-25T05:40:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: 8.1 continuing backfill to osd.0 from (215'3052,239'7937] MIN to 239'7937 2024-04-25T05:40:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 239'7818 2024-04-25T05:40:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 239'7901 2024-04-25T05:40:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:57 smithi060 ceph-mon[30898]: osdmap e243: 8 total, 8 up, 7 in 2024-04-25T05:40:58.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.1069 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 124121 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 86.0929 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 36.0687 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 220.938 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 0 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1377 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 577.1 2024-04-25T05:40:58.310 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3535 2024-04-25T05:40:58.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 0 2024-04-25T05:40:58.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0116001 2024-04-25T05:40:58.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0410368 2024-04-25T05:40:58.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 2.14806 2024-04-25T05:40:58.311 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00119975 2024-04-25T05:40:58.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:40:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:58.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:57 smithi043 ceph-mon[27010]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 239'7884 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 239'7776 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: 8.1 continuing backfill to osd.0 from (215'3052,239'7937] MIN to 239'7937 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 239'7818 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 239'7901 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: osdmap e243: 8 total, 8 up, 7 in 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:57 smithi043 ceph-mon[37492]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 239'7884 2024-04-25T05:40:58.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 239'7776 2024-04-25T05:40:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: 8.1 continuing backfill to osd.0 from (215'3052,239'7937] MIN to 239'7937 2024-04-25T05:40:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 239'7818 2024-04-25T05:40:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 239'7901 2024-04-25T05:40:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: osdmap e243: 8 total, 8 up, 7 in 2024-04-25T05:40:58.429 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-04-25T05:40:58.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:58.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:58.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:40:58.659 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:40:58.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:58.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:58.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:58.922 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:58.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:59.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:40:59.050 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:40:59.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:59.187 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:58 smithi060 ceph-mon[30898]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 5 active+remapped+backfilling, 144 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 841 op/s; 56230/368769 objects misplaced (15.248%); 43 MiB/s, 685 objects/s recovering 2024-04-25T05:40:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:40:58 smithi060 ceph-mon[30898]: osdmap e244: 8 total, 8 up, 7 in 2024-04-25T05:40:59.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:40:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:40:59.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 5 active+remapped+backfilling, 144 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 841 op/s; 56230/368769 objects misplaced (15.248%); 43 MiB/s, 685 objects/s recovering 2024-04-25T05:40:59.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[27010]: osdmap e244: 8 total, 8 up, 7 in 2024-04-25T05:40:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 5 active+remapped+backfilling, 144 active+clean; 7.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 53 MiB/s wr, 841 op/s; 56230/368769 objects misplaced (15.248%); 43 MiB/s, 685 objects/s recovering 2024-04-25T05:40:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:40:58 smithi043 ceph-mon[37492]: osdmap e244: 8 total, 8 up, 7 in 2024-04-25T05:40:59.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:40:59.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:40:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:40:59.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:59.585 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:59.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:59.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:40:59.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:40:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:40:59.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:40:59.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:00.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:00.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:00.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:40:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:40:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:00.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:00.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:00.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:00.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:00.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:00.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:00.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:00.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:00.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:00.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:00.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:00.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:00.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:00.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:41:01.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:01.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:01.224 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:01.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:00 smithi060 ceph-mon[30898]: pgmap v363: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 491 op/s; 68976/373053 objects misplaced (18.490%); 91 MiB/s, 1.47k objects/s recovering 2024-04-25T05:41:01.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:00 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:41:01.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.373 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:01.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:00 smithi043 ceph-mon[27010]: pgmap v363: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 491 op/s; 68976/373053 objects misplaced (18.490%); 91 MiB/s, 1.47k objects/s recovering 2024-04-25T05:41:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:00 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:41:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:00 smithi043 ceph-mon[37492]: pgmap v363: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 491 op/s; 68976/373053 objects misplaced (18.490%); 91 MiB/s, 1.47k objects/s recovering 2024-04-25T05:41:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:00 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:41:01.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:01.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:01.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:01.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:01.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:01.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:01.944 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:01.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:02.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:02.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:02.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:02.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:02.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:02.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:02.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:02.487 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:02.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:02.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:02.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:02.761 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:02.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:02.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:02.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:02.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:03.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:03.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:03.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:03.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: pgmap v364: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 867 B/s rd, 21 MiB/s wr, 333 op/s; 68976/373053 objects misplaced (18.490%); 62 MiB/s, 998 objects/s recovering 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: osdmap e245: 8 total, 8 up, 7 in 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:03.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:03.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:03.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:03.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: pgmap v364: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 867 B/s rd, 21 MiB/s wr, 333 op/s; 68976/373053 objects misplaced (18.490%); 62 MiB/s, 998 objects/s recovering 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: osdmap e245: 8 total, 8 up, 7 in 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: pgmap v364: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+undersized+remapped, 6 active+remapped+backfilling, 143 active+clean; 7.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 867 B/s rd, 21 MiB/s wr, 333 op/s; 68976/373053 objects misplaced (18.490%); 62 MiB/s, 998 objects/s recovering 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: osdmap e245: 8 total, 8 up, 7 in 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:03.423 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:03.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:03.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:03.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:03.688 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:03.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:03.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:03.928 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:03.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:03.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:04.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:04.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:04.216 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:04.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:04.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:04.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:04.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:04.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:04.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:04.516 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:04.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:04.653 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:04.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:04.787 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:04.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:04.916 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:04.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:05.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:05.040 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:05.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:05.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: pgmap v366: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 15 MiB/s wr, 868 op/s; 60986/361731 objects misplaced (16.859%); 100 MiB/s, 1.60k objects/s recovering 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:41:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:05.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:05.317 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: pgmap v366: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 15 MiB/s wr, 868 op/s; 60986/361731 objects misplaced (16.859%); 100 MiB/s, 1.60k objects/s recovering 2024-04-25T05:41:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:41:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: pgmap v366: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 15 MiB/s wr, 868 op/s; 60986/361731 objects misplaced (16.859%); 100 MiB/s, 1.60k objects/s recovering 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:41:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:05.419 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:05.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:05.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:05.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:05.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:05.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:05.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:05.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:05.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:05.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:05.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:06.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:06.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:06.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:05 smithi060 ceph-mon[30898]: osdmap e246: 8 total, 8 up, 7 in 2024-04-25T05:41:06.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:06.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:05 smithi043 ceph-mon[27010]: osdmap e246: 8 total, 8 up, 7 in 2024-04-25T05:41:06.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:05 smithi043 ceph-mon[37492]: osdmap e246: 8 total, 8 up, 7 in 2024-04-25T05:41:06.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:06.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:06.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:06.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:06.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:06.797 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:06.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:06.955 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:06.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:07.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:07.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:07.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:07.246 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:07.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:06 smithi060 ceph-mon[30898]: pgmap v368: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 891 B/s rd, 148 B/s wr, 548 op/s; 60986/361731 objects misplaced (16.859%); 55 MiB/s, 874 objects/s recovering 2024-04-25T05:41:07.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:07.384 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:07.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:06 smithi043 ceph-mon[27010]: pgmap v368: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 891 B/s rd, 148 B/s wr, 548 op/s; 60986/361731 objects misplaced (16.859%); 55 MiB/s, 874 objects/s recovering 2024-04-25T05:41:07.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:06 smithi043 ceph-mon[37492]: pgmap v368: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 891 B/s rd, 148 B/s wr, 548 op/s; 60986/361731 objects misplaced (16.859%); 55 MiB/s, 874 objects/s recovering 2024-04-25T05:41:07.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:07.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:07.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:07.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:07.655 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:07.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:07.910 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:07.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:07.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:08.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:08.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:08.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:08.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:08.266 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:41:08.267 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:41:08.267 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:41:08.267 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:41:08.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:41:08.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:08.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:08.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:08.491 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:08.594 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:08.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:08.761 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:08.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:08.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:08.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:09.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:09.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.179 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:09.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:09.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:09.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[27010]: pgmap v369: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 505 op/s; 60129/360951 objects misplaced (16.658%); 53 MiB/s, 845 objects/s recovering 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[27010]: osdmap e247: 8 total, 8 up, 7 in 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[37492]: pgmap v369: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 505 op/s; 60129/360951 objects misplaced (16.658%); 53 MiB/s, 845 objects/s recovering 2024-04-25T05:41:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-mon[37492]: osdmap e247: 8 total, 8 up, 7 in 2024-04-25T05:41:09.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:09.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:09.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:09 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:09 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:09 smithi060 ceph-mon[30898]: pgmap v369: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 505 op/s; 60129/360951 objects misplaced (16.658%); 53 MiB/s, 845 objects/s recovering 2024-04-25T05:41:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:09 smithi060 ceph-mon[30898]: osdmap e247: 8 total, 8 up, 7 in 2024-04-25T05:41:09.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:09.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:41:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:09.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:09.727 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:09.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:09.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:09.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:09.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:10.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:10.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:10.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:10.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:41:10.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:10.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:10.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:10.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:10.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:10.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:10.711 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:10.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:10.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:10.873 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:10.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:10.877 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:10.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:10.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:10.980 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:11.006 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:11.024 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:11.067 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:11.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:11 smithi043 ceph-mon[37492]: pgmap v371: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1.06k op/s; 53777/348273 objects misplaced (15.441%); 89 MiB/s, 1.42k objects/s recovering 2024-04-25T05:41:11.067 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:11 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3809377814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:11.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:11 smithi043 ceph-mon[27010]: pgmap v371: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1.06k op/s; 53777/348273 objects misplaced (15.441%); 89 MiB/s, 1.42k objects/s recovering 2024-04-25T05:41:11.068 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:11 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3809377814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:11.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:11.111 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:11.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:11.156 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:11.204 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:11.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:11.247 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:11.250 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:11.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:11.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:11.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:11 smithi060 ceph-mon[30898]: pgmap v371: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1.06k op/s; 53777/348273 objects misplaced (15.441%); 89 MiB/s, 1.42k objects/s recovering 2024-04-25T05:41:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:11 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3809377814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:11.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:11.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:11.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:11.707 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:11.735 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:11.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:11.841 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:11.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:11.960 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 6, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:41:11.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:41:11.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:11.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:12.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:12.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:12.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:12.271 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:12.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.412 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:12.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:12.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:12.544 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:12.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:12.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:12.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:12.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:12.877 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:12.888 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:12.888 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:12.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:12.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:12.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:13.053 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:41:13.053 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:13 smithi043 ceph-mon[27010]: pgmap v372: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 53777/348273 objects misplaced (15.441%); 40 MiB/s, 634 objects/s recovering 2024-04-25T05:41:13.054 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:13.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:13 smithi043 ceph-mon[37492]: pgmap v372: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 53777/348273 objects misplaced (15.441%); 40 MiB/s, 634 objects/s recovering 2024-04-25T05:41:13.054 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:13.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:13.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:13.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:13.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:13.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:13.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:13 smithi060 ceph-mon[30898]: pgmap v372: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s; 53777/348273 objects misplaced (15.441%); 40 MiB/s, 634 objects/s recovering 2024-04-25T05:41:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:13.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:13.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:13.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:13.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:13.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:13.824 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:13.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:13.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:13.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:14.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:14.093 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:14.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:14.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:41:14.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[27010]: osdmap e248: 8 total, 8 up, 7 in 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[37492]: osdmap e248: 8 total, 8 up, 7 in 2024-04-25T05:41:14.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:14.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:14.400 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:14 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:14.400 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3021661474' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:14.400 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:14 smithi060 ceph-mon[30898]: osdmap e248: 8 total, 8 up, 7 in 2024-04-25T05:41:14.400 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:14.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:14.539 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:14.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:14.675 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:14.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:14.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:14.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:14.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:15.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:15.083 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:15.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:15.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:15.227 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:15 smithi060 ceph-mon[30898]: pgmap v374: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 47109/333270 objects misplaced (14.135%); 74 MiB/s, 1.19k objects/s recovering 2024-04-25T05:41:15.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:15.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:15.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:15.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:15 smithi043 ceph-mon[27010]: pgmap v374: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 47109/333270 objects misplaced (14.135%); 74 MiB/s, 1.19k objects/s recovering 2024-04-25T05:41:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:15 smithi043 ceph-mon[37492]: pgmap v374: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 47109/333270 objects misplaced (14.135%); 74 MiB/s, 1.19k objects/s recovering 2024-04-25T05:41:15.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:15.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:15.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:15.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:15.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:41:15.733 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:15.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:15.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:15.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:15.945 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:15.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:16.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:41:16.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:16.082 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:41:16.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:16.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:16.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:16.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[27010]: osdmap e249: 8 total, 8 up, 7 in 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/260157419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[37492]: osdmap e249: 8 total, 8 up, 7 in 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/260157419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:16 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.423 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:16 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:16 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:16 smithi060 ceph-mon[30898]: osdmap e249: 8 total, 8 up, 7 in 2024-04-25T05:41:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/260157419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:16 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:41:16.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:16.561 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:16.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:16.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:16.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:16.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:16.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:16.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:16.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:17.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:41:17.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:17.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:17.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:17.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:17.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[27010]: pgmap v376: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.01k op/s; 47109/333270 objects misplaced (14.135%); 46 MiB/s, 731 objects/s recovering 2024-04-25T05:41:17.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:41:17.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[27010]: osdmap e250: 8 total, 8 up, 7 in 2024-04-25T05:41:17.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[37492]: pgmap v376: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.01k op/s; 47109/333270 objects misplaced (14.135%); 46 MiB/s, 731 objects/s recovering 2024-04-25T05:41:17.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:41:17.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:17 smithi043 ceph-mon[37492]: osdmap e250: 8 total, 8 up, 7 in 2024-04-25T05:41:17.360 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:17.367 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:17.376 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:17.397 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:17.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:17.455 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:17.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:17.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:17 smithi060 ceph-mon[30898]: pgmap v376: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 1.01k op/s; 47109/333270 objects misplaced (14.135%); 46 MiB/s, 731 objects/s recovering 2024-04-25T05:41:17.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:17 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:41:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:17 smithi060 ceph-mon[30898]: osdmap e250: 8 total, 8 up, 7 in 2024-04-25T05:41:17.504 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:17.549 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:17.600 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:17.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:17.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:17.652 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:17.697 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:17.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:17.783 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:17.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:17.917 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:17.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:18.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:18.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:18.259 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1080707259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:18.260 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:18.260 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1080707259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:18.260 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:18.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:18.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:18.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1080707259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:18.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:18.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:18.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:18.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:18.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:18.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:19.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:19.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:19.119 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:41:19.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:19.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:19.190 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[27010]: pgmap v378: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 883 op/s; 46328/332394 objects misplaced (13.938%); 54 MiB/s, 861 objects/s recovering 2024-04-25T05:41:19.190 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1511433381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.190 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.191 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[37492]: pgmap v378: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 883 op/s; 46328/332394 objects misplaced (13.938%); 54 MiB/s, 861 objects/s recovering 2024-04-25T05:41:19.191 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1511433381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.191 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:19.306 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.25115.0:143463 8.f e251/241)", 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:19.304732+0000", 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00034808700000000002, 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00037314800000000001, 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304732+0000", 2024-04-25T05:41:19.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304732+0000", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304733+0000", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3099999999999997e-07 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304737+0000", 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3359999999999997e-06 2024-04-25T05:41:19.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304747+0000", 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3379999999999994e-06 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304754+0000", 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1149999999999996e-06 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.309 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:143463 8.f 8:f34d6da3:::benchmark_data_smithi043_104001_object19339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:19.295105+0000", 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0099750510000000004, 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010062994, 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 143463 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.310 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295105+0000", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295105+0000", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295107+0000", 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.155e-06 2024-04-25T05:41:19.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295112+0000", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2750000000000003e-06 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295121+0000", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8289999999999997e-06 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295129+0000", 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1629999999999996e-06 2024-04-25T05:41:19.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295576+0000", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00044732799999999998 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295693+0000", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001165 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2,4", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.295816+0000", 2024-04-25T05:41:19.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012369099999999999 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.301586+0000", 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0057698189999999998 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.301603+0000", 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7266999999999999e-05 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:19.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303435+0000", 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018317170000000001 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:143465 8.c 8:30a9eb88:::benchmark_data_smithi043_104001_object19341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:19.299915+0000", 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0051658169999999996, 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053651619999999997, 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:41:19.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 143465 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299915+0000", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299915+0000", 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299916+0000", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.421e-06 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299921+0000", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0499999999999999e-06 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299933+0000", 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.222e-05 2024-04-25T05:41:19.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.299942+0000", 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6910000000000003e-06 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300100+0000", 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015825400000000001 2024-04-25T05:41:19.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300222+0000", 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012148500000000001 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,5,7", 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300349+0000", 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000126797 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304474+0000", 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041250530000000001 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.304514+0000", 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0927999999999998e-05 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.320 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:143466 8.2 e251/233 8:46a02348:::benchmark_data_smithi043_104001_object19342:head v 251'8901, mlcod=251'8898)", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:19.300951+0000", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0041291840000000002, 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0044561770000000004, 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300951+0000", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:41:19.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300951+0000", 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300952+0000", 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8100000000000003e-07 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300957+0000", 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3070000000000002e-06 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300963+0000", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8220000000000003e-06 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.300969+0000", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2040000000000002e-06 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.301059+0000", 2024-04-25T05:41:19.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9504000000000002e-05 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.301081+0000", 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2444999999999999e-05 2024-04-25T05:41:19.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:143468 8.0 8:04a86c3e:::benchmark_data_smithi043_104001_object19344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:19.303261+0000", 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0018191310000000001, 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022226720000000002, 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:19.325 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 143468 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303261+0000", 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303261+0000", 2024-04-25T05:41:19.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303263+0000", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4449999999999999e-06 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303268+0000", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4559999999999999e-06 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303280+0000", 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1347e-05 2024-04-25T05:41:19.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303287+0000", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7929999999999998e-06 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303480+0000", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019308699999999999 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303596+0000", 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116037 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:41:19.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:19.303715+0000", 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118739 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 5 2024-04-25T05:41:19.329 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:41:19.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:19.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:19.469 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:19 smithi060 ceph-mon[30898]: pgmap v378: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 883 op/s; 46328/332394 objects misplaced (13.938%); 54 MiB/s, 861 objects/s recovering 2024-04-25T05:41:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1511433381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:41:19.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:19.605 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:19.647 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:19.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:19.744 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:19.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:19.883 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:19.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:19.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:20.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:20.121 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:20.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:20.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:20.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:41:20.122 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[37492]: osdmap e251: 8 total, 8 up, 7 in 2024-04-25T05:41:20.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:20.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:20.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:20.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:41:20.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:41:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:20 smithi043 ceph-mon[27010]: osdmap e251: 8 total, 8 up, 7 in 2024-04-25T05:41:20.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:20.443 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:20 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:41:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:20 smithi060 ceph-mon[30898]: osdmap e251: 8 total, 8 up, 7 in 2024-04-25T05:41:20.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.598 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:20.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:20.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:20.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:20.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:20.865 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:20.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:20.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:20.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:41:21.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:21.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:21.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:21.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:21.299 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:21.362 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[27010]: pgmap v380: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[37492]: pgmap v380: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:41:21.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:21.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:21 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:21.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.434 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:21.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:21.483 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:21 smithi060 ceph-mon[30898]: pgmap v380: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:21 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:41:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:21 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:41:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:21 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:21 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:21.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:21.676 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:21.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:21.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:21.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:21.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:22.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:22.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:22.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:22.158 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:41:22.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:22.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[27010]: osdmap e252: 8 total, 8 up, 7 in 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4220182105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[37492]: osdmap e252: 8 total, 8 up, 7 in 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4220182105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:22 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:22.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:22 smithi060 ceph-mon[30898]: osdmap e252: 8 total, 8 up, 7 in 2024-04-25T05:41:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4220182105' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:22 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:41:22.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:22.541 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:22.585 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:41:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:41:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:41:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:41:22.586 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:41:22.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:22.676 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:22.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:22.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:41:22.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:22.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:22.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:22.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:23.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:23.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:23.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:23.374 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:23.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:41:23.420 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[27010]: pgmap v382: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:23.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:41:23.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[27010]: osdmap e253: 8 total, 8 up, 7 in 2024-04-25T05:41:23.421 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[37492]: pgmap v382: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:23.421 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:41:23.421 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:23 smithi043 ceph-mon[37492]: osdmap e253: 8 total, 8 up, 7 in 2024-04-25T05:41:23.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:23 smithi060 ceph-mon[30898]: pgmap v382: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 807 op/s; 40953/318750 objects misplaced (12.848%); 44 MiB/s, 705 objects/s recovering 2024-04-25T05:41:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:23 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:41:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:23 smithi060 ceph-mon[30898]: osdmap e253: 8 total, 8 up, 7 in 2024-04-25T05:41:23.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:23.521 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:23.563 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:23.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:23.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:23.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:23.818 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:23.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:23.965 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:23.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:24.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:24.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:24.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:41:24.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:24.241 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:24.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:24.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:24.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.524 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:24.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:24.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:24.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:24.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:24.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:24.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:24.933 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:24.963 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:24.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:25.022 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:25.033 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:25.039 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:25.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:25.090 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.093 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:25.122 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:25.150 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:25.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:25.225 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:25.267 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:25.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[27010]: pgmap v384: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s; 34719/303906 objects misplaced (11.424%); 83 MiB/s, 1.33k objects/s recovering 2024-04-25T05:41:25.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[27010]: osdmap e254: 8 total, 8 up, 7 in 2024-04-25T05:41:25.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3080951173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:25.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[37492]: pgmap v384: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s; 34719/303906 objects misplaced (11.424%); 83 MiB/s, 1.33k objects/s recovering 2024-04-25T05:41:25.300 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[37492]: osdmap e254: 8 total, 8 up, 7 in 2024-04-25T05:41:25.300 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3080951173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:25.312 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:25.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:25.361 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:25.385 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:25.391 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:25.391 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:25.391 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:25.392 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:25.392 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:25.392 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:25.392 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:25.392 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:25.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:25.499 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:25 smithi060 ceph-mon[30898]: pgmap v384: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s; 34719/303906 objects misplaced (11.424%); 83 MiB/s, 1.33k objects/s recovering 2024-04-25T05:41:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:25 smithi060 ceph-mon[30898]: osdmap e254: 8 total, 8 up, 7 in 2024-04-25T05:41:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3080951173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:25.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.637 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:25.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:25.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:25.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:25.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:25.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:26.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:26.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:26.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:26.198 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:26.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:26.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:26.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:26.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:26 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:41:26.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:26 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:41:26.465 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:26 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:41:26.465 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:26 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:41:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:26.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:26.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:26 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:41:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:26 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:41:26.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:26.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:26.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:26.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:26.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:26.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:26.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:26.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:27.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:27.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:27.148 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:27.218 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:41:27.218 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:41:27.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:27.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[27010]: pgmap v386: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 34719/303906 objects misplaced (11.424%); 47 MiB/s, 747 objects/s recovering 2024-04-25T05:41:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:41:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[37492]: pgmap v386: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 34719/303906 objects misplaced (11.424%); 47 MiB/s, 747 objects/s recovering 2024-04-25T05:41:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:41:27.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:27.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:27.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:27 smithi060 ceph-mon[30898]: pgmap v386: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 34719/303906 objects misplaced (11.424%); 47 MiB/s, 747 objects/s recovering 2024-04-25T05:41:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:41:27.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:27.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:27.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:27.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:27.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:27.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:27.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:27.949 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:27.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:28.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:28.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:28.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:28.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:28.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:28.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:28.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:28.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: osdmap e255: 8 total, 8 up, 7 in 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:41:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:28 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:41:28.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:41:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:41:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:41:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: osdmap e255: 8 total, 8 up, 7 in 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:41:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2106281188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:41:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/708165150' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:41:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: osdmap e255: 8 total, 8 up, 7 in 2024-04-25T05:41:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:41:28.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:28 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:41:28.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:28.723 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:28.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:28.857 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:28.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:28.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:41:28.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:29.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:29.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:29.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[37492]: pgmap v388: 158 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 864 op/s; 34171/303204 objects misplaced (11.270%); 52 MiB/s, 837 objects/s recovering 2024-04-25T05:41:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[27010]: pgmap v388: 158 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 864 op/s; 34171/303204 objects misplaced (11.270%); 52 MiB/s, 837 objects/s recovering 2024-04-25T05:41:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[27010]: osdmap e256: 8 total, 8 up, 7 in 2024-04-25T05:41:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[27010]: 3.c deep-scrub starts 2024-04-25T05:41:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[27010]: 3.c deep-scrub ok 2024-04-25T05:41:29.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:29.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:29.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:29.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:29 smithi060 ceph-mon[30898]: pgmap v388: 158 pgs: 1 peering, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 864 op/s; 34171/303204 objects misplaced (11.270%); 52 MiB/s, 837 objects/s recovering 2024-04-25T05:41:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:29 smithi060 ceph-mon[30898]: osdmap e256: 8 total, 8 up, 7 in 2024-04-25T05:41:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:29 smithi060 ceph-mon[30898]: 3.c deep-scrub starts 2024-04-25T05:41:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:29 smithi060 ceph-mon[30898]: 3.c deep-scrub ok 2024-04-25T05:41:29.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:29.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[37492]: osdmap e256: 8 total, 8 up, 7 in 2024-04-25T05:41:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[37492]: 3.c deep-scrub starts 2024-04-25T05:41:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-mon[37492]: 3.c deep-scrub ok 2024-04-25T05:41:29.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:29.713 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:29.713 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:29.715 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:29.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:29.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:29.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:29.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:30.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:30.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:30.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:30.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:30.364 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:30.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:30.497 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:30.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:30.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:30.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:30.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:30.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:30.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:30.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:31.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:31.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[27010]: pgmap v390: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 618 op/s; 28385/292785 objects misplaced (9.695%); 54 MiB/s, 863 objects/s recovering 2024-04-25T05:41:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[27010]: osdmap e257: 8 total, 8 up, 7 in 2024-04-25T05:41:31.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[27010]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:31.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[37492]: pgmap v390: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 618 op/s; 28385/292785 objects misplaced (9.695%); 54 MiB/s, 863 objects/s recovering 2024-04-25T05:41:31.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[37492]: osdmap e257: 8 total, 8 up, 7 in 2024-04-25T05:41:31.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:30 smithi043 ceph-mon[37492]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:31.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:31.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:30 smithi060 ceph-mon[30898]: pgmap v390: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 618 op/s; 28385/292785 objects misplaced (9.695%); 54 MiB/s, 863 objects/s recovering 2024-04-25T05:41:31.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:30 smithi060 ceph-mon[30898]: osdmap e257: 8 total, 8 up, 7 in 2024-04-25T05:41:31.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:30 smithi060 ceph-mon[30898]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:31.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:31.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:31.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:31.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:31.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:31.553 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:31.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:31.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:31.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:31.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:31.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:31.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:32.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:32.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[27010]: 4.12 deep-scrub starts 2024-04-25T05:41:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[27010]: 4.12 deep-scrub ok 2024-04-25T05:41:32.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[27010]: osdmap e258: 8 total, 8 up, 7 in 2024-04-25T05:41:32.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[37492]: 4.12 deep-scrub starts 2024-04-25T05:41:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[37492]: 4.12 deep-scrub ok 2024-04-25T05:41:32.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:31 smithi043 ceph-mon[37492]: osdmap e258: 8 total, 8 up, 7 in 2024-04-25T05:41:32.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:31 smithi060 ceph-mon[30898]: 4.12 deep-scrub starts 2024-04-25T05:41:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:31 smithi060 ceph-mon[30898]: 4.12 deep-scrub ok 2024-04-25T05:41:32.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:31 smithi060 ceph-mon[30898]: osdmap e258: 8 total, 8 up, 7 in 2024-04-25T05:41:32.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:32.324 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:32.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:32.452 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:32.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:32.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:32.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:32.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:32.846 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:32.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:32.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:32.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:33.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: pgmap v393: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 806 op/s; 28385/292785 objects misplaced (9.695%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:33.091 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: osdmap e259: 8 total, 8 up, 7 in 2024-04-25T05:41:33.092 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:33.092 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:41:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:41:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:41:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:41:33.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: pgmap v393: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 806 op/s; 28385/292785 objects misplaced (9.695%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: osdmap e259: 8 total, 8 up, 7 in 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: pgmap v393: 158 pgs: 3 peering, 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 806 op/s; 28385/292785 objects misplaced (9.695%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: osdmap e259: 8 total, 8 up, 7 in 2024-04-25T05:41:33.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:33.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:33.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:33.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:33.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:33.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:33.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:33.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:33.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:33.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:33.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:33.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:33.762 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:33.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:33.894 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:33.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:33.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:34.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:34.099 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:34.099 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:34.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:34.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:34.339 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:34.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:34.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:34.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:34.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:34.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:34.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.868 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:34.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:34.995 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:34.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:35.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:35.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:34 smithi043 ceph-mon[27010]: pgmap v395: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 913 B/s rd, 0 B/s wr, 793 op/s; 24213/282012 objects misplaced (8.586%); 44 MiB/s, 711 objects/s recovering 2024-04-25T05:41:35.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:34 smithi043 ceph-mon[37492]: pgmap v395: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 913 B/s rd, 0 B/s wr, 793 op/s; 24213/282012 objects misplaced (8.586%); 44 MiB/s, 711 objects/s recovering 2024-04-25T05:41:35.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:35.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:34 smithi060 ceph-mon[30898]: pgmap v395: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 913 B/s rd, 0 B/s wr, 793 op/s; 24213/282012 objects misplaced (8.586%); 44 MiB/s, 711 objects/s recovering 2024-04-25T05:41:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:35.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:35.417 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:35.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:35.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:35.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:35.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:35.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:35.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:36.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:36.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:36.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:36.195 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:36.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:36.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:36.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:36.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:36.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:36.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:36.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:36.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:36.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:36.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:37.111 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: 8.5 scrub starts 2024-04-25T05:41:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:41:37.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: pgmap v396: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 599 op/s; 24213/282012 objects misplaced (8.586%); 33 MiB/s, 521 objects/s recovering 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: 8.5 scrub starts 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: pgmap v396: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 599 op/s; 24213/282012 objects misplaced (8.586%); 33 MiB/s, 521 objects/s recovering 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:41:37.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:36 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:41:37.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:37.245 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: 8.5 scrub starts 2024-04-25T05:41:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:41:37.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:41:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: pgmap v396: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 599 op/s; 24213/282012 objects misplaced (8.586%); 33 MiB/s, 521 objects/s recovering 2024-04-25T05:41:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:41:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:36 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:41:37.273 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:37.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:41:37.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:37.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:37.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:37.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.703 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:37.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:37.734 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:37.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:37.841 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:37.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:37.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:38.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:38.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:38.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[27010]: 8.5 scrub ok 2024-04-25T05:41:38.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:41:38.131 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:41:38.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[37492]: 8.5 scrub ok 2024-04-25T05:41:38.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:41:38.132 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:37 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:41:38.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:38.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:37 smithi060 ceph-mon[30898]: 8.5 scrub ok 2024-04-25T05:41:38.217 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:37 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:41:38.218 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:37 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:41:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:38.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:38.450 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:38.450 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:38.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:38.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:38.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:38.656 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:38.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:38.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:38.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:38.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:38.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:39.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:39.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:39.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:38 smithi043 ceph-mon[27010]: pgmap v397: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 483 op/s; 24213/281244 objects misplaced (8.609%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:41:39.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:38 smithi043 ceph-mon[37492]: pgmap v397: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 483 op/s; 24213/281244 objects misplaced (8.609%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:41:39.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:39.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:39.235 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:38 smithi060 ceph-mon[30898]: pgmap v397: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 483 op/s; 24213/281244 objects misplaced (8.609%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:41:39.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:39.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:41:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:39.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:39.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:39.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.595 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:39.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:39.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:39.726 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:39.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:39.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:39.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:39.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:39.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:40.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:40.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:40.134 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:39 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:41:40.134 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:39 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:41:40.134 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:39 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:41:40.134 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:39 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:41:40.134 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:39 smithi060 ceph-mon[30898]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:41:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:41:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:41:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:41:40.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[27010]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:41:40.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:39 smithi043 ceph-mon[37492]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:41:40.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:40.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:41:40.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.300 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:40.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:40.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:40.425 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:40.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:40.560 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:40.644 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:40.644 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:40.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:40.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:40.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:40.815 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:40.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:40.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:40.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:40 smithi060 ceph-mon[30898]: pgmap v398: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 20446/268860 objects misplaced (7.605%); 42 MiB/s, 674 objects/s recovering 2024-04-25T05:41:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:40 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:41:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:40 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:41:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:40 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:41:40.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:40 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:41:41.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:41.085 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[27010]: pgmap v398: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 20446/268860 objects misplaced (7.605%); 42 MiB/s, 674 objects/s recovering 2024-04-25T05:41:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:41:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[37492]: pgmap v398: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s; 20446/268860 objects misplaced (7.605%); 42 MiB/s, 674 objects/s recovering 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:41:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:40 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:41:41.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:41.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:41.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:41.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:41.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:41:41.501 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:41.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:41.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:41.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:41.841 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:41.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:41.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:41.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:42.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:42.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:42.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:41 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:41:42.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:41 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:41:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:41 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:41:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:41 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:41:42.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:41 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:41:42.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:41 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:41:42.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:42.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:42.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:42.401 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:42.401 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:42.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:42.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:42.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:42.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:42.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:42.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:42.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:42.955 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:42.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:43.057 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:43.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:41:43.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:43.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:43.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:42 smithi060 ceph-mon[30898]: pgmap v399: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s; 20446/268860 objects misplaced (7.605%); 36 MiB/s, 579 objects/s recovering 2024-04-25T05:41:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:42 smithi060 ceph-mon[30898]: osdmap e260: 8 total, 8 up, 7 in 2024-04-25T05:41:43.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:43.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:43.370 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:43.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:42 smithi043 ceph-mon[27010]: pgmap v399: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s; 20446/268860 objects misplaced (7.605%); 36 MiB/s, 579 objects/s recovering 2024-04-25T05:41:43.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:42 smithi043 ceph-mon[27010]: osdmap e260: 8 total, 8 up, 7 in 2024-04-25T05:41:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:42 smithi043 ceph-mon[37492]: pgmap v399: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s; 20446/268860 objects misplaced (7.605%); 36 MiB/s, 579 objects/s recovering 2024-04-25T05:41:43.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:42 smithi043 ceph-mon[37492]: osdmap e260: 8 total, 8 up, 7 in 2024-04-25T05:41:43.447 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:43.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:43.518 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:43.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.642 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:43.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:43.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:43.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:43.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:41:43.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:43.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:43.917 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:44.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:44.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:44.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:43 smithi043 ceph-mon[27010]: osdmap e261: 8 total, 8 up, 7 in 2024-04-25T05:41:44.079 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:44.079 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:43 smithi043 ceph-mon[37492]: osdmap e261: 8 total, 8 up, 7 in 2024-04-25T05:41:44.079 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:44.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:44.241 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:43 smithi060 ceph-mon[30898]: osdmap e261: 8 total, 8 up, 7 in 2024-04-25T05:41:44.242 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:44.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:44.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:44.456 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:44.456 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:44.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:44.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:44.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:44.653 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:44.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:44.788 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:44.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:44.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:44.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:45.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:45.052 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:45.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:45.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:45.224 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:44 smithi060 ceph-mon[30898]: pgmap v402: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:45.224 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:44 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:41:45.224 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:44 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[27010]: pgmap v402: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[37492]: pgmap v402: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:41:45.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:44 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:41:45.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:41:45.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:45.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:45.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.473 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:45.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:45.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:45.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:45.681 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:45.696 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:45.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:45.764 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:45.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:45.817 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:45.865 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:45.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:45.876 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:45.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:45.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:45.910 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:45.954 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:45.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:46.003 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:46.056 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:46.059 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:46.060 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:46.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:41:46.081 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:41:46.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2008437436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:46.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:41:46.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:41:46.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2008437436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:46.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:46.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:46.178 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:45 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:41:46.178 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:45 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:41:46.178 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2008437436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:46.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:46.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:46.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.464 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:46.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:46.521 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:41:46.521 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:41:46.521 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:41:46.521 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:41:46.522 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:41:46.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:46.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:46.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.745 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:46.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:46.846 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:46.880 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:46.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:46.964 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:41:46.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:47.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:47.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:47.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:47.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:46 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:41:47.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:46 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:41:47.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:46 smithi060 ceph-mon[30898]: pgmap v403: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:47.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/608719203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.212 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:41:47.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:47.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:47.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[27010]: pgmap v403: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/608719203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[37492]: pgmap v403: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s; 15686/257511 objects misplaced (6.091%); 52 MiB/s, 827 objects/s recovering 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/608719203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:41:47.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:47.415 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:47.475 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:47.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:47.537 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:47.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:47.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:47.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:47.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:47.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:47.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:47.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:48.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:48.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:48.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:47 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:41:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:47 smithi060 ceph-mon[30898]: osdmap e262: 8 total, 8 up, 7 in 2024-04-25T05:41:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:47 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:48.272 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:175169 7.0 7:0d4e44d2:::benchmark_data_smithi043_74356_object58300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 224.79170109099999, 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13254632699999999, 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175169 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479531+0000", 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479532+0000", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.288e-06 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479537+0000", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9239999999999998e-06 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479548+0000", 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185e-05 2024-04-25T05:41:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479556+0000", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3139999999999998e-06 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479603+0000", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7651000000000002e-05 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479689+0000", 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5328999999999996e-05 2024-04-25T05:41:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.479769+0000", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0508999999999998e-05 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.481886+0000", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021172999999999999 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.492164+0000", 2024-04-25T05:41:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010277632 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612007+0000", 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.119842849 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612041+0000", 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3958999999999998e-05 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.612077+0000", 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6387000000000002e-05 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123943 8.4 8:2708029d:::benchmark_data_smithi043_104001_object123942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.379266694000002, 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.128519394, 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:41:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123943 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:41:48.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891967+0000", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5379999999999999e-06 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892016+0000", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9607000000000001e-05 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892052+0000", 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5642000000000002e-05 2024-04-25T05:41:48.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892061+0000", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4720000000000001e-06 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892114+0000", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3022999999999997e-05 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892283+0000", 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000168732 2024-04-25T05:41:48.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892847+0000", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056390200000000002 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897475+0000", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046274669999999997 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.901536+0000", 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040612139999999996 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917500+0000", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015963893999999999 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917511+0000", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0796e-05 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901955+0000", 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98444451099999997 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901968+0000", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2508e-05 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020400+0000", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.118432597 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020485+0000", 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4491000000000001e-05 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123953 8.4 8:27465e96:::benchmark_data_smithi043_104001_object123952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.374529398999996, 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1241087030000001, 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.287 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123953 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896704+0000", 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3349999999999999e-06 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896735+0000", 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1065000000000002e-05 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896745+0000", 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0189e-05 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896752+0000", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7800000000000003e-06 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897517+0000", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076539399999999997 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897672+0000", 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015488200000000001 2024-04-25T05:41:48.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897862+0000", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018961499999999999 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.902561+0000", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046995270000000002 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917555+0000", 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014993536999999999 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917566+0000", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1559999999999999e-05 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.902011+0000", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98444420700000002 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.902021+0000", 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.049e-05 2024-04-25T05:41:48.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020751+0000", 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1187298779999999 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020811+0000", 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0244000000000001e-05 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123959 8.d 8:b0698cdf:::benchmark_data_smithi043_104001_object123958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.343064794999997, 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.138867517, 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123959 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.294 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928169+0000", 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.406e-06 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928202+0000", 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3572000000000003e-05 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928214+0000", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2097e-05 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928220+0000", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8819999999999998e-06 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928280+0000", 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0406999999999999e-05 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928293+0000", 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.27e-05 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901652+0000", 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.973358995 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901666+0000", 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.401e-05 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048106+0000", 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1464394680000001 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048253+0000", 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147081 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048739+0000", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000486613 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057200+0000", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0084604820000000001 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066420+0000", 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0092204339999999996 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066598+0000", 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017789300000000001 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066859+0000", 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026048400000000001 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066917+0000", 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8510000000000001e-05 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067035+0000", 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117483 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123966 8.d 8:b64b38a9:::benchmark_data_smithi043_104001_object123965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-04-25T05:41:48.303 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.321704639000004, 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.136094049, 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123966 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949529+0000", 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5680000000000001e-06 2024-04-25T05:41:48.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949599+0000", 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0146000000000002e-05 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949616+0000", 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7413000000000001e-05 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949627+0000", 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0267e-05 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949695+0000", 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8750000000000004e-05 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949706+0000", 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0223e-05 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901696+0000", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95199062999999995 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901704+0000", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2169999999999994e-06 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057238+0000", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.155533602 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057382+0000", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014436399999999999 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.058010+0000", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00062746700000000004 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066646+0000", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086361749999999994 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067082+0000", 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043566699999999999 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067168+0000", 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6076999999999996e-05 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085439+0000", 2024-04-25T05:41:48.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.018271005 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085502+0000", 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2967999999999997e-05 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085621+0000", 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011951 2024-04-25T05:41:48.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124002 8.d 8:b79b1d51:::benchmark_data_smithi043_104001_object124001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 52.270052681000003, 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0846060230000001, 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.312 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124002 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001180+0000", 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6899999999999996e-07 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001301+0000", 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120989 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001312+0000", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.063e-05 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001317+0000", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6219999999999997e-06 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001365+0000", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7193e-05 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001374+0000", 2024-04-25T05:41:48.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0880000000000001e-06 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.061525+0000", 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0601515500000001 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.061653+0000", 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012791500000000001 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.062074+0000", 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00042062799999999998 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067121+0000", 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050470749999999998 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.073295+0000", 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0061738749999999997 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.080544+0000", 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072492839999999999 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085633+0000", 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050887900000000002 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085681+0000", 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7899000000000003e-05 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085785+0000", 2024-04-25T05:41:48.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000104516 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.320 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124101 8.8 8:1d082edc:::benchmark_data_smithi043_104001_object124100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.165507750000003, 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.272319637, 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.321 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124101 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.322 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105726+0000", 2024-04-25T05:41:48.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6500000000000001e-06 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105891+0000", 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000164845 2024-04-25T05:41:48.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105903+0000", 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2672e-05 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105911+0000", 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9670000000000001e-06 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105963+0000", 2024-04-25T05:41:48.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1233999999999998e-05 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.106112+0000", 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014978600000000001 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:41:48.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.106278+0000", 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016582200000000001 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.109610+0000", 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033317619999999998 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.109648+0000", 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8569000000000003e-05 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.377924+0000", 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26827566400000002 2024-04-25T05:41:48.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.377996+0000", 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2193000000000005e-05 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.378044+0000", 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7472999999999998e-05 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124117 8.8 8:17fb47bf:::benchmark_data_smithi043_104001_object124116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.229018+0000", 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.042213631999999, 2024-04-25T05:41:48.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14917126, 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124117 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229018+0000", 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229018+0000", 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229020+0000", 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6300000000000001e-06 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229097+0000", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7429999999999996e-05 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229112+0000", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4452e-05 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229126+0000", 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4371e-05 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229186+0000", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9385999999999997e-05 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229337+0000", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015112 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.229505+0000", 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000168708 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.232811+0000", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033055580000000001 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.232986+0000", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017539199999999999 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.378089+0000", 2024-04-25T05:41:48.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14510255799999999 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.378148+0000", 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9038e-05 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.378190+0000", 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1616999999999998e-05 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124834 8.8 8:1c6b1bac:::benchmark_data_smithi043_104001_object710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:59.232496+0000", 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.038735629000001, 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14493983299999999, 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124834 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232496+0000", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232496+0000", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232498+0000", 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4670000000000001e-06 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232505+0000", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4850000000000003e-06 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232514+0000", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8640000000000002e-06 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232521+0000", 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.345e-06 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232576+0000", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4095999999999998e-05 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232689+0000", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000113738 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.232803+0000", 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000113719 2024-04-25T05:41:48.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.238353+0000", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0055504109999999999 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.247210+0000", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0088566470000000005 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.377327+0000", 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13011677999999999 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.377385+0000", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8529999999999997e-05 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.377436+0000", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0751000000000002e-05 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124835 8.8 8:135b36b4:::benchmark_data_smithi043_104001_object711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:59.235014+0000", 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.036218179999999, 2024-04-25T05:41:48.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15363673, 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124835 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235014+0000", 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235014+0000", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235015+0000", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1820000000000001e-06 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235020+0000", 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2089999999999996e-06 2024-04-25T05:41:48.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235035+0000", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4948e-05 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.235043+0000", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.038e-06 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.238385+0000", 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033422590000000002 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.238494+0000", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000108489 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.238604+0000", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109836 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243897+0000", 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0052932400000000003 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.247331+0000", 2024-04-25T05:41:48.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034339739999999998 2024-04-25T05:41:48.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388555+0000", 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14122436299999999 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388611+0000", 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5563999999999999e-05 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388650+0000", 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9628000000000001e-05 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124840 8.8 8:19b5bffb:::benchmark_data_smithi043_104001_object716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:59.243067+0000", 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.028164838999999, 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14580757699999999, 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124840 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243067+0000", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243067+0000", 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243068+0000", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.252e-06 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243074+0000", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.665e-06 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243083+0000", 2024-04-25T05:41:48.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6400000000000003e-06 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243091+0000", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9310000000000008e-06 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.243941+0000", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084997899999999995 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.244088+0000", 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014767299999999999 2024-04-25T05:41:48.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.244234+0000", 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014609000000000001 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.247315+0000", 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003080812 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.247336+0000", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.107e-05 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388699+0000", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14136269300000001 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388836+0000", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000136959 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:59.388875+0000", 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8813000000000003e-05 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127833 8.d 8:b40552da:::benchmark_data_smithi043_104001_object3709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.446988050999998, 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15866919099999999, 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.358 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127833 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824246+0000", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7579999999999999e-06 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824252+0000", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0270000000000001e-06 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824263+0000", 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1644999999999999e-05 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824271+0000", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9889999999999992e-06 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828470+0000", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041991429999999998 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828597+0000", 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126799 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828726+0000", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012869300000000001 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842094+0000", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013367591999999999 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842428+0000", 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033413200000000001 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.845112+0000", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002684559 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982745+0000", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13763259899999999 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982785+0000", 2024-04-25T05:41:48.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0163000000000002e-05 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982913+0000", 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012809200000000001 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127834 8.d 8:b2f67481:::benchmark_data_smithi043_104001_object3710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:41:48.364 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.446720585999998, 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.164348995, 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127834 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824512+0000", 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4199999999999995e-07 2024-04-25T05:41:48.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824520+0000", 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.644e-06 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824529+0000", 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2590000000000006e-06 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824537+0000", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6249999999999998e-06 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831153+0000", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066163419999999999 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831291+0000", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013770899999999999 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831427+0000", 2024-04-25T05:41:48.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013640500000000001 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842101+0000", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010673741 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842497+0000", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039574500000000003 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.849394+0000", 2024-04-25T05:41:48.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068978620000000003 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988675+0000", 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13928105199999999 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988738+0000", 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2385999999999995e-05 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988860+0000", 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000122483 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127839 8.d 8:b29d5e75:::benchmark_data_smithi043_104001_object3715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.832809+0000", 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.438423356999998, 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15650397199999999, 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.371 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127839 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832809+0000", 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832809+0000", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832809+0000", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.89e-07 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832812+0000", 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7489999999999999e-06 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832818+0000", 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8000000000000004e-06 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.832823+0000", 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6710000000000004e-06 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.834348+0000", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00152502 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.834464+0000", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115734 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.834590+0000", 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125649 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842107+0000", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0075173000000000002 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842539+0000", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043154499999999998 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.849442+0000", 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069039720000000004 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989073+0000", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.139630538 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989190+0000", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011671900000000001 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989313+0000", 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000122786 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127846 8.d 8:b20cee10:::benchmark_data_smithi043_104001_object3722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.836884+0000", 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.434347944999999, 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15257949400000001, 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127846 2024-04-25T05:41:48.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836884+0000", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836884+0000", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836885+0000", 2024-04-25T05:41:48.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2079999999999999e-06 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836890+0000", 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7859999999999996e-06 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836899+0000", 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5270000000000004e-06 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.836906+0000", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7889999999999997e-06 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.837954+0000", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001047936 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.838097+0000", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014324499999999999 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.838241+0000", 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000143991 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842113+0000", 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038716850000000001 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.848257+0000", 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0061442709999999998 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.849478+0000", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012205829999999999 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989358+0000", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13987970399999999 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989392+0000", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4396000000000001e-05 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989463+0000", 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1372999999999996e-05 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127852 8.4 8:28015ae5:::benchmark_data_smithi043_104001_object3728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.842597+0000", 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.428635294999999, 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13943143899999999, 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.384 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127852 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842597+0000", 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842597+0000", 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842598+0000", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5900000000000005e-07 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842602+0000", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8409999999999999e-06 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842615+0000", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2106000000000001e-05 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842622+0000", 2024-04-25T05:41:48.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7239999999999992e-06 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842749+0000", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126413 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842865+0000", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116168 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4,6", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.843002+0000", 2024-04-25T05:41:48.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013746900000000001 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.847039+0000", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040371249999999999 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.847322+0000", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028218600000000003 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.848227+0000", 2024-04-25T05:41:48.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00090536700000000002 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.981552+0000", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13332531 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.981615+0000", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3022999999999996e-05 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982028+0000", 2024-04-25T05:41:48.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000412748 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127857 8.d 8:bf88734e:::benchmark_data_smithi043_104001_object3733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.852660+0000", 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.418571729, 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13695539500000001, 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.391 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127857 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852660+0000", 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852660+0000", 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852662+0000", 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5039999999999999e-06 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852669+0000", 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9460000000000001e-06 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852685+0000", 2024-04-25T05:41:48.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6407000000000001e-05 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852695+0000", 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8239999999999995e-06 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852944+0000", 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024950299999999999 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.853080+0000", 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013537600000000001 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.853214+0000", 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013407299999999999 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.857454+0000", 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042405489999999997 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.857493+0000", 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8928999999999999e-05 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.857550+0000", 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6718000000000001e-05 2024-04-25T05:41:48.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989511+0000", 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13196110599999999 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989545+0000", 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3822999999999998e-05 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.989616+0000", 2024-04-25T05:41:48.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0636999999999995e-05 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127858 8.4 8:26d20dff:::benchmark_data_smithi043_104001_object3734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.852716+0000", 2024-04-25T05:41:48.403 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.418516142000001, 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13367062499999999, 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127858 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852716+0000", 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852716+0000", 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852716+0000", 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5599999999999995e-07 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852720+0000", 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3289999999999998e-06 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852727+0000", 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9059999999999996e-06 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852733+0000", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0109999999999997e-06 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.852838+0000", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105052 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.853011+0000", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017319300000000001 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4,6", 2024-04-25T05:41:48.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.853169+0000", 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015809900000000001 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.857558+0000", 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043891349999999997 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.857941+0000", 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000383159 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.858674+0000", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00073259300000000002 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982818+0000", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.124144164 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982876+0000", 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8038000000000002e-05 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.986386+0000", 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00351039 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:128384 8.4 8:24dba3fe:::benchmark_data_smithi043_104001_object4260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:03.549932+0000", 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.721300135, 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14597813900000001, 2024-04-25T05:41:48.410 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 128384 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549932+0000", 2024-04-25T05:41:48.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549932+0000", 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549933+0000", 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.497e-06 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549939+0000", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.293e-06 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549948+0000", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3489999999999998e-06 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.549955+0000", 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4630000000000004e-06 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.551069+0000", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001113705 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.551185+0000", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011618200000000001 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4,6", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.551338+0000", 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015243 2024-04-25T05:41:48.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.556078+0000", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047407780000000002 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.556177+0000", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8049999999999998e-05 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.557513+0000", 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013365670000000001 2024-04-25T05:41:48.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.694081+0000", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.136567991 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.695766+0000", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0016850859999999999 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.695910+0000", 2024-04-25T05:41:48.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014374799999999999 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:128388 8.4 8:24936a41:::benchmark_data_smithi043_104001_object4264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:03.550959+0000", 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 44.720272559999998, 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.145187913, 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:48.417 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 128388 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550959+0000", 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550959+0000", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550960+0000", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550968+0000", 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9149999999999996e-06 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550982+0000", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4133e-05 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.550990+0000", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.328e-06 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.553706+0000", 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002716244 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.553802+0000", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5853999999999994e-05 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4,6", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.553884+0000", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2439000000000001e-05 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.556234+0000", 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00234995 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.557065+0000", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00083066800000000001 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.557561+0000", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000495902 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.695963+0000", 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.138402105 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.696023+0000", 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0056000000000002e-05 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:03.696147+0000", 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000124399 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:48.423 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:41:48.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:48.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:41:48.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[27010]: osdmap e262: 8 total, 8 up, 7 in 2024-04-25T05:41:48.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:48.424 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:48.425 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:41:48.425 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:41:48.425 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[37492]: osdmap e262: 8 total, 8 up, 7 in 2024-04-25T05:41:48.425 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:41:48.425 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:47 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:41:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:48.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:48.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.584 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:48.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:48.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:48.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:48.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:48.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:48.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:48.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:49.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:49.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:49.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:41:49.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:49.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:49.247 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:48 smithi060 ceph-mon[30898]: pgmap v405: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 15686/256785 objects misplaced (6.109%); 41 MiB/s, 659 objects/s recovering 2024-04-25T05:41:49.247 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:48 smithi060 ceph-mon[30898]: osdmap e263: 8 total, 8 up, 7 in 2024-04-25T05:41:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:49.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:49.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:41:49.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:48 smithi043 ceph-mon[27010]: pgmap v405: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 15686/256785 objects misplaced (6.109%); 41 MiB/s, 659 objects/s recovering 2024-04-25T05:41:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:48 smithi043 ceph-mon[27010]: osdmap e263: 8 total, 8 up, 7 in 2024-04-25T05:41:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:48 smithi043 ceph-mon[37492]: pgmap v405: 158 pgs: 5 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 15686/256785 objects misplaced (6.109%); 41 MiB/s, 659 objects/s recovering 2024-04-25T05:41:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:48 smithi043 ceph-mon[37492]: osdmap e263: 8 total, 8 up, 7 in 2024-04-25T05:41:49.408 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:49.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:49.561 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:49.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:49.675 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:49.712 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:49.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:49.849 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:49.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:49.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:49.982 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:49.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:50.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:50.118 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:50.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:50.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:168997 8.9 8:942c722a:::benchmark_data_smithi043_104001_object44873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:50.268167+0000", 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.033387751, 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033416329000000002, 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:41:50.302 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 168997 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268167+0000", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268167+0000", 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:41:50.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268168+0000", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.234e-06 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268174+0000", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6439999999999997e-06 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268189+0000", 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5169e-05 2024-04-25T05:41:50.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268197+0000", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5220000000000001e-06 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268252+0000", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5479000000000002e-05 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268359+0000", 2024-04-25T05:41:50.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010636 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.268472+0000", 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000113402 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.271201+0000", 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027288669999999998 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:41:50.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:50.271601+0000", 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039985599999999997 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 1 2024-04-25T05:41:50.307 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:41:50.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:50.389 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:50.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:50.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:50.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:50.660 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:50.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:50.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:50.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:50.941 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:50.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:51.013 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:41:51.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:51.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:51.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:51 smithi060 ceph-mon[30898]: pgmap v407: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 739 B/s rd, 0 B/s wr, 958 op/s; 11118/244671 objects misplaced (4.544%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:51.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1969914106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.101 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:41:51.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.187 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:51.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:51.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:51.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[27010]: pgmap v407: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 739 B/s rd, 0 B/s wr, 958 op/s; 11118/244671 objects misplaced (4.544%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1969914106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[37492]: pgmap v407: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 739 B/s rd, 0 B/s wr, 958 op/s; 11118/244671 objects misplaced (4.544%); 70 MiB/s, 1.13k objects/s recovering 2024-04-25T05:41:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1969914106' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:51 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:41:51.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:51.473 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.510 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:51.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:51.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:51.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:51.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:51.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:51.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:51.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:51.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:52.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.068 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:52.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:52.207 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:52.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:52.309 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:52.331 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:52 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:41:52.331 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:52 smithi060 ceph-mon[30898]: osdmap e264: 8 total, 8 up, 7 in 2024-04-25T05:41:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:41:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:52 smithi043 ceph-mon[27010]: osdmap e264: 8 total, 8 up, 7 in 2024-04-25T05:41:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:41:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:52 smithi043 ceph-mon[37492]: osdmap e264: 8 total, 8 up, 7 in 2024-04-25T05:41:52.445 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:41:52.445 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:41:52.445 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:41:52.445 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:41:52.445 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:41:52.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:52.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:52.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:52.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:52.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:52.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:41:52.799 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:52.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:52.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:52.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:52.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:53.009 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:53.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:53.113 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:53 smithi060 ceph-mon[30898]: pgmap v409: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 713 op/s; 11118/244671 objects misplaced (4.544%); 40 MiB/s, 641 objects/s recovering 2024-04-25T05:41:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:53.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:53.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:41:53.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:53.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:53.340 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:53 smithi043 ceph-mon[27010]: pgmap v409: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 713 op/s; 11118/244671 objects misplaced (4.544%); 40 MiB/s, 641 objects/s recovering 2024-04-25T05:41:53.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:53 smithi043 ceph-mon[37492]: pgmap v409: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 713 op/s; 11118/244671 objects misplaced (4.544%); 40 MiB/s, 641 objects/s recovering 2024-04-25T05:41:53.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:53.477 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:53.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:41:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:41:53.548 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:53.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:53.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:53.621 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:53.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:53.637 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:53.680 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:53.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:53.732 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:53.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:53.777 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:53.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:41:53.822 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:53.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:53.874 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:53.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:53.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:53.928 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:53.936 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:53.936 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:53.937 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:54.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.079 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:54.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/900361650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/900361650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:54.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:54.213 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:54.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:54.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:54.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:54.415 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:41:54.415 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/900361650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:41:54.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:54.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:54.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:54.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:54.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:54.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:54.811 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:54.811 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:54.856 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:41:54.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:54.927 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:54.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:55.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:55.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:55.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[27010]: pgmap v410: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.33k op/s; 8076/230118 objects misplaced (3.510%); 57 MiB/s, 916 objects/s recovering 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[27010]: osdmap e265: 8 total, 8 up, 7 in 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[37492]: pgmap v410: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.33k op/s; 8076/230118 objects misplaced (3.510%); 57 MiB/s, 916 objects/s recovering 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:41:55.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:55 smithi043 ceph-mon[37492]: osdmap e265: 8 total, 8 up, 7 in 2024-04-25T05:41:55.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:55.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:55.289 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:55 smithi060 ceph-mon[30898]: pgmap v410: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.33k op/s; 8076/230118 objects misplaced (3.510%); 57 MiB/s, 916 objects/s recovering 2024-04-25T05:41:55.289 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:41:55.289 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2448415375' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:41:55.289 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:55 smithi060 ceph-mon[30898]: osdmap e265: 8 total, 8 up, 7 in 2024-04-25T05:41:55.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:55.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:55.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:55.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:41:55.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:55.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:55.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:55.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:55.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:55.953 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:55.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:56.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:41:56.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:56.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:56.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:56.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:56.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:56.451 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:56.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:56.551 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:56.551 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:56.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:56.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:56.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.762 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:56.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:56.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:56.896 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:56.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:56.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:57.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:57.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:57.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:41:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:57.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:57.194 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:57 smithi060 ceph-mon[30898]: pgmap v412: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 654 B/s rd, 0 B/s wr, 965 op/s; 8076/230118 objects misplaced (3.510%); 50 MiB/s, 803 objects/s recovering 2024-04-25T05:41:57.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:41:57.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:57.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:57 smithi043 ceph-mon[27010]: pgmap v412: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 654 B/s rd, 0 B/s wr, 965 op/s; 8076/230118 objects misplaced (3.510%); 50 MiB/s, 803 objects/s recovering 2024-04-25T05:41:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:57 smithi043 ceph-mon[37492]: pgmap v412: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 654 B/s rd, 0 B/s wr, 965 op/s; 8076/230118 objects misplaced (3.510%); 50 MiB/s, 803 objects/s recovering 2024-04-25T05:41:57.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:57.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:57.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:57.554 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:41:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.600 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:57.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:57.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:57.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:57.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:41:57.893 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:57.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:57.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:41:57.922 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:41:58.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:58.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:41:58.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:41:58.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:41:58.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:41:58.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:58.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:58.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:58.605 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:58.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:58.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:58.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:58.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:58.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:58.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:59.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:41:59.095 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:41:59.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:41:59.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:59 smithi060 ceph-mon[30898]: pgmap v413: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 8076/229134 objects misplaced (3.525%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:41:59.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/219053334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.148 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:41:59 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.233 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:41:59.234 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:41:59.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:59.276 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:41:59.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[27010]: pgmap v413: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 8076/229134 objects misplaced (3.525%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:41:59.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/219053334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.320 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:41:59.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[37492]: pgmap v413: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 8076/229134 objects misplaced (3.525%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:41:59.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/219053334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:41:59.366 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:41:59.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:59.414 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:41:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:59.461 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:41:59.503 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:41:59.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:41:59.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:41:59.575 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:41:59.617 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:41:59.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:41:59.664 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:41:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:41:59.689 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:41:59.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:41:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:41:59.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:41:59.861 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:41:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:41:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:41:59.867 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:41:59.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:41:59.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:41:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:41:59.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:00.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:00.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:00.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:00.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:00.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: osdmap e266: 8 total, 8 up, 7 in 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3993649305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3822167790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[27010]: osdmap e267: 8 total, 8 up, 7 in 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: osdmap e266: 8 total, 8 up, 7 in 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3993649305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3822167790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:00.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:42:00.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:00 smithi043 ceph-mon[37492]: osdmap e267: 8 total, 8 up, 7 in 2024-04-25T05:42:00.180 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:00.180 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:00.180 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: osdmap e266: 8 total, 8 up, 7 in 2024-04-25T05:42:00.180 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3993649305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.181 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:42:00.181 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3822167790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:00.181 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:42:00.181 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:00 smithi060 ceph-mon[30898]: osdmap e267: 8 total, 8 up, 7 in 2024-04-25T05:42:00.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:00.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:42:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:00.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:00.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:00.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:00.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:00.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:00.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:00.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:00.667 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:42:00.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:00.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:00.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:00.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:01.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:01.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:01.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:01.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:01.335 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:01 smithi043 ceph-mon[27010]: pgmap v415: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 5523/215394 objects misplaced (2.564%); 36 MiB/s, 570 objects/s recovering 2024-04-25T05:42:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:01 smithi043 ceph-mon[37492]: pgmap v415: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 5523/215394 objects misplaced (2.564%); 36 MiB/s, 570 objects/s recovering 2024-04-25T05:42:01.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:01.452 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:01 smithi060 ceph-mon[30898]: pgmap v415: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 5523/215394 objects misplaced (2.564%); 36 MiB/s, 570 objects/s recovering 2024-04-25T05:42:01.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:01.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:01.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-25T05:42:01.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61441 7.5 7:a916e294:::benchmark_data_smithi043_74356_object61440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:42:01.649 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 369.47199254899999, 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098210427000000003, 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61441 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174496+0000", 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174498+0000", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6130000000000001e-06 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174543+0000", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5485999999999998e-05 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174552+0000", 2024-04-25T05:42:01.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.174557+0000", 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2220000000000001e-06 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175099+0000", 2024-04-25T05:42:01.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000542155 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175220+0000", 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121154 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.175330+0000", 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109385 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176611+0000", 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001281199 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.182295+0000", 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056836250000000003 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185068+0000", 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027733480000000001 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.185119+0000", 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1099e-05 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272707+0000", 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.087587571000000003 2024-04-25T05:42:01.656 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:61442 7.5 7:af9252c3:::benchmark_data_smithi043_74356_object61441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 369.46967626899999, 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096090660999999994, 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.657 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 61442 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176812+0000", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176814+0000", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.533e-06 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176873+0000", 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8678000000000003e-05 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176883+0000", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0701e-05 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176892+0000", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4479999999999994e-06 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.176940+0000", 2024-04-25T05:42:01.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8111000000000001e-05 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177024+0000", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4243e-05 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.177104+0000", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9800000000000002e-05 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.178395+0000", 2024-04-25T05:42:01.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001291489 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.184162+0000", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005766658 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272787+0000", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088625132999999995 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272844+0000", 2024-04-25T05:42:01.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7209000000000001e-05 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:35:52.272903+0000", 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8658e-05 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130500 7.5 7:a4f80177:::benchmark_data_smithi043_74356_object13631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 297.51928595999999, 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10185490899999999, 2024-04-25T05:42:01.663 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130500 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127204+0000", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4050000000000001e-06 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127211+0000", 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6930000000000001e-06 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127225+0000", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3815000000000001e-05 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127234+0000", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1719999999999996e-06 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127294+0000", 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9904000000000001e-05 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127421+0000", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012758499999999999 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127543+0000", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012183199999999999 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.129963+0000", 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024198269999999998 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.130082+0000", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.228908+0000", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098825536000000005 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229009+0000", 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101307 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229058+0000", 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8513000000000001e-05 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131177 7.5 7:a3c24791:::benchmark_data_smithi043_74356_object14308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 296.5890617, 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1201996, 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.669 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131177 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057428+0000", 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5799999999999998e-07 2024-04-25T05:42:01.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057435+0000", 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3420000000000002e-06 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057451+0000", 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6067999999999999e-05 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057460+0000", 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8990000000000007e-06 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058761+0000", 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013012029999999999 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058890+0000", 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012905600000000001 2024-04-25T05:42:01.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.059004+0000", 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011395699999999999 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.061474+0000", 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024696000000000002 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.063725+0000", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022509140000000001 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177555+0000", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11383001700000001 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177596+0000", 2024-04-25T05:42:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0970000000000002e-05 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177627+0000", 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0815999999999998e-05 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131521 7.5 7:a454f61c:::benchmark_data_smithi043_74356_object14652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:42:01.676 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 296.07342747199999, 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14413125099999999, 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131521 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573063+0000", 2024-04-25T05:42:01.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.449e-06 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573069+0000", 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1099999999999999e-06 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573082+0000", 2024-04-25T05:42:01.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3181e-05 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573089+0000", 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2300000000000002e-06 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573149+0000", 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9386999999999999e-05 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573243+0000", 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4758000000000003e-05 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573328+0000", 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4320999999999994e-05 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.575369+0000", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00204163 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.577122+0000", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017531059999999999 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717083+0000", 2024-04-25T05:42:01.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13996059199999999 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717144+0000", 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1328000000000006e-05 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717193+0000", 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8158999999999997e-05 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.684 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131530 7.5 7:a6cf53ae:::benchmark_data_smithi043_74356_object14661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 296.03181843800002, 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10267322399999999, 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.685 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131530 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614672+0000", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2419999999999999e-06 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614677+0000", 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5010000000000004e-06 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614688+0000", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1085e-05 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614697+0000", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4749999999999993e-06 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614749+0000", 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2003000000000003e-05 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614882+0000", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013371999999999999 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.615037+0000", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015419300000000001 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.617721+0000", 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026844970000000001 2024-04-25T05:42:01.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.621195+0000", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034735780000000002 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717252+0000", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096057038999999997 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717303+0000", 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1363000000000003e-05 2024-04-25T05:42:01.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717344+0000", 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0528000000000002e-05 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:188543 7.5 7:ac5985ff:::benchmark_data_smithi043_74356_object71674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 220.328157175, 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10455339800000001, 2024-04-25T05:42:01.691 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 188543 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318333+0000", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3009999999999999e-06 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318338+0000", 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9389999999999996e-06 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318349+0000", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1216999999999999e-05 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318357+0000", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6359999999999994e-06 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318398+0000", 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1177999999999998e-05 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318596+0000", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019781900000000001 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318760+0000", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000164198 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321782+0000", 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003021714 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422025+0000", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10024319900000001 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422069+0000", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3764999999999999e-05 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422680+0000", 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061150799999999997 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422770+0000", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0245000000000001e-05 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422885+0000", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114679 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.24734.0:188544 7.8 e165/88 7:16da1a97:::benchmark_data_smithi043_74356_object71675:head v 165'11663, mlcod=165'11660)", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:42:01.697 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 220.32520764700001, 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100748163, 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321281+0000", 2024-04-25T05:42:01.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321282+0000", 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5000000000000002e-07 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321292+0000", 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0033e-05 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321300+0000", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4540000000000007e-06 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321307+0000", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.742e-06 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321365+0000", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7552000000000001e-05 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321377+0000", 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2500999999999999e-05 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421970+0000", 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100592746 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422029+0000", 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9484999999999997e-05 2024-04-25T05:42:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPushReply(7.6 165/160 [PushReplyOp(7:6e80069b:::benchmark_data_smithi043_74356_object89093:head)])", 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 220.32372218200001, 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100303718, 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322767+0000", 2024-04-25T05:42:01.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6100000000000002e-07 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322770+0000", 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0000000000000001e-06 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.322775+0000", 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6929999999999998e-06 2024-04-25T05:42:01.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421939+0000", 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099164609000000001 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422361+0000", 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00042228799999999999 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.423070+0000", 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00070886699999999996 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:188543 7.5 e165/160 ondisk, result = 0)", 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 220.32242818700001, 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097973324000000001, 2024-04-25T05:42:01.706 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:42:01.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324061+0000", 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5799999999999999e-07 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.324064+0000", 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8289999999999999e-06 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421928+0000", 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.097864313999999994 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421944+0000", 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5991e-05 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422007+0000", 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2378000000000005e-05 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422017+0000", 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0069e-05 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422034+0000", 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7184999999999999e-05 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.24734.0:188543 7.5 e165/160 ondisk, result = 0)", 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 220.32097553599999, 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096562752000000002, 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.711 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325513+0000", 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325514+0000", 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2300000000000001e-07 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.325518+0000", 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2030000000000002e-06 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421938+0000", 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096420230999999995 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.421950+0000", 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1931e-05 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422056+0000", 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105924 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422063+0000", 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8070000000000002e-06 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422076+0000", 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3033e-05 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:195837 7.5 7:ad811058:::benchmark_data_smithi043_74356_object78968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 210.390426875, 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096179872999999999, 2024-04-25T05:42:01.716 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195837 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:42:01.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256062+0000", 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256063+0000", 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5999999999999999e-06 2024-04-25T05:42:01.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256068+0000", 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8849999999999998e-06 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256080+0000", 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1585e-05 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256087+0000", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3289999999999996e-06 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256191+0000", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103528 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256285+0000", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3924000000000004e-05 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:42:01.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.256361+0000", 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6687999999999995e-05 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.260072+0000", 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0037107910000000002 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.262721+0000", 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026484740000000001 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.263704+0000", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0009836109999999999 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.351998+0000", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088294124000000002 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.352074+0000", 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5123000000000006e-05 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.352242+0000", 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016821100000000001 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196430 7.6 7:67a0b441:::benchmark_data_smithi043_74356_object79561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 209.747996457, 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17769133400000001, 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196430 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898496+0000", 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5209999999999998e-06 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898500+0000", 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9099999999999998e-06 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898509+0000", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0329999999999997e-06 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898514+0000", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5659999999999997e-06 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898569+0000", 2024-04-25T05:42:01.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4317999999999997e-05 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898692+0000", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012370400000000001 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898805+0000", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011234300000000001 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900717+0000", 2024-04-25T05:42:01.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001912569 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900846+0000", 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128825 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076080+0000", 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17523397700000001 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076138+0000", 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7815000000000001e-05 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076184+0000", 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5753000000000001e-05 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196431 7.6 7:64208f17:::benchmark_data_smithi043_74356_object79562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 209.74498160499999, 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17835189000000001, 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.730 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196431 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901509+0000", 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7439999999999999e-06 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901517+0000", 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8520000000000004e-06 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901526+0000", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4679999999999998e-06 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901534+0000", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8720000000000002e-06 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901588+0000", 2024-04-25T05:42:01.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3638999999999999e-05 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901700+0000", 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112355 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901816+0000", 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116359 2024-04-25T05:42:01.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.903841+0000", 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020240940000000002 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907001+0000", 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031600640000000002 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079789+0000", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17278797700000001 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079821+0000", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2480000000000001e-05 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079859+0000", 2024-04-25T05:42:01.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7985999999999998e-05 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196437 7.5 7:a3615d17:::benchmark_data_smithi043_74356_object79568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 209.74194224999999, 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16688376599999999, 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.737 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196437 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904548+0000", 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904553+0000", 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2429999999999996e-06 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904563+0000", 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0046e-05 2024-04-25T05:42:01.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904571+0000", 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0169999999999996e-06 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904710+0000", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013878799999999999 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904838+0000", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128014 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904935+0000", 2024-04-25T05:42:01.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7627999999999994e-05 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.906988+0000", 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020523189999999999 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908135+0000", 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00114685 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908341+0000", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020669199999999999 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071248+0000", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.162907146 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071322+0000", 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3274000000000004e-05 2024-04-25T05:42:01.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071430+0000", 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010847400000000001 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196438 7.6 7:6894f35c:::benchmark_data_smithi043_74356_object79569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:42:01.746 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 209.741837522, 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17980945200000001, 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196438 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904652+0000", 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2099999999999996e-07 2024-04-25T05:42:01.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904655+0000", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4249999999999998e-06 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904666+0000", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185999999999999e-05 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904673+0000", 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8009999999999998e-06 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904726+0000", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2479999999999999e-05 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904836+0000", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110016 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904947+0000", 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111721 2024-04-25T05:42:01.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907188+0000", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022400810000000001 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907463+0000", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027584899999999999 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084360+0000", 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17689687000000001 2024-04-25T05:42:01.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084421+0000", 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1032000000000001e-05 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084461+0000", 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9369999999999997e-05 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198457 7.5 7:aeb56927:::benchmark_data_smithi043_74356_object81588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:42:01.752 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 205.84229325000001, 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97568060099999998, 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198457 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804197+0000", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.544e-06 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804204+0000", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7569999999999998e-06 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804216+0000", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1841000000000001e-05 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804224+0000", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7099999999999996e-06 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804278+0000", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3275000000000001e-05 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804284+0000", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6030000000000001e-06 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773320+0000", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96903589899999998 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773471+0000", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015128900000000001 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773614+0000", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014288699999999999 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777300+0000", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036858519999999999 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778657+0000", 2024-04-25T05:42:01.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013569280000000001 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779814+0000", 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001156481 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779846+0000", 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2005999999999998e-05 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.758 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779876+0000", 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0528999999999999e-05 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198459 7.5 7:a46444d1:::benchmark_data_smithi043_74356_object81590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 205.84195015399999, 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.977629847, 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.759 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198459 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3499999999999996e-07 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804544+0000", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3390000000000003e-06 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804553+0000", 2024-04-25T05:42:01.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.969e-06 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804559+0000", 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.55e-06 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804600+0000", 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1149999999999997e-05 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804606+0000", 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4179999999999996e-06 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775688+0000", 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97108227300000005 2024-04-25T05:42:01.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.765 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.765 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775807+0000", 2024-04-25T05:42:01.765 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011906899999999999 2024-04-25T05:42:01.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775929+0000", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012165199999999999 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779949+0000", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040201059999999999 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779988+0000", 2024-04-25T05:42:01.766 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8943000000000003e-05 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782076+0000", 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020885729999999998 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782128+0000", 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2080999999999998e-05 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.767 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782168+0000", 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0089000000000002e-05 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198466 7.5 7:a9a716c6:::benchmark_data_smithi043_74356_object81597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 205.83703072599999, 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97804308600000001, 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198466 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809459+0000", 2024-04-25T05:42:01.770 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4529999999999999e-06 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809465+0000", 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8669999999999999e-06 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809479+0000", 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.324e-05 2024-04-25T05:42:01.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809487+0000", 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2020000000000004e-06 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809543+0000", 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6376999999999999e-05 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.772 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809555+0000", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1435e-05 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777347+0000", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96779198399999999 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777465+0000", 2024-04-25T05:42:01.773 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011815999999999999 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777591+0000", 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012601299999999999 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779953+0000", 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023620949999999998 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782689+0000", 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027361059999999999 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787431+0000", 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047419230000000003 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787467+0000", 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6081000000000002e-05 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787501+0000", 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4150000000000003e-05 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198479 7.5 7:a8eb3f00:::benchmark_data_smithi043_74356_object81610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 205.823335778, 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96450478500000003, 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:42:01.777 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198479 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:42:01.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823155+0000", 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.717e-06 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:42:01.779 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823172+0000", 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7611000000000001e-05 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823191+0000", 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8842e-05 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823202+0000", 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0407e-05 2024-04-25T05:42:01.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823258+0000", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6428999999999998e-05 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823265+0000", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4699999999999996e-06 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778700+0000", 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95543422700000002 2024-04-25T05:42:01.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778769+0000", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9831000000000003e-05 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778858+0000", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8744000000000003e-05 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779956+0000", 2024-04-25T05:42:01.782 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010981400000000001 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782730+0000", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002773546 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787552+0000", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048221590000000003 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787612+0000", 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9880999999999998e-05 2024-04-25T05:42:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787658+0000", 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5781000000000001e-05 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:42:01.784 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:42:01.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:01.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:01.938 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:01.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:02.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:02.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:02.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:42:02.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:02.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:02.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:02.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:02 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:02 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:02.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:02 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:02.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:02 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:02.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:02.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:02 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:02.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:02 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:02.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:02.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:02.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:42:02.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:02.813 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:02.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:02.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:02.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:03.049 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:03.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:03.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:03.198 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:03.205 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:03.240 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:03.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:03.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:03.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[37492]: pgmap v417: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 0 B/s wr, 705 op/s; 5523/215394 objects misplaced (2.564%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:42:03.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[37492]: osdmap e268: 8 total, 8 up, 7 in 2024-04-25T05:42:03.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:03.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[27010]: pgmap v417: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 0 B/s wr, 705 op/s; 5523/215394 objects misplaced (2.564%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:42:03.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[27010]: osdmap e268: 8 total, 8 up, 7 in 2024-04-25T05:42:03.353 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:03.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:03.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:03 smithi060 ceph-mon[30898]: pgmap v417: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 0 B/s wr, 705 op/s; 5523/215394 objects misplaced (2.564%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:42:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:03 smithi060 ceph-mon[30898]: osdmap e268: 8 total, 8 up, 7 in 2024-04-25T05:42:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:03.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:03.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:03.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:03.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:03.767 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:03.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:03.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:03.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:03.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:04.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:04.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:04.161 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:04.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:04.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:04.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:04.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:04.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:04.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:04.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:04.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:04.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.745 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:04.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:04.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:42:04.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:04.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:183768 8.c e269/233 8:3ca5fbb8:::benchmark_data_smithi043_104001_object59644:head v 269'11492, mlcod=268'11485)", 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:04.941734+0000", 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00087268199999999995, 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00090463800000000003, 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:04.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941734+0000", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941734+0000", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941735+0000", 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5099999999999998e-07 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941743+0000", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7800000000000001e-06 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941754+0000", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0648999999999999e-05 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941761+0000", 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9289999999999998e-06 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941876+0000", 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000114503 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.942034+0000", 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015833899999999999 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:183771 8.c e269/233)", 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:04.941782+0000", 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00082506799999999998, 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00092953700000000003, 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941782+0000", 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941782+0000", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941782+0000", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6300000000000001e-07 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941787+0000", 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6800000000000001e-06 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941795+0000", 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7849999999999992e-06 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:04.941800+0000", 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7690000000000002e-06 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-04-25T05:42:04.949 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:42:04.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:05.039 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:05.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:42:05.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:05.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:05.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:05.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:05.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: pgmap v419: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.64k op/s; 2823/199554 objects misplaced (1.415%); 47 MiB/s, 751 objects/s recovering 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1916323522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[27010]: osdmap e269: 8 total, 8 up, 7 in 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: pgmap v419: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.64k op/s; 2823/199554 objects misplaced (1.415%); 47 MiB/s, 751 objects/s recovering 2024-04-25T05:42:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1916323522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:42:05.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:05.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:05.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:05 smithi043 ceph-mon[37492]: osdmap e269: 8 total, 8 up, 7 in 2024-04-25T05:42:05.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:05.424 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: pgmap v419: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.64k op/s; 2823/199554 objects misplaced (1.415%); 47 MiB/s, 751 objects/s recovering 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1916323522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:05 smithi060 ceph-mon[30898]: osdmap e269: 8 total, 8 up, 7 in 2024-04-25T05:42:05.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:05.563 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:05.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:05.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:05.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:05.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:05.871 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.0 to 1 (10000) 2024-04-25T05:42:05.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:05.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:05.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:06.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:06.127 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1533095406' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[27010]: osdmap e270: 8 total, 8 up, 7 in 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1533095406' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:42:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:06 smithi043 ceph-mon[37492]: osdmap e270: 8 total, 8 up, 7 in 2024-04-25T05:42:06.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:06.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:06.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:06.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:06.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1533095406' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:42:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:42:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:06 smithi060 ceph-mon[30898]: osdmap e270: 8 total, 8 up, 7 in 2024-04-25T05:42:06.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:06.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:06.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:06.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:06.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:06.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:06.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:06.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:06.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:42:06.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:07.024 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:07.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: pgmap v421: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 2823/199554 objects misplaced (1.415%); 25 MiB/s, 397 objects/s recovering 2024-04-25T05:42:07.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:42:07.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:42:07.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:42:07.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:42:07.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:07.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:07.146 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:07.157 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:07.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:07.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:07.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:07.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[27010]: osdmap e271: 8 total, 8 up, 7 in 2024-04-25T05:42:07.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: pgmap v421: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 2823/199554 objects misplaced (1.415%); 25 MiB/s, 397 objects/s recovering 2024-04-25T05:42:07.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:42:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:42:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:42:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:42:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:07 smithi043 ceph-mon[37492]: osdmap e271: 8 total, 8 up, 7 in 2024-04-25T05:42:07.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:07.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: pgmap v421: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 880 op/s; 2823/199554 objects misplaced (1.415%); 25 MiB/s, 397 objects/s recovering 2024-04-25T05:42:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:42:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:42:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:42:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:42:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:07 smithi060 ceph-mon[30898]: osdmap e271: 8 total, 8 up, 7 in 2024-04-25T05:42:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:07.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:07.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:07.649 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:42:07.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:42:07.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:07.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:07.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:07.899 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:07.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:08.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:08.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:08.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:08.164 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:08.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:42:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:42:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[27010]: osdmap e272: 8 total, 8 up, 7 in 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:08.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:08 smithi043 ceph-mon[37492]: osdmap e272: 8 total, 8 up, 7 in 2024-04-25T05:42:08.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:08.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:42:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:08 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:42:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:08 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:42:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:08 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:08 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:08 smithi060 ceph-mon[30898]: osdmap e272: 8 total, 8 up, 7 in 2024-04-25T05:42:08.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.574 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:08.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:08.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.706 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:08.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:08.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:08.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:08.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:08.979 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:08.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:09.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:09.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:09.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:09.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.385 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:09.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[27010]: pgmap v424: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 960 op/s; 2823/198903 objects misplaced (1.419%); 26 MiB/s, 416 objects/s recovering 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[37492]: pgmap v424: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 960 op/s; 2823/198903 objects misplaced (1.419%); 26 MiB/s, 416 objects/s recovering 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:42:09.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:42:09.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:09 smithi060 ceph-mon[30898]: pgmap v424: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 893 B/s rd, 0 B/s wr, 960 op/s; 2823/198903 objects misplaced (1.419%); 26 MiB/s, 416 objects/s recovering 2024-04-25T05:42:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:09 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:42:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:09 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:42:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:09.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:09.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:09.659 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:09.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:09.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:09.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:09.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:09.958 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:10.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:10.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:10.071 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:10.130 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:10.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:10.174 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:10.216 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:10.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:42:10.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: osdmap e273: 8 total, 8 up, 7 in 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3048298984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:42:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:42:10.218 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: osdmap e273: 8 total, 8 up, 7 in 2024-04-25T05:42:10.218 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3048298984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:10.256 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:10.297 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:10.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:10.336 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:10.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:10.374 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:10.382 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:10.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:10.488 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: osdmap e273: 8 total, 8 up, 7 in 2024-04-25T05:42:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3048298984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:10.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:10.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:10.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:10.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:10.882 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:10.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:10.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:11.014 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:11.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:11.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:11.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:11.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:11.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:11.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[27010]: pgmap v426: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:11.399 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[37492]: pgmap v426: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:11.400 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:11 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:11.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:11 smithi060 ceph-mon[30898]: pgmap v426: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:11 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:42:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:11 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:42:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:11 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:11 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:11.516 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:11.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:11.639 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:11.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:11.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:11.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:11.888 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:11.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:11.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:12.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:12.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:12.138 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:12.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.270 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:12.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:12.350 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:42:12.350 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:42:12.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:12.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:12 smithi060 ceph-mon[30898]: 8.e scrub starts 2024-04-25T05:42:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:12 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:42:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:12 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:42:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:12 smithi060 ceph-mon[30898]: osdmap e274: 8 total, 8 up, 7 in 2024-04-25T05:42:12.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:12.542 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[27010]: 8.e scrub starts 2024-04-25T05:42:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:42:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:42:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[27010]: osdmap e274: 8 total, 8 up, 7 in 2024-04-25T05:42:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[37492]: 8.e scrub starts 2024-04-25T05:42:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:42:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:42:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:12 smithi043 ceph-mon[37492]: osdmap e274: 8 total, 8 up, 7 in 2024-04-25T05:42:12.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:12.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:12.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.817 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:12.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:12.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:12.955 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:12.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:12.994 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:13.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:13.090 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:13.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:13.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:13.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:13.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[37492]: pgmap v429: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[37492]: osdmap e275: 8 total, 8 up, 7 in 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[27010]: pgmap v429: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:42:13.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:13 smithi043 ceph-mon[27010]: osdmap e275: 8 total, 8 up, 7 in 2024-04-25T05:42:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.492 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:13.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:13.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:13 smithi060 ceph-mon[30898]: pgmap v429: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 704 op/s; 313/189090 objects misplaced (0.166%); 31 MiB/s, 488 objects/s recovering 2024-04-25T05:42:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:13 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:42:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:13 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:42:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:13 smithi060 ceph-mon[30898]: osdmap e275: 8 total, 8 up, 7 in 2024-04-25T05:42:13.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:13.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:13.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:13.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:13.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:13.888 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:13.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:13.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:14.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:14.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.158 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:14.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:14.292 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:14.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:14.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:14.443 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:14.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:42:14.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:42:14.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:42:14.446 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:14.446 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:42:14.446 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:14 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:42:14.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:14.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:14 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:42:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:14 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:42:14.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:14.578 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:14.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:14.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:14.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:14.843 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:14.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:14.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:14.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:15.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:15.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:15.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: pgmap v431: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 1.23k op/s; 29 MiB/s, 458 objects/s recovering 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: 8.1 scrub starts 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:15.244 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:15.245 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:42:15.245 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:15 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:42:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.246 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:15.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:15.383 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: pgmap v431: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 1.23k op/s; 29 MiB/s, 458 objects/s recovering 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: 8.1 scrub starts 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:42:15.463 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: pgmap v431: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 1.23k op/s; 29 MiB/s, 458 objects/s recovering 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: 8.1 scrub starts 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:42:15.464 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:15 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:42:15.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:15.521 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:15.623 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:15.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:15.698 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:15.761 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:42:15.765 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:42:15.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.814 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:15.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:15.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:42:15.952 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:15.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:16.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:16.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:16.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:16.237 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:42:16.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:16.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:16.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 6.7 scrub starts 2024-04-25T05:42:16.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 6.7 scrub ok 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 8.1 scrub ok 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: osdmap e276: 8 total, 8 up, 7 in 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:42:16.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 6.7 scrub starts 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 6.7 scrub ok 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 8.1 scrub ok 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: osdmap e276: 8 total, 8 up, 7 in 2024-04-25T05:42:16.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:42:16.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:42:16.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:16.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:16.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 6.7 scrub starts 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 6.7 scrub ok 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 8.1 scrub ok 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: osdmap e276: 8 total, 8 up, 7 in 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:42:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:42:16.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:16.628 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:42:16.628 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:42:16.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:16.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:16.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:16.799 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:16.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:16.936 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:16.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:17.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:17.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:17.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:17.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:17.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:17.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:17.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:17.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: pgmap v433: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 0 B/s wr, 868 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: osdmap e277: 8 total, 8 up, 7 in 2024-04-25T05:42:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: 6.6 scrub starts 2024-04-25T05:42:17.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:17 smithi060 ceph-mon[30898]: 6.6 scrub ok 2024-04-25T05:42:17.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:17.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: pgmap v433: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 0 B/s wr, 868 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: osdmap e277: 8 total, 8 up, 7 in 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: 6.6 scrub starts 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[27010]: 6.6 scrub ok 2024-04-25T05:42:17.604 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: from='client.25352 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: pgmap v433: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 0 B/s wr, 868 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2449838950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: osdmap e277: 8 total, 8 up, 7 in 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: 6.6 scrub starts 2024-04-25T05:42:17.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:17 smithi043 ceph-mon[37492]: 6.6 scrub ok 2024-04-25T05:42:17.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:17.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:42:17.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:17.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:17.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:17.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:17.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:18.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:18.156 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:18.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:18.258 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:18.291 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:18.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.423 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:18.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:18.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.559 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:18.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:18.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:18.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:18.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:18.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:42:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:42:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[27010]: pgmap v435: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:18.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:42:18.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:42:18.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:18.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:18.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:18 smithi043 ceph-mon[37492]: pgmap v435: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:18.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:18.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:18.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:18 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:42:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:18 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:42:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:18 smithi060 ceph-mon[30898]: pgmap v435: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 3.2 MiB/s, 51 objects/s recovering 2024-04-25T05:42:19.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:19.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:19.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:19.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:19.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:19.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:19.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:19.469 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:19.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:19.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:19.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:19.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:19.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:19.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:19.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:19.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:19.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[37492]: 4.2 deep-scrub starts 2024-04-25T05:42:19.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[37492]: 4.2 deep-scrub ok 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[37492]: 1.0 deep-scrub starts 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[37492]: 1.0 deep-scrub ok 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[27010]: 4.2 deep-scrub starts 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[27010]: 4.2 deep-scrub ok 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[27010]: 1.0 deep-scrub starts 2024-04-25T05:42:19.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:19 smithi043 ceph-mon[27010]: 1.0 deep-scrub ok 2024-04-25T05:42:19.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:19.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:19 smithi060 ceph-mon[30898]: 4.2 deep-scrub starts 2024-04-25T05:42:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:19 smithi060 ceph-mon[30898]: 4.2 deep-scrub ok 2024-04-25T05:42:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:19 smithi060 ceph-mon[30898]: 1.0 deep-scrub starts 2024-04-25T05:42:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:19 smithi060 ceph-mon[30898]: 1.0 deep-scrub ok 2024-04-25T05:42:20.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:20.110 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:20.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:42:20.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:20.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:20.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:20.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:20.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:20.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:20.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:20.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:20.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:20.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:20.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:42:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[27010]: 4.1a scrub starts 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[27010]: 4.1a scrub ok 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[27010]: pgmap v436: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[37492]: 4.1a scrub starts 2024-04-25T05:42:20.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[37492]: 4.1a scrub ok 2024-04-25T05:42:20.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:20 smithi043 ceph-mon[37492]: pgmap v436: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-04-25T05:42:20.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:20.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:20.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:20 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:42:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:20 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:42:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:20 smithi060 ceph-mon[30898]: 4.1a scrub starts 2024-04-25T05:42:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:20 smithi060 ceph-mon[30898]: 4.1a scrub ok 2024-04-25T05:42:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:20 smithi060 ceph-mon[30898]: pgmap v436: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.55k op/s; 2.5 MiB/s, 40 objects/s recovering 2024-04-25T05:42:21.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:21.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:21.122 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-04-25T05:42:15.194284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3776780851,"num_objects":57851,"num_object_clones":0,"num_object_copies":173553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57851,"num_whiteouts":0,"num_read":1473,"num_read_kb":1193,"num_write":191251,"num_write_kb":7943790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63961,"num_bytes_recovered":4171243691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73060,"ondisk_log_size":73060,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14537676,"kb_used_data":12082808,"kb_used_omap":10,"kb_used_meta":2454709,"kb_avail":641678388,"statfs":{"total":671965249536,"available":657078669312,"internally_reserved":0,"allocated":12372795392,"data_stored":12368995866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2513622123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-339410944,"num_objects":-5179,"num_object_clones":0,"num_object_copies":-15537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-313,"num_objects_unfound":0,"num_objects_dirty":-5179,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5179,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":20250624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.373231"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":124,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214234+0000","last_change":"2024-04-25T05:41:35.583008+0000","last_active":"2024-04-25T05:42:12.214234+0000","last_peered":"2024-04-25T05:42:12.214234+0000","last_clean":"2024-04-25T05:42:12.214234+0000","last_became_active":"2024-04-25T05:40:09.734217+0000","last_became_peered":"2024-04-25T05:40:09.734217+0000","last_unstale":"2024-04-25T05:42:12.214234+0000","last_undegraded":"2024-04-25T05:42:12.214234+0000","last_fullsized":"2024-04-25T05:42:12.214234+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:01.911520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171477+0000","last_change":"2024-04-25T05:40:37.473300+0000","last_active":"2024-04-25T05:42:12.171477+0000","last_peered":"2024-04-25T05:42:12.171477+0000","last_clean":"2024-04-25T05:42:12.171477+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:42:12.171477+0000","last_undegraded":"2024-04-25T05:42:12.171477+0000","last_fullsized":"2024-04-25T05:42:12.171477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:37.473018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.473018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":101,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173320+0000","last_change":"2024-04-25T05:40:55.954113+0000","last_active":"2024-04-25T05:42:12.173320+0000","last_peered":"2024-04-25T05:42:12.173320+0000","last_clean":"2024-04-25T05:42:12.173320+0000","last_became_active":"2024-04-25T05:40:55.953726+0000","last_became_peered":"2024-04-25T05:40:55.953726+0000","last_unstale":"2024-04-25T05:42:12.173320+0000","last_undegraded":"2024-04-25T05:42:12.173320+0000","last_fullsized":"2024-04-25T05:42:12.173320+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":518,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175588+0000","last_change":"2024-04-25T05:42:06.942759+0000","last_active":"2024-04-25T05:42:12.175588+0000","last_peered":"2024-04-25T05:42:12.175588+0000","last_clean":"2024-04-25T05:42:12.175588+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:42:12.175588+0000","last_undegraded":"2024-04-25T05:42:12.175588+0000","last_fullsized":"2024-04-25T05:42:12.175588+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:42:06.942600+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:42:06.942600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:25.919821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220699+0000","last_change":"2024-04-25T05:39:29.799097+0000","last_active":"2024-04-25T05:42:12.220699+0000","last_peered":"2024-04-25T05:42:12.220699+0000","last_clean":"2024-04-25T05:42:12.220699+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:42:12.220699+0000","last_undegraded":"2024-04-25T05:42:12.220699+0000","last_fullsized":"2024-04-25T05:42:12.220699+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":413,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171481+0000","last_change":"2024-04-25T05:40:13.865954+0000","last_active":"2024-04-25T05:42:12.171481+0000","last_peered":"2024-04-25T05:42:12.171481+0000","last_clean":"2024-04-25T05:42:12.171481+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:42:12.171481+0000","last_undegraded":"2024-04-25T05:42:12.171481+0000","last_fullsized":"2024-04-25T05:42:12.171481+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:13.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:13.865834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":465,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221440+0000","last_change":"2024-04-25T05:40:55.982555+0000","last_active":"2024-04-25T05:42:12.221440+0000","last_peered":"2024-04-25T05:42:12.221440+0000","last_clean":"2024-04-25T05:42:12.221440+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:42:12.221440+0000","last_undegraded":"2024-04-25T05:42:12.221440+0000","last_fullsized":"2024-04-25T05:42:12.221440+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.485053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":597,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220596+0000","last_change":"2024-04-25T05:40:56.000345+0000","last_active":"2024-04-25T05:42:12.220596+0000","last_peered":"2024-04-25T05:42:12.220596+0000","last_clean":"2024-04-25T05:42:12.220596+0000","last_became_active":"2024-04-25T05:40:55.974452+0000","last_became_peered":"2024-04-25T05:40:55.974452+0000","last_unstale":"2024-04-25T05:42:12.220596+0000","last_undegraded":"2024-04-25T05:42:12.220596+0000","last_fullsized":"2024-04-25T05:42:12.220596+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:39:41.219950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.219950+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172000+0000","last_change":"2024-04-25T05:40:43.441659+0000","last_active":"2024-04-25T05:42:12.172000+0000","last_peered":"2024-04-25T05:42:12.172000+0000","last_clean":"2024-04-25T05:42:12.172000+0000","last_became_active":"2024-04-25T05:40:43.441308+0000","last_became_peered":"2024-04-25T05:40:43.441308+0000","last_unstale":"2024-04-25T05:42:12.172000+0000","last_undegraded":"2024-04-25T05:42:12.172000+0000","last_fullsized":"2024-04-25T05:42:12.172000+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.213950+0000","last_change":"2024-04-25T05:40:43.453942+0000","last_active":"2024-04-25T05:42:12.213950+0000","last_peered":"2024-04-25T05:42:12.213950+0000","last_clean":"2024-04-25T05:42:12.213950+0000","last_became_active":"2024-04-25T05:40:43.453366+0000","last_became_peered":"2024-04-25T05:40:43.453366+0000","last_unstale":"2024-04-25T05:42:12.213950+0000","last_undegraded":"2024-04-25T05:42:12.213950+0000","last_fullsized":"2024-04-25T05:42:12.213950+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":466,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214377+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:42:12.214377+0000","last_peered":"2024-04-25T05:42:12.214377+0000","last_clean":"2024-04-25T05:42:12.214377+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:42:12.214377+0000","last_undegraded":"2024-04-25T05:42:12.214377+0000","last_fullsized":"2024-04-25T05:42:12.214377+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":196,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222292+0000","last_change":"2024-04-25T05:40:43.462701+0000","last_active":"2024-04-25T05:42:12.222292+0000","last_peered":"2024-04-25T05:42:12.222292+0000","last_clean":"2024-04-25T05:42:12.222292+0000","last_became_active":"2024-04-25T05:40:43.433465+0000","last_became_peered":"2024-04-25T05:40:43.433465+0000","last_unstale":"2024-04-25T05:42:12.222292+0000","last_undegraded":"2024-04-25T05:42:12.222292+0000","last_fullsized":"2024-04-25T05:42:12.222292+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:36:49.747877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:49.747877+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212164+0000","last_change":"2024-04-25T05:41:31.571478+0000","last_active":"2024-04-25T05:42:12.212164+0000","last_peered":"2024-04-25T05:42:12.212164+0000","last_clean":"2024-04-25T05:42:12.212164+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:42:12.212164+0000","last_undegraded":"2024-04-25T05:42:12.212164+0000","last_fullsized":"2024-04-25T05:42:12.212164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.853234+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175687+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:42:12.175687+0000","last_peered":"2024-04-25T05:42:12.175687+0000","last_clean":"2024-04-25T05:42:12.175687+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:42:12.175687+0000","last_undegraded":"2024-04-25T05:42:12.175687+0000","last_fullsized":"2024-04-25T05:42:12.175687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.487676+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214513+0000","last_change":"2024-04-25T05:40:49.444611+0000","last_active":"2024-04-25T05:42:12.214513+0000","last_peered":"2024-04-25T05:42:12.214513+0000","last_clean":"2024-04-25T05:42:12.214513+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:42:12.214513+0000","last_undegraded":"2024-04-25T05:42:12.214513+0000","last_fullsized":"2024-04-25T05:42:12.214513+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:49.444492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.444492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":179,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.374054+0000","last_change":"2024-04-25T05:42:13.374107+0000","last_active":"2024-04-25T05:42:13.374054+0000","last_peered":"2024-04-25T05:42:13.374054+0000","last_clean":"2024-04-25T05:42:13.374054+0000","last_became_active":"2024-04-25T05:40:55.984807+0000","last_became_peered":"2024-04-25T05:40:55.984807+0000","last_unstale":"2024-04-25T05:42:13.374054+0000","last_undegraded":"2024-04-25T05:42:13.374054+0000","last_fullsized":"2024-04-25T05:42:13.374054+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.061132+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176724+0000","last_change":"2024-04-25T05:40:14.837450+0000","last_active":"2024-04-25T05:42:12.176724+0000","last_peered":"2024-04-25T05:42:12.176724+0000","last_clean":"2024-04-25T05:42:12.176724+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:42:12.176724+0000","last_undegraded":"2024-04-25T05:42:12.176724+0000","last_fullsized":"2024-04-25T05:42:12.176724+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.837215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.837215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174633+0000","last_change":"2024-04-25T05:40:43.451998+0000","last_active":"2024-04-25T05:42:12.174633+0000","last_peered":"2024-04-25T05:42:12.174633+0000","last_clean":"2024-04-25T05:42:12.174633+0000","last_became_active":"2024-04-25T05:40:43.449712+0000","last_became_peered":"2024-04-25T05:40:43.449712+0000","last_unstale":"2024-04-25T05:42:12.174633+0000","last_undegraded":"2024-04-25T05:42:12.174633+0000","last_fullsized":"2024-04-25T05:42:12.174633+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:41.506653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.506653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173019+0000","last_change":"2024-04-25T05:40:43.435522+0000","last_active":"2024-04-25T05:42:12.173019+0000","last_peered":"2024-04-25T05:42:12.173019+0000","last_clean":"2024-04-25T05:42:12.173019+0000","last_became_active":"2024-04-25T05:40:43.434830+0000","last_became_peered":"2024-04-25T05:40:43.434830+0000","last_unstale":"2024-04-25T05:42:12.173019+0000","last_undegraded":"2024-04-25T05:42:12.173019+0000","last_fullsized":"2024-04-25T05:42:12.173019+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":305,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174130+0000","last_change":"2024-04-25T05:40:56.033623+0000","last_active":"2024-04-25T05:42:12.174130+0000","last_peered":"2024-04-25T05:42:12.174130+0000","last_clean":"2024-04-25T05:42:12.174130+0000","last_became_active":"2024-04-25T05:40:55.991291+0000","last_became_peered":"2024-04-25T05:40:55.991291+0000","last_unstale":"2024-04-25T05:42:12.174130+0000","last_undegraded":"2024-04-25T05:42:12.174130+0000","last_fullsized":"2024-04-25T05:42:12.174130+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:39:42.756928+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.756928+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222505+0000","last_change":"2024-04-25T05:39:29.799140+0000","last_active":"2024-04-25T05:42:12.222505+0000","last_peered":"2024-04-25T05:42:12.222505+0000","last_clean":"2024-04-25T05:42:12.222505+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:42:12.222505+0000","last_undegraded":"2024-04-25T05:42:12.222505+0000","last_fullsized":"2024-04-25T05:42:12.222505+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":123,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221200+0000","last_change":"2024-04-25T05:40:55.973839+0000","last_active":"2024-04-25T05:42:12.221200+0000","last_peered":"2024-04-25T05:42:12.221200+0000","last_clean":"2024-04-25T05:42:12.221200+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:42:12.221200+0000","last_undegraded":"2024-04-25T05:42:12.221200+0000","last_fullsized":"2024-04-25T05:42:12.221200+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":459,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177104+0000","last_change":"2024-04-25T05:42:11.885667+0000","last_active":"2024-04-25T05:42:12.177104+0000","last_peered":"2024-04-25T05:42:12.177104+0000","last_clean":"2024-04-25T05:42:12.177104+0000","last_became_active":"2024-04-25T05:40:55.964565+0000","last_became_peered":"2024-04-25T05:40:55.964565+0000","last_unstale":"2024-04-25T05:42:12.177104+0000","last_undegraded":"2024-04-25T05:42:12.177104+0000","last_fullsized":"2024-04-25T05:42:12.177104+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:11.885545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:11.885545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.830666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221117+0000","last_change":"2024-04-25T05:40:15.536952+0000","last_active":"2024-04-25T05:42:12.221117+0000","last_peered":"2024-04-25T05:42:12.221117+0000","last_clean":"2024-04-25T05:42:12.221117+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:42:12.221117+0000","last_undegraded":"2024-04-25T05:42:12.221117+0000","last_fullsized":"2024-04-25T05:42:12.221117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:40:15.536802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.536802+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":453,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176529+0000","last_change":"2024-04-25T05:40:55.989126+0000","last_active":"2024-04-25T05:42:12.176529+0000","last_peered":"2024-04-25T05:42:12.176529+0000","last_clean":"2024-04-25T05:42:12.176529+0000","last_became_active":"2024-04-25T05:40:55.974471+0000","last_became_peered":"2024-04-25T05:40:55.974471+0000","last_unstale":"2024-04-25T05:42:12.176529+0000","last_undegraded":"2024-04-25T05:42:12.176529+0000","last_fullsized":"2024-04-25T05:42:12.176529+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:40:10.849562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:10.849562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":198,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172411+0000","last_change":"2024-04-25T05:39:38.521412+0000","last_active":"2024-04-25T05:42:12.172411+0000","last_peered":"2024-04-25T05:42:12.172411+0000","last_clean":"2024-04-25T05:42:12.172411+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:42:12.172411+0000","last_undegraded":"2024-04-25T05:42:12.172411+0000","last_fullsized":"2024-04-25T05:42:12.172411+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:39:38.521265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.521265+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":455,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171441+0000","last_change":"2024-04-25T05:40:40.500687+0000","last_active":"2024-04-25T05:42:12.171441+0000","last_peered":"2024-04-25T05:42:12.171441+0000","last_clean":"2024-04-25T05:42:12.171441+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:42:12.171441+0000","last_undegraded":"2024-04-25T05:42:12.171441+0000","last_fullsized":"2024-04-25T05:42:12.171441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:40.500174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.500174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":109,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173061+0000","last_change":"2024-04-25T05:40:43.482944+0000","last_active":"2024-04-25T05:42:12.173061+0000","last_peered":"2024-04-25T05:42:12.173061+0000","last_clean":"2024-04-25T05:42:12.173061+0000","last_became_active":"2024-04-25T05:40:43.442720+0000","last_became_peered":"2024-04-25T05:40:43.442720+0000","last_unstale":"2024-04-25T05:42:12.173061+0000","last_undegraded":"2024-04-25T05:42:12.173061+0000","last_fullsized":"2024-04-25T05:42:12.173061+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":434,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174263+0000","last_change":"2024-04-25T05:40:15.703133+0000","last_active":"2024-04-25T05:42:12.174263+0000","last_peered":"2024-04-25T05:42:12.174263+0000","last_clean":"2024-04-25T05:42:12.174263+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:42:12.174263+0000","last_undegraded":"2024-04-25T05:42:12.174263+0000","last_fullsized":"2024-04-25T05:42:12.174263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.702999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.702999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":101,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173297+0000","last_change":"2024-04-25T05:40:55.967146+0000","last_active":"2024-04-25T05:42:12.173297+0000","last_peered":"2024-04-25T05:42:12.173297+0000","last_clean":"2024-04-25T05:42:12.173297+0000","last_became_active":"2024-04-25T05:40:55.966806+0000","last_became_peered":"2024-04-25T05:40:55.966806+0000","last_unstale":"2024-04-25T05:42:12.173297+0000","last_undegraded":"2024-04-25T05:42:12.173297+0000","last_fullsized":"2024-04-25T05:42:12.173297+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220207+0000","last_change":"2024-04-25T05:40:50.562106+0000","last_active":"2024-04-25T05:42:12.220207+0000","last_peered":"2024-04-25T05:42:12.220207+0000","last_clean":"2024-04-25T05:42:12.220207+0000","last_became_active":"2024-04-25T05:40:43.440363+0000","last_became_peered":"2024-04-25T05:40:43.440363+0000","last_unstale":"2024-04-25T05:42:12.220207+0000","last_undegraded":"2024-04-25T05:42:12.220207+0000","last_fullsized":"2024-04-25T05:42:12.220207+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.560317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.560317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:18.263940+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":176,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172581+0000","last_change":"2024-04-25T05:39:40.474408+0000","last_active":"2024-04-25T05:42:12.172581+0000","last_peered":"2024-04-25T05:42:12.172581+0000","last_clean":"2024-04-25T05:42:12.172581+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:42:12.172581+0000","last_undegraded":"2024-04-25T05:42:12.172581+0000","last_fullsized":"2024-04-25T05:42:12.172581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:40.474140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.474140+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":453,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175933+0000","last_change":"2024-04-25T05:41:39.031952+0000","last_active":"2024-04-25T05:42:12.175933+0000","last_peered":"2024-04-25T05:42:12.175933+0000","last_clean":"2024-04-25T05:42:12.175933+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:42:12.175933+0000","last_undegraded":"2024-04-25T05:42:12.175933+0000","last_fullsized":"2024-04-25T05:42:12.175933+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:41:39.031784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.031784+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.769610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":162,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171838+0000","last_change":"2024-04-25T05:41:40.152573+0000","last_active":"2024-04-25T05:42:12.171838+0000","last_peered":"2024-04-25T05:42:12.171838+0000","last_clean":"2024-04-25T05:42:12.171838+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:42:12.171838+0000","last_undegraded":"2024-04-25T05:42:12.171838+0000","last_fullsized":"2024-04-25T05:42:12.171838+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:40.152411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:40.152411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:02.701355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173263+0000","last_change":"2024-04-25T05:40:43.446312+0000","last_active":"2024-04-25T05:42:12.173263+0000","last_peered":"2024-04-25T05:42:12.173263+0000","last_clean":"2024-04-25T05:42:12.173263+0000","last_became_active":"2024-04-25T05:40:43.445367+0000","last_became_peered":"2024-04-25T05:40:43.445367+0000","last_unstale":"2024-04-25T05:42:12.173263+0000","last_undegraded":"2024-04-25T05:42:12.173263+0000","last_fullsized":"2024-04-25T05:42:12.173263+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220303+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:42:12.220303+0000","last_peered":"2024-04-25T05:42:12.220303+0000","last_clean":"2024-04-25T05:42:12.220303+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:42:12.220303+0000","last_undegraded":"2024-04-25T05:42:12.220303+0000","last_fullsized":"2024-04-25T05:42:12.220303+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:23.390274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220923+0000","last_change":"2024-04-25T05:40:43.435991+0000","last_active":"2024-04-25T05:42:12.220923+0000","last_peered":"2024-04-25T05:42:12.220923+0000","last_clean":"2024-04-25T05:42:12.220923+0000","last_became_active":"2024-04-25T05:40:43.435420+0000","last_became_peered":"2024-04-25T05:40:43.435420+0000","last_unstale":"2024-04-25T05:42:12.220923+0000","last_undegraded":"2024-04-25T05:42:12.220923+0000","last_fullsized":"2024-04-25T05:42:12.220923+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:39:03.160752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176300+0000","last_change":"2024-04-25T05:40:53.928517+0000","last_active":"2024-04-25T05:42:12.176300+0000","last_peered":"2024-04-25T05:42:12.176300+0000","last_clean":"2024-04-25T05:42:12.176300+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:42:12.176300+0000","last_undegraded":"2024-04-25T05:42:12.176300+0000","last_fullsized":"2024-04-25T05:42:12.176300+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:53.928372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.928372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.213617+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:42:12.213617+0000","last_peered":"2024-04-25T05:42:12.213617+0000","last_clean":"2024-04-25T05:42:12.213617+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:42:12.213617+0000","last_undegraded":"2024-04-25T05:42:12.213617+0000","last_fullsized":"2024-04-25T05:42:12.213617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":513,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220567+0000","last_change":"2024-04-25T05:40:49.558496+0000","last_active":"2024-04-25T05:42:12.220567+0000","last_peered":"2024-04-25T05:42:12.220567+0000","last_clean":"2024-04-25T05:42:12.220567+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:42:12.220567+0000","last_undegraded":"2024-04-25T05:42:12.220567+0000","last_fullsized":"2024-04-25T05:42:12.220567+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:49.558354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.558354+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171785+0000","last_change":"2024-04-25T05:40:43.446609+0000","last_active":"2024-04-25T05:42:12.171785+0000","last_peered":"2024-04-25T05:42:12.171785+0000","last_clean":"2024-04-25T05:42:12.171785+0000","last_became_active":"2024-04-25T05:40:43.446278+0000","last_became_peered":"2024-04-25T05:40:43.446278+0000","last_unstale":"2024-04-25T05:42:12.171785+0000","last_undegraded":"2024-04-25T05:42:12.171785+0000","last_fullsized":"2024-04-25T05:42:12.171785+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220606+0000","last_change":"2024-04-25T05:40:43.433578+0000","last_active":"2024-04-25T05:42:12.220606+0000","last_peered":"2024-04-25T05:42:12.220606+0000","last_clean":"2024-04-25T05:42:12.220606+0000","last_became_active":"2024-04-25T05:40:43.433240+0000","last_became_peered":"2024-04-25T05:40:43.433240+0000","last_unstale":"2024-04-25T05:42:12.220606+0000","last_undegraded":"2024-04-25T05:42:12.220606+0000","last_fullsized":"2024-04-25T05:42:12.220606+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.526768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.526768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176808+0000","last_change":"2024-04-25T05:40:17.791243+0000","last_active":"2024-04-25T05:42:12.176808+0000","last_peered":"2024-04-25T05:42:12.176808+0000","last_clean":"2024-04-25T05:42:12.176808+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:42:12.176808+0000","last_undegraded":"2024-04-25T05:42:12.176808+0000","last_fullsized":"2024-04-25T05:42:12.176808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:17.791104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.791104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"257'13","reported_seq":367,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.213961+0000","last_change":"2024-04-25T05:41:30.532539+0000","last_active":"2024-04-25T05:42:12.213961+0000","last_peered":"2024-04-25T05:42:12.213961+0000","last_clean":"2024-04-25T05:42:12.213961+0000","last_became_active":"2024-04-25T05:40:55.954865+0000","last_became_peered":"2024-04-25T05:40:55.954865+0000","last_unstale":"2024-04-25T05:42:12.213961+0000","last_undegraded":"2024-04-25T05:42:12.213961+0000","last_fullsized":"2024-04-25T05:42:12.213961+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:59.262752+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220358+0000","last_change":"2024-04-25T05:40:43.434425+0000","last_active":"2024-04-25T05:42:12.220358+0000","last_peered":"2024-04-25T05:42:12.220358+0000","last_clean":"2024-04-25T05:42:12.220358+0000","last_became_active":"2024-04-25T05:40:43.434051+0000","last_became_peered":"2024-04-25T05:40:43.434051+0000","last_unstale":"2024-04-25T05:42:12.220358+0000","last_undegraded":"2024-04-25T05:42:12.220358+0000","last_fullsized":"2024-04-25T05:42:12.220358+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:04.446776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:04.446776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172816+0000","last_change":"2024-04-25T05:40:06.644197+0000","last_active":"2024-04-25T05:42:12.172816+0000","last_peered":"2024-04-25T05:42:12.172816+0000","last_clean":"2024-04-25T05:42:12.172816+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:42:12.172816+0000","last_undegraded":"2024-04-25T05:42:12.172816+0000","last_fullsized":"2024-04-25T05:42:12.172816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.644083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.644083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":133,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176415+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:42:12.176415+0000","last_peered":"2024-04-25T05:42:12.176415+0000","last_clean":"2024-04-25T05:42:12.176415+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:42:12.176415+0000","last_undegraded":"2024-04-25T05:42:12.176415+0000","last_fullsized":"2024-04-25T05:42:12.176415+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.152477+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":540,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176874+0000","last_change":"2024-04-25T05:40:56.002007+0000","last_active":"2024-04-25T05:42:12.176874+0000","last_peered":"2024-04-25T05:42:12.176874+0000","last_clean":"2024-04-25T05:42:12.176874+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:42:12.176874+0000","last_undegraded":"2024-04-25T05:42:12.176874+0000","last_fullsized":"2024-04-25T05:42:12.176874+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:40:17.835081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.835081+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173536+0000","last_change":"2024-04-25T05:40:43.443261+0000","last_active":"2024-04-25T05:42:12.173536+0000","last_peered":"2024-04-25T05:42:12.173536+0000","last_clean":"2024-04-25T05:42:12.173536+0000","last_became_active":"2024-04-25T05:40:43.439460+0000","last_became_peered":"2024-04-25T05:40:43.439460+0000","last_unstale":"2024-04-25T05:42:12.173536+0000","last_undegraded":"2024-04-25T05:42:12.173536+0000","last_fullsized":"2024-04-25T05:42:12.173536+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"233'10","reported_seq":574,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214553+0000","last_change":"2024-04-25T05:41:45.481735+0000","last_active":"2024-04-25T05:42:12.214553+0000","last_peered":"2024-04-25T05:42:12.214553+0000","last_clean":"2024-04-25T05:42:12.214553+0000","last_became_active":"2024-04-25T05:40:43.453246+0000","last_became_peered":"2024-04-25T05:40:43.453246+0000","last_unstale":"2024-04-25T05:42:12.214553+0000","last_undegraded":"2024-04-25T05:42:12.214553+0000","last_fullsized":"2024-04-25T05:42:12.214553+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:48.532916+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176237+0000","last_change":"2024-04-25T05:41:37.012236+0000","last_active":"2024-04-25T05:42:12.176237+0000","last_peered":"2024-04-25T05:42:12.176237+0000","last_clean":"2024-04-25T05:42:12.176237+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:42:12.176237+0000","last_undegraded":"2024-04-25T05:42:12.176237+0000","last_fullsized":"2024-04-25T05:42:12.176237+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:37.012159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:37.012159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.873027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":481,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221031+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:42:12.221031+0000","last_peered":"2024-04-25T05:42:12.221031+0000","last_clean":"2024-04-25T05:42:12.221031+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:42:12.221031+0000","last_undegraded":"2024-04-25T05:42:12.221031+0000","last_fullsized":"2024-04-25T05:42:12.221031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:15.927150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":82,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212046+0000","last_change":"2024-04-25T05:40:55.954634+0000","last_active":"2024-04-25T05:42:12.212046+0000","last_peered":"2024-04-25T05:42:12.212046+0000","last_clean":"2024-04-25T05:42:12.212046+0000","last_became_active":"2024-04-25T05:40:55.954181+0000","last_became_peered":"2024-04-25T05:40:55.954181+0000","last_unstale":"2024-04-25T05:42:12.212046+0000","last_undegraded":"2024-04-25T05:42:12.212046+0000","last_fullsized":"2024-04-25T05:42:12.212046+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:09.913324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.913324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":547,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221512+0000","last_change":"2024-04-25T05:40:56.012131+0000","last_active":"2024-04-25T05:42:12.221512+0000","last_peered":"2024-04-25T05:42:12.221512+0000","last_clean":"2024-04-25T05:42:12.221512+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:42:12.221512+0000","last_undegraded":"2024-04-25T05:42:12.221512+0000","last_fullsized":"2024-04-25T05:42:12.221512+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:40:37.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.529721+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173691+0000","last_change":"2024-04-25T05:40:43.434277+0000","last_active":"2024-04-25T05:42:12.173691+0000","last_peered":"2024-04-25T05:42:12.173691+0000","last_clean":"2024-04-25T05:42:12.173691+0000","last_became_active":"2024-04-25T05:40:43.433933+0000","last_became_peered":"2024-04-25T05:40:43.433933+0000","last_unstale":"2024-04-25T05:42:12.173691+0000","last_undegraded":"2024-04-25T05:42:12.173691+0000","last_fullsized":"2024-04-25T05:42:12.173691+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":483,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214786+0000","last_change":"2024-04-25T05:42:08.829436+0000","last_active":"2024-04-25T05:42:12.214786+0000","last_peered":"2024-04-25T05:42:12.214786+0000","last_clean":"2024-04-25T05:42:12.214786+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:42:12.214786+0000","last_undegraded":"2024-04-25T05:42:12.214786+0000","last_fullsized":"2024-04-25T05:42:12.214786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:22.650865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":89,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177237+0000","last_change":"2024-04-25T05:42:10.907844+0000","last_active":"2024-04-25T05:42:12.177237+0000","last_peered":"2024-04-25T05:42:12.177237+0000","last_clean":"2024-04-25T05:42:12.177237+0000","last_became_active":"2024-04-25T05:40:55.953924+0000","last_became_peered":"2024-04-25T05:40:55.953924+0000","last_unstale":"2024-04-25T05:42:12.177237+0000","last_undegraded":"2024-04-25T05:42:12.177237+0000","last_fullsized":"2024-04-25T05:42:12.177237+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:10.907707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.907707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:31.135983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":430,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222417+0000","last_change":"2024-04-25T05:40:38.559014+0000","last_active":"2024-04-25T05:42:12.222417+0000","last_peered":"2024-04-25T05:42:12.222417+0000","last_clean":"2024-04-25T05:42:12.222417+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:42:12.222417+0000","last_undegraded":"2024-04-25T05:42:12.222417+0000","last_fullsized":"2024-04-25T05:42:12.222417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:38.558895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:38.558895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":472,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175740+0000","last_change":"2024-04-25T05:40:50.906267+0000","last_active":"2024-04-25T05:42:12.175740+0000","last_peered":"2024-04-25T05:42:12.175740+0000","last_clean":"2024-04-25T05:42:12.175740+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:42:12.175740+0000","last_undegraded":"2024-04-25T05:42:12.175740+0000","last_fullsized":"2024-04-25T05:42:12.175740+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.905458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.905458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:20.239647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":109,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173183+0000","last_change":"2024-04-25T05:40:43.481667+0000","last_active":"2024-04-25T05:42:12.173183+0000","last_peered":"2024-04-25T05:42:12.173183+0000","last_clean":"2024-04-25T05:42:12.173183+0000","last_became_active":"2024-04-25T05:40:43.441558+0000","last_became_peered":"2024-04-25T05:40:43.441558+0000","last_unstale":"2024-04-25T05:42:12.173183+0000","last_undegraded":"2024-04-25T05:42:12.173183+0000","last_fullsized":"2024-04-25T05:42:12.173183+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:36:20.154900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:20.154900+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175635+0000","last_change":"2024-04-25T05:41:27.185161+0000","last_active":"2024-04-25T05:42:12.175635+0000","last_peered":"2024-04-25T05:42:12.175635+0000","last_clean":"2024-04-25T05:42:12.175635+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:42:12.175635+0000","last_undegraded":"2024-04-25T05:42:12.175635+0000","last_fullsized":"2024-04-25T05:42:12.175635+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.185044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.185044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:28.448062+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212925+0000","last_change":"2024-04-25T05:40:46.378509+0000","last_active":"2024-04-25T05:42:12.212925+0000","last_peered":"2024-04-25T05:42:12.212925+0000","last_clean":"2024-04-25T05:42:12.212925+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:42:12.212925+0000","last_undegraded":"2024-04-25T05:42:12.212925+0000","last_fullsized":"2024-04-25T05:42:12.212925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:46.378394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:40:46.378394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":168,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171280+0000","last_change":"2024-04-25T05:40:43.431845+0000","last_active":"2024-04-25T05:42:12.171280+0000","last_peered":"2024-04-25T05:42:12.171280+0000","last_clean":"2024-04-25T05:42:12.171280+0000","last_became_active":"2024-04-25T05:40:43.431502+0000","last_became_peered":"2024-04-25T05:40:43.431502+0000","last_unstale":"2024-04-25T05:42:12.171280+0000","last_undegraded":"2024-04-25T05:42:12.171280+0000","last_fullsized":"2024-04-25T05:42:12.171280+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.918723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.918723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":463,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222510+0000","last_change":"2024-04-25T05:40:41.507574+0000","last_active":"2024-04-25T05:42:12.222510+0000","last_peered":"2024-04-25T05:42:12.222510+0000","last_clean":"2024-04-25T05:42:12.222510+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:42:12.222510+0000","last_undegraded":"2024-04-25T05:42:12.222510+0000","last_fullsized":"2024-04-25T05:42:12.222510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:41.507400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.507400+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"275'12037","reported_seq":26666,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.845876+0000","last_change":"2024-04-25T05:41:28.270522+0000","last_active":"2024-04-25T05:42:12.845876+0000","last_peered":"2024-04-25T05:42:12.845876+0000","last_clean":"2024-04-25T05:42:12.845876+0000","last_became_active":"2024-04-25T05:41:28.224071+0000","last_became_peered":"2024-04-25T05:41:28.224071+0000","last_unstale":"2024-04-25T05:42:12.845876+0000","last_undegraded":"2024-04-25T05:42:12.845876+0000","last_fullsized":"2024-04-25T05:42:12.845876+0000","mapping_epoch":255,"log_start":"239'7637","ondisk_log_start":"239'7637","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'7237","last_scrub_stamp":"2024-04-25T05:40:43.047218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.047218+0000","objects_scrubbed":7237,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245301248,"num_objects":3743,"num_object_clones":0,"num_object_copies":11229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12037,"num_write_kb":504960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7185,"num_bytes_recovered":470810624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":466,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214698+0000","last_change":"2024-04-25T05:40:51.460612+0000","last_active":"2024-04-25T05:42:12.214698+0000","last_peered":"2024-04-25T05:42:12.214698+0000","last_clean":"2024-04-25T05:42:12.214698+0000","last_became_active":"2024-04-25T05:40:43.453025+0000","last_became_peered":"2024-04-25T05:40:43.453025+0000","last_unstale":"2024-04-25T05:42:12.214698+0000","last_undegraded":"2024-04-25T05:42:12.214698+0000","last_fullsized":"2024-04-25T05:42:12.214698+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.460330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.460330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177169+0000","last_change":"2024-04-25T05:41:25.239411+0000","last_active":"2024-04-25T05:42:12.177169+0000","last_peered":"2024-04-25T05:42:12.177169+0000","last_clean":"2024-04-25T05:42:12.177169+0000","last_became_active":"2024-04-25T05:40:43.429608+0000","last_became_peered":"2024-04-25T05:40:43.429608+0000","last_unstale":"2024-04-25T05:42:12.177169+0000","last_undegraded":"2024-04-25T05:42:12.177169+0000","last_fullsized":"2024-04-25T05:42:12.177169+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:41:25.239292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:25.239292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:33.712255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":543,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212744+0000","last_change":"2024-04-25T05:39:30.386374+0000","last_active":"2024-04-25T05:42:12.212744+0000","last_peered":"2024-04-25T05:42:12.212744+0000","last_clean":"2024-04-25T05:42:12.212744+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:42:12.212744+0000","last_undegraded":"2024-04-25T05:42:12.212744+0000","last_fullsized":"2024-04-25T05:42:12.212744+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:39:30.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.386275+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":577,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176182+0000","last_change":"2024-04-25T05:39:33.128642+0000","last_active":"2024-04-25T05:42:12.176182+0000","last_peered":"2024-04-25T05:42:12.176182+0000","last_clean":"2024-04-25T05:42:12.176182+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:42:12.176182+0000","last_undegraded":"2024-04-25T05:42:12.176182+0000","last_fullsized":"2024-04-25T05:42:12.176182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'10","last_scrub_stamp":"2024-04-25T05:39:33.128477+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.128477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"275'12001","reported_seq":12419,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.856940+0000","last_change":"2024-04-25T05:41:36.031565+0000","last_active":"2024-04-25T05:42:12.856940+0000","last_peered":"2024-04-25T05:42:12.856940+0000","last_clean":"2024-04-25T05:42:12.856940+0000","last_became_active":"2024-04-25T05:39:35.778659+0000","last_became_peered":"2024-04-25T05:39:35.778659+0000","last_unstale":"2024-04-25T05:42:12.856940+0000","last_undegraded":"2024-04-25T05:42:12.856940+0000","last_fullsized":"2024-04-25T05:42:12.856940+0000","mapping_epoch":209,"log_start":"235'7260","ondisk_log_start":"235'7260","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9942","last_scrub_stamp":"2024-04-25T05:41:36.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.031226+0000","objects_scrubbed":5781,"log_size":4741,"log_dups_size":0,"ondisk_log_size":4741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.717941+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220802+0000","last_change":"2024-04-25T05:40:55.989744+0000","last_active":"2024-04-25T05:42:12.220802+0000","last_peered":"2024-04-25T05:42:12.220802+0000","last_clean":"2024-04-25T05:42:12.220802+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:42:12.220802+0000","last_undegraded":"2024-04-25T05:42:12.220802+0000","last_fullsized":"2024-04-25T05:42:12.220802+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.579078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.579078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177295+0000","last_change":"2024-04-25T05:39:38.030757+0000","last_active":"2024-04-25T05:42:12.177295+0000","last_peered":"2024-04-25T05:42:12.177295+0000","last_clean":"2024-04-25T05:42:12.177295+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:42:12.177295+0000","last_undegraded":"2024-04-25T05:42:12.177295+0000","last_fullsized":"2024-04-25T05:42:12.177295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.030627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.030627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171314+0000","last_change":"2024-04-25T05:40:56.145089+0000","last_active":"2024-04-25T05:42:12.171314+0000","last_peered":"2024-04-25T05:42:12.171314+0000","last_clean":"2024-04-25T05:42:12.171314+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:42:12.171314+0000","last_undegraded":"2024-04-25T05:42:12.171314+0000","last_fullsized":"2024-04-25T05:42:12.171314+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:40:37.943948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.943948+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":444,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171913+0000","last_change":"2024-04-25T05:40:55.988511+0000","last_active":"2024-04-25T05:42:12.171913+0000","last_peered":"2024-04-25T05:42:12.171913+0000","last_clean":"2024-04-25T05:42:12.171913+0000","last_became_active":"2024-04-25T05:40:55.988228+0000","last_became_peered":"2024-04-25T05:40:55.988228+0000","last_unstale":"2024-04-25T05:42:12.171913+0000","last_undegraded":"2024-04-25T05:42:12.171913+0000","last_fullsized":"2024-04-25T05:42:12.171913+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"275'11931","reported_seq":26698,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.271190+0000","last_change":"2024-04-25T05:41:42.929604+0000","last_active":"2024-04-25T05:42:13.271190+0000","last_peered":"2024-04-25T05:42:13.271190+0000","last_clean":"2024-04-25T05:42:13.271190+0000","last_became_active":"2024-04-25T05:41:42.899661+0000","last_became_peered":"2024-04-25T05:41:42.899661+0000","last_unstale":"2024-04-25T05:42:13.271190+0000","last_undegraded":"2024-04-25T05:42:13.271190+0000","last_fullsized":"2024-04-25T05:42:13.271190+0000","mapping_epoch":260,"log_start":"234'7157","ondisk_log_start":"234'7157","created":198,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237436928,"num_objects":3623,"num_object_clones":0,"num_object_copies":10869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11931,"num_write_kb":497728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7289,"num_bytes_recovered":477691904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":440,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175314+0000","last_change":"2024-04-25T05:40:55.982002+0000","last_active":"2024-04-25T05:42:12.175314+0000","last_peered":"2024-04-25T05:42:12.175314+0000","last_clean":"2024-04-25T05:42:12.175314+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:42:12.175314+0000","last_undegraded":"2024-04-25T05:42:12.175314+0000","last_fullsized":"2024-04-25T05:42:12.175314+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.074043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.074043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":443,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172460+0000","last_change":"2024-04-25T05:40:55.953740+0000","last_active":"2024-04-25T05:42:12.172460+0000","last_peered":"2024-04-25T05:42:12.172460+0000","last_clean":"2024-04-25T05:42:12.172460+0000","last_became_active":"2024-04-25T05:40:55.953404+0000","last_became_peered":"2024-04-25T05:40:55.953404+0000","last_unstale":"2024-04-25T05:42:12.172460+0000","last_undegraded":"2024-04-25T05:42:12.172460+0000","last_fullsized":"2024-04-25T05:42:12.172460+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":480,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177091+0000","last_change":"2024-04-25T05:40:08.880333+0000","last_active":"2024-04-25T05:42:12.177091+0000","last_peered":"2024-04-25T05:42:12.177091+0000","last_clean":"2024-04-25T05:42:12.177091+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:42:12.177091+0000","last_undegraded":"2024-04-25T05:42:12.177091+0000","last_fullsized":"2024-04-25T05:42:12.177091+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:08.880207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.880207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":184,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171878+0000","last_change":"2024-04-25T05:39:30.591246+0000","last_active":"2024-04-25T05:42:12.171878+0000","last_peered":"2024-04-25T05:42:12.171878+0000","last_clean":"2024-04-25T05:42:12.171878+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:42:12.171878+0000","last_undegraded":"2024-04-25T05:42:12.171878+0000","last_fullsized":"2024-04-25T05:42:12.171878+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:39:30.591091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.591091+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"275'11908","reported_seq":13040,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.353397+0000","last_change":"2024-04-25T05:40:53.660560+0000","last_active":"2024-04-25T05:42:13.353397+0000","last_peered":"2024-04-25T05:42:13.353397+0000","last_clean":"2024-04-25T05:42:13.353397+0000","last_became_active":"2024-04-25T05:39:35.777746+0000","last_became_peered":"2024-04-25T05:39:35.777746+0000","last_unstale":"2024-04-25T05:42:13.353397+0000","last_undegraded":"2024-04-25T05:42:13.353397+0000","last_fullsized":"2024-04-25T05:42:13.353397+0000","mapping_epoch":209,"log_start":"239'7613","ondisk_log_start":"239'7613","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7624","last_scrub_stamp":"2024-04-25T05:40:53.660173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.660173+0000","objects_scrubbed":7548,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:19.465644+0000","scrub_duration":3156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229376000,"num_objects":3500,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11908,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":101,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174266+0000","last_change":"2024-04-25T05:40:55.972738+0000","last_active":"2024-04-25T05:42:12.174266+0000","last_peered":"2024-04-25T05:42:12.174266+0000","last_clean":"2024-04-25T05:42:12.174266+0000","last_became_active":"2024-04-25T05:40:55.972405+0000","last_became_peered":"2024-04-25T05:40:55.972405+0000","last_unstale":"2024-04-25T05:42:12.174266+0000","last_undegraded":"2024-04-25T05:42:12.174266+0000","last_fullsized":"2024-04-25T05:42:12.174266+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":88,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177347+0000","last_change":"2024-04-25T05:42:05.954387+0000","last_active":"2024-04-25T05:42:12.177347+0000","last_peered":"2024-04-25T05:42:12.177347+0000","last_clean":"2024-04-25T05:42:12.177347+0000","last_became_active":"2024-04-25T05:40:55.981082+0000","last_became_peered":"2024-04-25T05:40:55.981082+0000","last_unstale":"2024-04-25T05:42:12.177347+0000","last_undegraded":"2024-04-25T05:42:12.177347+0000","last_fullsized":"2024-04-25T05:42:12.177347+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:05.954259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:05.954259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.753300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":466,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214882+0000","last_change":"2024-04-25T05:41:28.560703+0000","last_active":"2024-04-25T05:42:12.214882+0000","last_peered":"2024-04-25T05:42:12.214882+0000","last_clean":"2024-04-25T05:42:12.214882+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:42:12.214882+0000","last_undegraded":"2024-04-25T05:42:12.214882+0000","last_fullsized":"2024-04-25T05:42:12.214882+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:36.353516+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":107,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173202+0000","last_change":"2024-04-25T05:40:43.470705+0000","last_active":"2024-04-25T05:42:12.173202+0000","last_peered":"2024-04-25T05:42:12.173202+0000","last_clean":"2024-04-25T05:42:12.173202+0000","last_became_active":"2024-04-25T05:40:43.441309+0000","last_became_peered":"2024-04-25T05:40:43.441309+0000","last_unstale":"2024-04-25T05:42:12.173202+0000","last_undegraded":"2024-04-25T05:42:12.173202+0000","last_fullsized":"2024-04-25T05:42:12.173202+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"275'11726","reported_seq":11891,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.831758+0000","last_change":"2024-04-25T05:39:19.233374+0000","last_active":"2024-04-25T05:42:12.831758+0000","last_peered":"2024-04-25T05:42:12.831758+0000","last_clean":"2024-04-25T05:42:12.831758+0000","last_became_active":"2024-04-25T05:39:19.233055+0000","last_became_peered":"2024-04-25T05:39:19.233055+0000","last_unstale":"2024-04-25T05:42:12.831758+0000","last_undegraded":"2024-04-25T05:42:12.831758+0000","last_fullsized":"2024-04-25T05:42:12.831758+0000","mapping_epoch":198,"log_start":"239'7378","ondisk_log_start":"239'7378","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231866368,"num_objects":3538,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":488448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":639,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.211864+0000","last_change":"2024-04-25T05:40:58.922339+0000","last_active":"2024-04-25T05:42:12.211864+0000","last_peered":"2024-04-25T05:42:12.211864+0000","last_clean":"2024-04-25T05:42:12.211864+0000","last_became_active":"2024-04-25T05:40:58.921770+0000","last_became_peered":"2024-04-25T05:40:58.921770+0000","last_unstale":"2024-04-25T05:42:12.211864+0000","last_undegraded":"2024-04-25T05:42:12.211864+0000","last_fullsized":"2024-04-25T05:42:12.211864+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:07.411609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:07.411609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":426,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221919+0000","last_change":"2024-04-25T05:36:29.285525+0000","last_active":"2024-04-25T05:42:12.221919+0000","last_peered":"2024-04-25T05:42:12.221919+0000","last_clean":"2024-04-25T05:42:12.221919+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:42:12.221919+0000","last_undegraded":"2024-04-25T05:42:12.221919+0000","last_fullsized":"2024-04-25T05:42:12.221919+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:29.285241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:36:29.285241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":486,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214558+0000","last_change":"2024-04-25T05:39:29.827029+0000","last_active":"2024-04-25T05:42:12.214558+0000","last_peered":"2024-04-25T05:42:12.214558+0000","last_clean":"2024-04-25T05:42:12.214558+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:42:12.214558+0000","last_undegraded":"2024-04-25T05:42:12.214558+0000","last_fullsized":"2024-04-25T05:42:12.214558+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173111+0000","last_change":"2024-04-25T05:40:43.435305+0000","last_active":"2024-04-25T05:42:12.173111+0000","last_peered":"2024-04-25T05:42:12.173111+0000","last_clean":"2024-04-25T05:42:12.173111+0000","last_became_active":"2024-04-25T05:40:43.434803+0000","last_became_peered":"2024-04-25T05:40:43.434803+0000","last_unstale":"2024-04-25T05:42:12.173111+0000","last_undegraded":"2024-04-25T05:42:12.173111+0000","last_fullsized":"2024-04-25T05:42:12.173111+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.849716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.849716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"243'46","reported_seq":642,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174453+0000","last_change":"2024-04-25T05:40:56.036908+0000","last_active":"2024-04-25T05:42:12.174453+0000","last_peered":"2024-04-25T05:42:12.174453+0000","last_clean":"2024-04-25T05:42:12.174453+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:42:12.174453+0000","last_undegraded":"2024-04-25T05:42:12.174453+0000","last_fullsized":"2024-04-25T05:42:12.174453+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:39:42.723973+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.723973+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173056+0000","last_change":"2024-04-25T05:40:43.442114+0000","last_active":"2024-04-25T05:42:12.173056+0000","last_peered":"2024-04-25T05:42:12.173056+0000","last_clean":"2024-04-25T05:42:12.173056+0000","last_became_active":"2024-04-25T05:40:43.441816+0000","last_became_peered":"2024-04-25T05:40:43.441816+0000","last_unstale":"2024-04-25T05:42:12.173056+0000","last_undegraded":"2024-04-25T05:42:12.173056+0000","last_fullsized":"2024-04-25T05:42:12.173056+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"275'11715","reported_seq":11880,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.391792+0000","last_change":"2024-04-25T05:39:19.228876+0000","last_active":"2024-04-25T05:42:13.391792+0000","last_peered":"2024-04-25T05:42:13.391792+0000","last_clean":"2024-04-25T05:42:13.391792+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:42:13.391792+0000","last_undegraded":"2024-04-25T05:42:13.391792+0000","last_fullsized":"2024-04-25T05:42:13.391792+0000","mapping_epoch":198,"log_start":"239'7597","ondisk_log_start":"239'7597","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229965824,"num_objects":3509,"num_object_clones":0,"num_object_copies":10527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":761,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175812+0000","last_change":"2024-04-25T05:41:44.999875+0000","last_active":"2024-04-25T05:42:12.175812+0000","last_peered":"2024-04-25T05:42:12.175812+0000","last_clean":"2024-04-25T05:42:12.175812+0000","last_became_active":"2024-04-25T05:40:45.848289+0000","last_became_peered":"2024-04-25T05:40:45.848289+0000","last_unstale":"2024-04-25T05:42:12.175812+0000","last_undegraded":"2024-04-25T05:42:12.175812+0000","last_fullsized":"2024-04-25T05:42:12.175812+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.306297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.177262+0000","last_change":"2024-04-25T05:40:52.973425+0000","last_active":"2024-04-25T05:42:12.177262+0000","last_peered":"2024-04-25T05:42:12.177262+0000","last_clean":"2024-04-25T05:42:12.177262+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:42:12.177262+0000","last_undegraded":"2024-04-25T05:42:12.177262+0000","last_fullsized":"2024-04-25T05:42:12.177262+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:52.973302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.973302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":479,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171792+0000","last_change":"2024-04-25T05:40:55.966682+0000","last_active":"2024-04-25T05:42:12.171792+0000","last_peered":"2024-04-25T05:42:12.171792+0000","last_clean":"2024-04-25T05:42:12.171792+0000","last_became_active":"2024-04-25T05:40:55.966367+0000","last_became_peered":"2024-04-25T05:40:55.966367+0000","last_unstale":"2024-04-25T05:42:12.171792+0000","last_undegraded":"2024-04-25T05:42:12.171792+0000","last_fullsized":"2024-04-25T05:42:12.171792+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:42.474808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:42.474808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220289+0000","last_change":"2024-04-25T05:40:55.991779+0000","last_active":"2024-04-25T05:42:12.220289+0000","last_peered":"2024-04-25T05:42:12.220289+0000","last_clean":"2024-04-25T05:42:12.220289+0000","last_became_active":"2024-04-25T05:40:55.990929+0000","last_became_peered":"2024-04-25T05:40:55.990929+0000","last_unstale":"2024-04-25T05:42:12.220289+0000","last_undegraded":"2024-04-25T05:42:12.220289+0000","last_fullsized":"2024-04-25T05:42:12.220289+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.764632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.764632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":527,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172080+0000","last_change":"2024-04-25T05:40:43.477675+0000","last_active":"2024-04-25T05:42:12.172080+0000","last_peered":"2024-04-25T05:42:12.172080+0000","last_clean":"2024-04-25T05:42:12.172080+0000","last_became_active":"2024-04-25T05:40:43.450842+0000","last_became_peered":"2024-04-25T05:40:43.450842+0000","last_unstale":"2024-04-25T05:42:12.172080+0000","last_undegraded":"2024-04-25T05:42:12.172080+0000","last_fullsized":"2024-04-25T05:42:12.172080+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.592911+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":80,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.213847+0000","last_change":"2024-04-25T05:40:55.910414+0000","last_active":"2024-04-25T05:42:12.213847+0000","last_peered":"2024-04-25T05:42:12.213847+0000","last_clean":"2024-04-25T05:42:12.213847+0000","last_became_active":"2024-04-25T05:40:55.910074+0000","last_became_peered":"2024-04-25T05:40:55.910074+0000","last_unstale":"2024-04-25T05:42:12.213847+0000","last_undegraded":"2024-04-25T05:42:12.213847+0000","last_fullsized":"2024-04-25T05:42:12.213847+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.980299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.980299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"275'11821","reported_seq":26962,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.831282+0000","last_change":"2024-04-25T05:41:32.354405+0000","last_active":"2024-04-25T05:42:12.831282+0000","last_peered":"2024-04-25T05:42:12.831282+0000","last_clean":"2024-04-25T05:42:12.831282+0000","last_became_active":"2024-04-25T05:41:32.341701+0000","last_became_peered":"2024-04-25T05:41:32.341701+0000","last_unstale":"2024-04-25T05:42:12.831282+0000","last_undegraded":"2024-04-25T05:42:12.831282+0000","last_fullsized":"2024-04-25T05:42:12.831282+0000","mapping_epoch":258,"log_start":"235'7127","ondisk_log_start":"235'7127","created":198,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6983","last_scrub_stamp":"2024-04-25T05:40:41.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.157014+0000","objects_scrubbed":6915,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233897984,"num_objects":3569,"num_object_clones":0,"num_object_copies":10707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11821,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":487325696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":434,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212165+0000","last_change":"2024-04-25T05:39:09.353431+0000","last_active":"2024-04-25T05:42:12.212165+0000","last_peered":"2024-04-25T05:42:12.212165+0000","last_clean":"2024-04-25T05:42:12.212165+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:42:12.212165+0000","last_undegraded":"2024-04-25T05:42:12.212165+0000","last_fullsized":"2024-04-25T05:42:12.212165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.353310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":755,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175463+0000","last_change":"2024-04-25T05:40:45.848744+0000","last_active":"2024-04-25T05:42:12.175463+0000","last_peered":"2024-04-25T05:42:12.175463+0000","last_clean":"2024-04-25T05:42:12.175463+0000","last_became_active":"2024-04-25T05:40:45.847131+0000","last_became_peered":"2024-04-25T05:40:45.847131+0000","last_unstale":"2024-04-25T05:42:12.175463+0000","last_undegraded":"2024-04-25T05:42:12.175463+0000","last_fullsized":"2024-04-25T05:42:12.175463+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.104548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.104548+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222167+0000","last_change":"2024-04-25T05:42:08.356121+0000","last_active":"2024-04-25T05:42:12.222167+0000","last_peered":"2024-04-25T05:42:12.222167+0000","last_clean":"2024-04-25T05:42:12.222167+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:42:12.222167+0000","last_undegraded":"2024-04-25T05:42:12.222167+0000","last_fullsized":"2024-04-25T05:42:12.222167+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:10.802129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":212,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172197+0000","last_change":"2024-04-25T05:40:40.957968+0000","last_active":"2024-04-25T05:42:12.172197+0000","last_peered":"2024-04-25T05:42:12.172197+0000","last_clean":"2024-04-25T05:42:12.172197+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:42:12.172197+0000","last_undegraded":"2024-04-25T05:42:12.172197+0000","last_fullsized":"2024-04-25T05:42:12.172197+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:40:40.957665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.957665+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176534+0000","last_change":"2024-04-25T05:41:20.125777+0000","last_active":"2024-04-25T05:42:12.176534+0000","last_peered":"2024-04-25T05:42:12.176534+0000","last_clean":"2024-04-25T05:42:12.176534+0000","last_became_active":"2024-04-25T05:40:55.953691+0000","last_became_peered":"2024-04-25T05:40:55.953691+0000","last_unstale":"2024-04-25T05:42:12.176534+0000","last_undegraded":"2024-04-25T05:42:12.176534+0000","last_fullsized":"2024-04-25T05:42:12.176534+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:20.125501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:41:20.125501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.894302+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"275'12068","reported_seq":22615,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.350188+0000","last_change":"2024-04-25T05:42:07.903446+0000","last_active":"2024-04-25T05:42:13.350188+0000","last_peered":"2024-04-25T05:42:13.350188+0000","last_clean":"2024-04-25T05:42:13.350188+0000","last_became_active":"2024-04-25T05:42:07.871807+0000","last_became_peered":"2024-04-25T05:42:07.871807+0000","last_unstale":"2024-04-25T05:42:13.350188+0000","last_undegraded":"2024-04-25T05:42:13.350188+0000","last_fullsized":"2024-04-25T05:42:13.350188+0000","mapping_epoch":271,"log_start":"239'7733","ondisk_log_start":"239'7733","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7666","last_scrub_stamp":"2024-04-25T05:40:52.458924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.458924+0000","objects_scrubbed":7586,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:13.572852+0000","scrub_duration":2997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235012096,"num_objects":3586,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12068,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5139,"num_bytes_recovered":336789504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":628,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172081+0000","last_change":"2024-04-25T05:39:32.814133+0000","last_active":"2024-04-25T05:42:12.172081+0000","last_peered":"2024-04-25T05:42:12.172081+0000","last_clean":"2024-04-25T05:42:12.172081+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:42:12.172081+0000","last_undegraded":"2024-04-25T05:42:12.172081+0000","last_fullsized":"2024-04-25T05:42:12.172081+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.010323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.010323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171628+0000","last_change":"2024-04-25T05:40:47.469910+0000","last_active":"2024-04-25T05:42:12.171628+0000","last_peered":"2024-04-25T05:42:12.171628+0000","last_clean":"2024-04-25T05:42:12.171628+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:42:12.171628+0000","last_undegraded":"2024-04-25T05:42:12.171628+0000","last_fullsized":"2024-04-25T05:42:12.171628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":95,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174374+0000","last_change":"2024-04-25T05:40:43.435255+0000","last_active":"2024-04-25T05:42:12.174374+0000","last_peered":"2024-04-25T05:42:12.174374+0000","last_clean":"2024-04-25T05:42:12.174374+0000","last_became_active":"2024-04-25T05:40:43.434912+0000","last_became_peered":"2024-04-25T05:40:43.434912+0000","last_unstale":"2024-04-25T05:42:12.174374+0000","last_undegraded":"2024-04-25T05:42:12.174374+0000","last_fullsized":"2024-04-25T05:42:12.174374+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.785355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.785355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":973,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175838+0000","last_change":"2024-04-25T05:41:36.018017+0000","last_active":"2024-04-25T05:42:12.175838+0000","last_peered":"2024-04-25T05:42:12.175838+0000","last_clean":"2024-04-25T05:42:12.175838+0000","last_became_active":"2024-04-25T05:40:43.431415+0000","last_became_peered":"2024-04-25T05:40:43.431415+0000","last_unstale":"2024-04-25T05:42:12.175838+0000","last_undegraded":"2024-04-25T05:42:12.175838+0000","last_fullsized":"2024-04-25T05:42:12.175838+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:41:36.017857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.017857+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:39.489209+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":481,"num_read_kb":476,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":280,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212815+0000","last_change":"2024-04-25T05:40:12.370708+0000","last_active":"2024-04-25T05:42:12.212815+0000","last_peered":"2024-04-25T05:42:12.212815+0000","last_clean":"2024-04-25T05:42:12.212815+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:42:12.212815+0000","last_undegraded":"2024-04-25T05:42:12.212815+0000","last_fullsized":"2024-04-25T05:42:12.212815+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212624+0000","last_change":"2024-04-25T05:40:55.418588+0000","last_active":"2024-04-25T05:42:12.212624+0000","last_peered":"2024-04-25T05:42:12.212624+0000","last_clean":"2024-04-25T05:42:12.212624+0000","last_became_active":"2024-04-25T05:40:43.454300+0000","last_became_peered":"2024-04-25T05:40:43.454300+0000","last_unstale":"2024-04-25T05:42:12.212624+0000","last_undegraded":"2024-04-25T05:42:12.212624+0000","last_fullsized":"2024-04-25T05:42:12.212624+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:55.418499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.418499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"275'12201","reported_seq":25370,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.400771+0000","last_change":"2024-04-25T05:41:48.020080+0000","last_active":"2024-04-25T05:42:13.400771+0000","last_peered":"2024-04-25T05:42:13.400771+0000","last_clean":"2024-04-25T05:42:13.400771+0000","last_became_active":"2024-04-25T05:41:47.997545+0000","last_became_peered":"2024-04-25T05:41:47.997545+0000","last_unstale":"2024-04-25T05:42:13.400771+0000","last_undegraded":"2024-04-25T05:42:13.400771+0000","last_fullsized":"2024-04-25T05:42:13.400771+0000","mapping_epoch":262,"log_start":"244'7992","ondisk_log_start":"244'7992","created":198,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237830144,"num_objects":3629,"num_object_clones":0,"num_object_copies":10887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12201,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6494,"num_bytes_recovered":425590784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":634,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172288+0000","last_change":"2024-04-25T05:40:45.845890+0000","last_active":"2024-04-25T05:42:12.172288+0000","last_peered":"2024-04-25T05:42:12.172288+0000","last_clean":"2024-04-25T05:42:12.172288+0000","last_became_active":"2024-04-25T05:40:45.845535+0000","last_became_peered":"2024-04-25T05:40:45.845535+0000","last_unstale":"2024-04-25T05:42:12.172288+0000","last_undegraded":"2024-04-25T05:42:12.172288+0000","last_fullsized":"2024-04-25T05:42:12.172288+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:02.832193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.832193+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171588+0000","last_change":"2024-04-25T05:40:55.966981+0000","last_active":"2024-04-25T05:42:12.171588+0000","last_peered":"2024-04-25T05:42:12.171588+0000","last_clean":"2024-04-25T05:42:12.171588+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:42:12.171588+0000","last_undegraded":"2024-04-25T05:42:12.171588+0000","last_fullsized":"2024-04-25T05:42:12.171588+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:30.760468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.760468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174283+0000","last_change":"2024-04-25T05:40:55.977907+0000","last_active":"2024-04-25T05:42:12.174283+0000","last_peered":"2024-04-25T05:42:12.174283+0000","last_clean":"2024-04-25T05:42:12.174283+0000","last_became_active":"2024-04-25T05:40:55.977574+0000","last_became_peered":"2024-04-25T05:40:55.977574+0000","last_unstale":"2024-04-25T05:42:12.174283+0000","last_undegraded":"2024-04-25T05:42:12.174283+0000","last_fullsized":"2024-04-25T05:42:12.174283+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:16.651620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:16.651620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":115,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172916+0000","last_change":"2024-04-25T05:40:43.491684+0000","last_active":"2024-04-25T05:42:12.172916+0000","last_peered":"2024-04-25T05:42:12.172916+0000","last_clean":"2024-04-25T05:42:12.172916+0000","last_became_active":"2024-04-25T05:40:43.446526+0000","last_became_peered":"2024-04-25T05:40:43.446526+0000","last_unstale":"2024-04-25T05:42:12.172916+0000","last_undegraded":"2024-04-25T05:42:12.172916+0000","last_fullsized":"2024-04-25T05:42:12.172916+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:38:22.404901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:38:22.404901+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":168,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171947+0000","last_change":"2024-04-25T05:40:43.429997+0000","last_active":"2024-04-25T05:42:12.171947+0000","last_peered":"2024-04-25T05:42:12.171947+0000","last_clean":"2024-04-25T05:42:12.171947+0000","last_became_active":"2024-04-25T05:40:43.429667+0000","last_became_peered":"2024-04-25T05:40:43.429667+0000","last_unstale":"2024-04-25T05:42:12.171947+0000","last_undegraded":"2024-04-25T05:42:12.171947+0000","last_fullsized":"2024-04-25T05:42:12.171947+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.495957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.495957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"275'11722","reported_seq":20825,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.267510+0000","last_change":"2024-04-25T05:42:12.209425+0000","last_active":"2024-04-25T05:42:13.267510+0000","last_peered":"2024-04-25T05:42:13.267510+0000","last_clean":"2024-04-25T05:42:13.267510+0000","last_became_active":"2024-04-25T05:42:12.177948+0000","last_became_peered":"2024-04-25T05:42:12.177948+0000","last_unstale":"2024-04-25T05:42:13.267510+0000","last_undegraded":"2024-04-25T05:42:13.267510+0000","last_fullsized":"2024-04-25T05:42:13.267510+0000","mapping_epoch":274,"log_start":"231'6939","ondisk_log_start":"231'6939","created":198,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":999,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235798528,"num_objects":3598,"num_object_clones":0,"num_object_copies":10794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":490240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4450,"num_bytes_recovered":291635200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":625,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172475+0000","last_change":"2024-04-25T05:41:31.093059+0000","last_active":"2024-04-25T05:42:12.172475+0000","last_peered":"2024-04-25T05:42:12.172475+0000","last_clean":"2024-04-25T05:42:12.172475+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:42:12.172475+0000","last_undegraded":"2024-04-25T05:42:12.172475+0000","last_fullsized":"2024-04-25T05:42:12.172475+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:31.092923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.092923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.069935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":101,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172091+0000","last_change":"2024-04-25T05:40:55.970443+0000","last_active":"2024-04-25T05:42:12.172091+0000","last_peered":"2024-04-25T05:42:12.172091+0000","last_clean":"2024-04-25T05:42:12.172091+0000","last_became_active":"2024-04-25T05:40:55.970110+0000","last_became_peered":"2024-04-25T05:40:55.970110+0000","last_unstale":"2024-04-25T05:42:12.172091+0000","last_undegraded":"2024-04-25T05:42:12.172091+0000","last_fullsized":"2024-04-25T05:42:12.172091+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:35:44.317344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:35:44.317344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214677+0000","last_change":"2024-04-25T05:40:55.975334+0000","last_active":"2024-04-25T05:42:12.214677+0000","last_peered":"2024-04-25T05:42:12.214677+0000","last_clean":"2024-04-25T05:42:12.214677+0000","last_became_active":"2024-04-25T05:40:55.974871+0000","last_became_peered":"2024-04-25T05:40:55.974871+0000","last_unstale":"2024-04-25T05:42:12.214677+0000","last_undegraded":"2024-04-25T05:42:12.214677+0000","last_fullsized":"2024-04-25T05:42:12.214677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":93,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172717+0000","last_change":"2024-04-25T05:40:56.020340+0000","last_active":"2024-04-25T05:42:12.172717+0000","last_peered":"2024-04-25T05:42:12.172717+0000","last_clean":"2024-04-25T05:42:12.172717+0000","last_became_active":"2024-04-25T05:40:55.964721+0000","last_became_peered":"2024-04-25T05:40:55.964721+0000","last_unstale":"2024-04-25T05:42:12.172717+0000","last_undegraded":"2024-04-25T05:42:12.172717+0000","last_fullsized":"2024-04-25T05:42:12.172717+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:39:09.887340+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.887340+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"233'12","reported_seq":117,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172787+0000","last_change":"2024-04-25T05:40:43.475139+0000","last_active":"2024-04-25T05:42:12.172787+0000","last_peered":"2024-04-25T05:42:12.172787+0000","last_clean":"2024-04-25T05:42:12.172787+0000","last_became_active":"2024-04-25T05:40:43.445833+0000","last_became_peered":"2024-04-25T05:40:43.445833+0000","last_unstale":"2024-04-25T05:42:12.172787+0000","last_undegraded":"2024-04-25T05:42:12.172787+0000","last_fullsized":"2024-04-25T05:42:12.172787+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:40:06.822122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.822122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"275'11946","reported_seq":26185,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.852535+0000","last_change":"2024-04-25T05:41:28.231423+0000","last_active":"2024-04-25T05:42:12.852535+0000","last_peered":"2024-04-25T05:42:12.852535+0000","last_clean":"2024-04-25T05:42:12.852535+0000","last_became_active":"2024-04-25T05:41:28.213582+0000","last_became_peered":"2024-04-25T05:41:28.213582+0000","last_unstale":"2024-04-25T05:42:12.852535+0000","last_undegraded":"2024-04-25T05:42:12.852535+0000","last_fullsized":"2024-04-25T05:42:12.852535+0000","mapping_epoch":255,"log_start":"239'7529","ondisk_log_start":"239'7529","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237436952,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11946,"num_write_kb":498177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7030,"num_bytes_recovered":460652568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":613,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220807+0000","last_change":"2024-04-25T05:38:58.078848+0000","last_active":"2024-04-25T05:42:12.220807+0000","last_peered":"2024-04-25T05:42:12.220807+0000","last_clean":"2024-04-25T05:42:12.220807+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:42:12.220807+0000","last_undegraded":"2024-04-25T05:42:12.220807+0000","last_fullsized":"2024-04-25T05:42:12.220807+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:58.078483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:58.078483+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":79,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176119+0000","last_change":"2024-04-25T05:40:55.981504+0000","last_active":"2024-04-25T05:42:12.176119+0000","last_peered":"2024-04-25T05:42:12.176119+0000","last_clean":"2024-04-25T05:42:12.176119+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:42:12.176119+0000","last_undegraded":"2024-04-25T05:42:12.176119+0000","last_fullsized":"2024-04-25T05:42:12.176119+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.781719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.781719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":99,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.173812+0000","last_change":"2024-04-25T05:40:43.463116+0000","last_active":"2024-04-25T05:42:12.173812+0000","last_peered":"2024-04-25T05:42:12.173812+0000","last_clean":"2024-04-25T05:42:12.173812+0000","last_became_active":"2024-04-25T05:40:43.443728+0000","last_became_peered":"2024-04-25T05:40:43.443728+0000","last_unstale":"2024-04-25T05:42:12.173812+0000","last_undegraded":"2024-04-25T05:42:12.173812+0000","last_fullsized":"2024-04-25T05:42:12.173812+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":176,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212048+0000","last_change":"2024-04-25T05:40:48.391768+0000","last_active":"2024-04-25T05:42:12.212048+0000","last_peered":"2024-04-25T05:42:12.212048+0000","last_clean":"2024-04-25T05:42:12.212048+0000","last_became_active":"2024-04-25T05:40:09.733866+0000","last_became_peered":"2024-04-25T05:40:09.733866+0000","last_unstale":"2024-04-25T05:42:12.212048+0000","last_undegraded":"2024-04-25T05:42:12.212048+0000","last_fullsized":"2024-04-25T05:42:12.212048+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:40:48.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.391383+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":546,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220639+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:42:12.220639+0000","last_peered":"2024-04-25T05:42:12.220639+0000","last_clean":"2024-04-25T05:42:12.220639+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:42:12.220639+0000","last_undegraded":"2024-04-25T05:42:12.220639+0000","last_fullsized":"2024-04-25T05:42:12.220639+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:17.034593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"275'11942","reported_seq":22134,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.255907+0000","last_change":"2024-04-25T05:42:06.908338+0000","last_active":"2024-04-25T05:42:13.255907+0000","last_peered":"2024-04-25T05:42:13.255907+0000","last_clean":"2024-04-25T05:42:13.255907+0000","last_became_active":"2024-04-25T05:42:06.870659+0000","last_became_peered":"2024-04-25T05:42:06.870659+0000","last_unstale":"2024-04-25T05:42:13.255907+0000","last_undegraded":"2024-04-25T05:42:13.255907+0000","last_fullsized":"2024-04-25T05:42:13.255907+0000","mapping_epoch":270,"log_start":"237'7368","ondisk_log_start":"237'7368","created":198,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237633536,"num_objects":3626,"num_object_clones":0,"num_object_copies":10878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11942,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5005,"num_bytes_recovered":328007680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":639,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176031+0000","last_change":"2024-04-25T05:40:58.920790+0000","last_active":"2024-04-25T05:42:12.176031+0000","last_peered":"2024-04-25T05:42:12.176031+0000","last_clean":"2024-04-25T05:42:12.176031+0000","last_became_active":"2024-04-25T05:40:58.920400+0000","last_became_peered":"2024-04-25T05:40:58.920400+0000","last_unstale":"2024-04-25T05:42:12.176031+0000","last_undegraded":"2024-04-25T05:42:12.176031+0000","last_fullsized":"2024-04-25T05:42:12.176031+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:37.793923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.793923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":122,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222395+0000","last_change":"2024-04-25T05:40:55.975104+0000","last_active":"2024-04-25T05:42:12.222395+0000","last_peered":"2024-04-25T05:42:12.222395+0000","last_clean":"2024-04-25T05:42:12.222395+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:42:12.222395+0000","last_undegraded":"2024-04-25T05:42:12.222395+0000","last_fullsized":"2024-04-25T05:42:12.222395+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.719427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.719427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.222165+0000","last_change":"2024-04-25T05:40:43.516969+0000","last_active":"2024-04-25T05:42:12.222165+0000","last_peered":"2024-04-25T05:42:12.222165+0000","last_clean":"2024-04-25T05:42:12.222165+0000","last_became_active":"2024-04-25T05:40:43.440253+0000","last_became_peered":"2024-04-25T05:40:43.440253+0000","last_unstale":"2024-04-25T05:42:12.222165+0000","last_undegraded":"2024-04-25T05:42:12.222165+0000","last_fullsized":"2024-04-25T05:42:12.222165+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:43.516897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.516897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":555,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220526+0000","last_change":"2024-04-25T05:40:48.519454+0000","last_active":"2024-04-25T05:42:12.220526+0000","last_peered":"2024-04-25T05:42:12.220526+0000","last_clean":"2024-04-25T05:42:12.220526+0000","last_became_active":"2024-04-25T05:40:43.440689+0000","last_became_peered":"2024-04-25T05:40:43.440689+0000","last_unstale":"2024-04-25T05:42:12.220526+0000","last_undegraded":"2024-04-25T05:42:12.220526+0000","last_fullsized":"2024-04-25T05:42:12.220526+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:40:48.519288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.519288+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":437,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175659+0000","last_change":"2024-04-25T05:40:08.742282+0000","last_active":"2024-04-25T05:42:12.175659+0000","last_peered":"2024-04-25T05:42:12.175659+0000","last_clean":"2024-04-25T05:42:12.175659+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:42:12.175659+0000","last_undegraded":"2024-04-25T05:42:12.175659+0000","last_fullsized":"2024-04-25T05:42:12.175659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:08.742146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.742146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"275'11761","reported_seq":12674,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.365639+0000","last_change":"2024-04-25T05:39:34.781974+0000","last_active":"2024-04-25T05:42:13.365639+0000","last_peered":"2024-04-25T05:42:13.365639+0000","last_clean":"2024-04-25T05:42:13.365639+0000","last_became_active":"2024-04-25T05:39:34.780422+0000","last_became_peered":"2024-04-25T05:39:34.780422+0000","last_unstale":"2024-04-25T05:42:13.365639+0000","last_undegraded":"2024-04-25T05:42:13.365639+0000","last_fullsized":"2024-04-25T05:42:13.365639+0000","mapping_epoch":208,"log_start":"239'7469","ondisk_log_start":"239'7469","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233504768,"num_objects":3563,"num_object_clones":0,"num_object_copies":10689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":490368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":24117248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":631,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214103+0000","last_change":"2024-04-25T05:40:58.924826+0000","last_active":"2024-04-25T05:42:12.214103+0000","last_peered":"2024-04-25T05:42:12.214103+0000","last_clean":"2024-04-25T05:42:12.214103+0000","last_became_active":"2024-04-25T05:40:58.923753+0000","last_became_peered":"2024-04-25T05:40:58.923753+0000","last_unstale":"2024-04-25T05:42:12.214103+0000","last_undegraded":"2024-04-25T05:42:12.214103+0000","last_fullsized":"2024-04-25T05:42:12.214103+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.214231+0000","last_change":"2024-04-25T05:40:50.460093+0000","last_active":"2024-04-25T05:42:12.214231+0000","last_peered":"2024-04-25T05:42:12.214231+0000","last_clean":"2024-04-25T05:42:12.214231+0000","last_became_active":"2024-04-25T05:40:43.454117+0000","last_became_peered":"2024-04-25T05:40:43.454117+0000","last_unstale":"2024-04-25T05:42:12.214231+0000","last_undegraded":"2024-04-25T05:42:12.214231+0000","last_fullsized":"2024-04-25T05:42:12.214231+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"233'11","reported_seq":184,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.213731+0000","last_change":"2024-04-25T05:41:38.552749+0000","last_active":"2024-04-25T05:42:12.213731+0000","last_peered":"2024-04-25T05:42:12.213731+0000","last_clean":"2024-04-25T05:42:12.213731+0000","last_became_active":"2024-04-25T05:40:43.453268+0000","last_became_peered":"2024-04-25T05:40:43.453268+0000","last_unstale":"2024-04-25T05:42:12.213731+0000","last_undegraded":"2024-04-25T05:42:12.213731+0000","last_fullsized":"2024-04-25T05:42:12.213731+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:52.068035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":446,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220411+0000","last_change":"2024-04-25T05:40:51.592334+0000","last_active":"2024-04-25T05:42:12.220411+0000","last_peered":"2024-04-25T05:42:12.220411+0000","last_clean":"2024-04-25T05:42:12.220411+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:42:12.220411+0000","last_undegraded":"2024-04-25T05:42:12.220411+0000","last_fullsized":"2024-04-25T05:42:12.220411+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.592111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.592111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172444+0000","last_change":"2024-04-25T05:39:39.713342+0000","last_active":"2024-04-25T05:42:12.172444+0000","last_peered":"2024-04-25T05:42:12.172444+0000","last_clean":"2024-04-25T05:42:12.172444+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:42:12.172444+0000","last_undegraded":"2024-04-25T05:42:12.172444+0000","last_fullsized":"2024-04-25T05:42:12.172444+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.713165+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"275'11790","reported_seq":26318,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.816746+0000","last_change":"2024-04-25T05:41:28.229751+0000","last_active":"2024-04-25T05:42:12.816746+0000","last_peered":"2024-04-25T05:42:12.816746+0000","last_clean":"2024-04-25T05:42:12.816746+0000","last_became_active":"2024-04-25T05:41:28.214541+0000","last_became_peered":"2024-04-25T05:41:28.214541+0000","last_unstale":"2024-04-25T05:42:12.816746+0000","last_undegraded":"2024-04-25T05:42:12.816746+0000","last_fullsized":"2024-04-25T05:42:12.816746+0000","mapping_epoch":255,"log_start":"239'7418","ondisk_log_start":"239'7418","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6914","last_scrub_stamp":"2024-04-25T05:40:40.935916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.935916+0000","objects_scrubbed":6834,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238026752,"num_objects":3632,"num_object_clones":0,"num_object_copies":10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7133,"num_bytes_recovered":467468288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":438,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221359+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:42:12.221359+0000","last_peered":"2024-04-25T05:42:12.221359+0000","last_clean":"2024-04-25T05:42:12.221359+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:42:12.221359+0000","last_undegraded":"2024-04-25T05:42:12.221359+0000","last_fullsized":"2024-04-25T05:42:12.221359+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":458,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212523+0000","last_change":"2024-04-25T05:41:27.501946+0000","last_active":"2024-04-25T05:42:12.212523+0000","last_peered":"2024-04-25T05:42:12.212523+0000","last_clean":"2024-04-25T05:42:12.212523+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:42:12.212523+0000","last_undegraded":"2024-04-25T05:42:12.212523+0000","last_fullsized":"2024-04-25T05:42:12.212523+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.501828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.501828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.447143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":459,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175802+0000","last_change":"2024-04-25T05:40:55.964874+0000","last_active":"2024-04-25T05:42:12.175802+0000","last_peered":"2024-04-25T05:42:12.175802+0000","last_clean":"2024-04-25T05:42:12.175802+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:42:12.175802+0000","last_undegraded":"2024-04-25T05:42:12.175802+0000","last_fullsized":"2024-04-25T05:42:12.175802+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:39.870293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:39.870293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.172207+0000","last_change":"2024-04-25T05:40:09.839273+0000","last_active":"2024-04-25T05:42:12.172207+0000","last_peered":"2024-04-25T05:42:12.172207+0000","last_clean":"2024-04-25T05:42:12.172207+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:42:12.172207+0000","last_undegraded":"2024-04-25T05:42:12.172207+0000","last_fullsized":"2024-04-25T05:42:12.172207+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:09.838661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.838661+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"275'12217","reported_seq":22913,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.370550+0000","last_change":"2024-04-25T05:42:07.900284+0000","last_active":"2024-04-25T05:42:13.370550+0000","last_peered":"2024-04-25T05:42:13.370550+0000","last_clean":"2024-04-25T05:42:13.370550+0000","last_became_active":"2024-04-25T05:42:07.869166+0000","last_became_peered":"2024-04-25T05:42:07.869166+0000","last_unstale":"2024-04-25T05:42:13.370550+0000","last_undegraded":"2024-04-25T05:42:13.370550+0000","last_fullsized":"2024-04-25T05:42:13.370550+0000","mapping_epoch":271,"log_start":"239'7852","ondisk_log_start":"239'7852","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5255,"num_bytes_recovered":344391680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":626,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.212645+0000","last_change":"2024-04-25T05:40:58.929257+0000","last_active":"2024-04-25T05:42:12.212645+0000","last_peered":"2024-04-25T05:42:12.212645+0000","last_clean":"2024-04-25T05:42:12.212645+0000","last_became_active":"2024-04-25T05:40:58.926534+0000","last_became_peered":"2024-04-25T05:40:58.926534+0000","last_unstale":"2024-04-25T05:42:12.212645+0000","last_undegraded":"2024-04-25T05:42:12.212645+0000","last_fullsized":"2024-04-25T05:42:12.212645+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":81,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221232+0000","last_change":"2024-04-25T05:40:55.974556+0000","last_active":"2024-04-25T05:42:12.221232+0000","last_peered":"2024-04-25T05:42:12.221232+0000","last_clean":"2024-04-25T05:42:12.221232+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:42:12.221232+0000","last_undegraded":"2024-04-25T05:42:12.221232+0000","last_fullsized":"2024-04-25T05:42:12.221232+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.019675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.019675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174663+0000","last_change":"2024-04-25T05:40:44.537276+0000","last_active":"2024-04-25T05:42:12.174663+0000","last_peered":"2024-04-25T05:42:12.174663+0000","last_clean":"2024-04-25T05:42:12.174663+0000","last_became_active":"2024-04-25T05:40:43.449807+0000","last_became_peered":"2024-04-25T05:40:43.449807+0000","last_unstale":"2024-04-25T05:42:12.174663+0000","last_undegraded":"2024-04-25T05:42:12.174663+0000","last_fullsized":"2024-04-25T05:42:12.174663+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:12.180641+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":162,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171789+0000","last_change":"2024-04-25T05:39:37.518793+0000","last_active":"2024-04-25T05:42:12.171789+0000","last_peered":"2024-04-25T05:42:12.171789+0000","last_clean":"2024-04-25T05:42:12.171789+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:42:12.171789+0000","last_undegraded":"2024-04-25T05:42:12.171789+0000","last_fullsized":"2024-04-25T05:42:12.171789+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.518653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.518653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":145,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.174372+0000","last_change":"2024-04-25T05:40:56.017339+0000","last_active":"2024-04-25T05:42:12.174372+0000","last_peered":"2024-04-25T05:42:12.174372+0000","last_clean":"2024-04-25T05:42:12.174372+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:42:12.174372+0000","last_undegraded":"2024-04-25T05:42:12.174372+0000","last_fullsized":"2024-04-25T05:42:12.174372+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:34.605704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.605704+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"275'11837","reported_seq":12002,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:13.382888+0000","last_change":"2024-04-25T05:39:19.231808+0000","last_active":"2024-04-25T05:42:13.382888+0000","last_peered":"2024-04-25T05:42:13.382888+0000","last_clean":"2024-04-25T05:42:13.382888+0000","last_became_active":"2024-04-25T05:39:19.231525+0000","last_became_peered":"2024-04-25T05:39:19.231525+0000","last_unstale":"2024-04-25T05:42:13.382888+0000","last_undegraded":"2024-04-25T05:42:13.382888+0000","last_fullsized":"2024-04-25T05:42:13.382888+0000","mapping_epoch":198,"log_start":"239'7651","ondisk_log_start":"239'7651","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229572608,"num_objects":3503,"num_object_clones":0,"num_object_copies":10509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":490880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":753,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.175386+0000","last_change":"2024-04-25T05:40:58.930476+0000","last_active":"2024-04-25T05:42:12.175386+0000","last_peered":"2024-04-25T05:42:12.175386+0000","last_clean":"2024-04-25T05:42:12.175386+0000","last_became_active":"2024-04-25T05:40:58.930089+0000","last_became_peered":"2024-04-25T05:40:58.930089+0000","last_unstale":"2024-04-25T05:42:12.175386+0000","last_undegraded":"2024-04-25T05:42:12.175386+0000","last_fullsized":"2024-04-25T05:42:12.175386+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:39:01.087145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:39:01.087145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":122,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.171912+0000","last_change":"2024-04-25T05:40:55.964673+0000","last_active":"2024-04-25T05:42:12.171912+0000","last_peered":"2024-04-25T05:42:12.171912+0000","last_clean":"2024-04-25T05:42:12.171912+0000","last_became_active":"2024-04-25T05:40:55.964302+0000","last_became_peered":"2024-04-25T05:40:55.964302+0000","last_unstale":"2024-04-25T05:42:12.171912+0000","last_undegraded":"2024-04-25T05:42:12.171912+0000","last_fullsized":"2024-04-25T05:42:12.171912+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:02.813078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.813078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"243'10","reported_seq":197,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.176972+0000","last_change":"2024-04-25T05:40:55.966632+0000","last_active":"2024-04-25T05:42:12.176972+0000","last_peered":"2024-04-25T05:42:12.176972+0000","last_clean":"2024-04-25T05:42:12.176972+0000","last_became_active":"2024-04-25T05:40:55.955721+0000","last_became_peered":"2024-04-25T05:40:55.955721+0000","last_unstale":"2024-04-25T05:42:12.176972+0000","last_undegraded":"2024-04-25T05:42:12.176972+0000","last_fullsized":"2024-04-25T05:42:12.176972+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-04-25T05:39:33.738307+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.738307+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.220507+0000","last_change":"2024-04-25T05:40:43.433407+0000","last_active":"2024-04-25T05:42:12.220507+0000","last_peered":"2024-04-25T05:42:12.220507+0000","last_clean":"2024-04-25T05:42:12.220507+0000","last_became_active":"2024-04-25T05:40:43.432950+0000","last_became_peered":"2024-04-25T05:40:43.432950+0000","last_unstale":"2024-04-25T05:42:12.220507+0000","last_undegraded":"2024-04-25T05:42:12.220507+0000","last_fullsized":"2024-04-25T05:42:12.220507+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:36:54.712385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:36:54.712385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":150,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-04-25T05:42:12.221306+0000","last_change":"2024-04-25T05:40:09.731382+0000","last_active":"2024-04-25T05:42:12.221306+0000","last_peered":"2024-04-25T05:42:12.221306+0000","last_clean":"2024-04-25T05:42:12.221306+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:42:12.221306+0000","last_undegraded":"2024-04-25T05:42:12.221306+0000","last_fullsized":"2024-04-25T05:42:12.221306+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:36:30.640848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:36:30.640848+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3776184344,"num_objects":57621,"num_object_clones":0,"num_object_copies":172863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190623,"num_write_kb":7943745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63646,"num_bytes_recovered":4170973208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12680310784,"data_stored":12680298568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70903,"ondisk_log_size":70903,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":481,"num_read_kb":476,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":974,"num_read_kb":699,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":5396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":652,"ondisk_log_size":652,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070568,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730552,"kb_used_data":1420332,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92014600,"statfs":{"total":95995035648,"available":94222950400,"internally_reserved":0,"allocated":1454419968,"data_stored":1453883931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317651402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364908,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1979972,"kb_used_data":1655864,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91765180,"statfs":{"total":95995035648,"available":93967544320,"internally_reserved":0,"allocated":1695604736,"data_stored":1695063193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1989404,"kb_used_data":1615312,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91755748,"statfs":{"total":95995035648,"available":93957885952,"internally_reserved":0,"allocated":1654079488,"data_stored":1653512579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1805312,"kb_used_data":1409012,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91939840,"statfs":{"total":95995035648,"available":94146396160,"internally_reserved":0,"allocated":1442828288,"data_stored":1442289390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2615824,"kb_used_data":2187524,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91129328,"statfs":{"total":95995035648,"available":93316431872,"internally_reserved":0,"allocated":2240024576,"data_stored":2239470131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":438499783},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986293,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729012,"kb_used_data":1427116,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92016140,"statfs":{"total":95995035648,"available":94224527360,"internally_reserved":0,"allocated":1461366784,"data_stored":1460829807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691953,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2687600,"kb_used_data":2367648,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91057552,"statfs":{"total":95995035648,"available":93242933248,"internally_reserved":0,"allocated":2424471552,"data_stored":2423946835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1418657792,"data_stored":1418657792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1636433920,"data_stored":1636433920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":463273984,"data_stored":463273984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2235629568,"data_stored":2235629568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1439236096,"data_stored":1439236096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2388004864,"data_stored":2388000792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1668222976,"data_stored":1668218904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1430851584,"data_stored":1430847512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:42:21.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:42:21.191 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:21.196 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:21.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:21.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:21.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:21.393 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:21.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:21.530 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:21.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:21.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:21.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:21.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:21.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 8.3 scrub starts 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 6.13 scrub starts 2024-04-25T05:42:21.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[27010]: 6.13 scrub ok 2024-04-25T05:42:21.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:42:21.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:42:21.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 8.3 scrub starts 2024-04-25T05:42:21.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:42:21.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:42:21.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 6.13 scrub starts 2024-04-25T05:42:21.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:21 smithi043 ceph-mon[37492]: 6.13 scrub ok 2024-04-25T05:42:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:21.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:21.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 8.3 scrub starts 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 6.13 scrub starts 2024-04-25T05:42:21.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:21 smithi060 ceph-mon[30898]: 6.13 scrub ok 2024-04-25T05:42:22.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:22.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:22.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:22.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:22.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:22.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.469 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:22.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:22.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:22.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:22.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.742 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:22.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:22.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:22.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:22.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: 8.3 scrub ok 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: 4.16 scrub starts 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: 4.16 scrub ok 2024-04-25T05:42:22.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[27010]: pgmap v437: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 835 op/s 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: 8.3 scrub ok 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: 4.16 scrub starts 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: 4.16 scrub ok 2024-04-25T05:42:22.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:22 smithi043 ceph-mon[37492]: pgmap v437: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 835 op/s 2024-04-25T05:42:22.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:42:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:42:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: 8.3 scrub ok 2024-04-25T05:42:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: 4.16 scrub starts 2024-04-25T05:42:22.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: 4.16 scrub ok 2024-04-25T05:42:23.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:22 smithi060 ceph-mon[30898]: pgmap v437: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 835 op/s 2024-04-25T05:42:23.002 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:23.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:23.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:23.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:23.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:23.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:23.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:23.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:23.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:23.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:42:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:42:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:42:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:42:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:42:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:42:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:42:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:42:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:23 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:23.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:23.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:23.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.826 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:23.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:23.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:23.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:23.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:42:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:42:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:42:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:42:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:23 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:24.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:24.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:24.127 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:24.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:24.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:24.204 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:24.230 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:24.251 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:24.293 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:24.330 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:24.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:24.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:24.379 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:24.421 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:24.426 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:24.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:24.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:24.605 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:42:24.607 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:42:24.607 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:205586 8.b e278/198 8:d2243558:::benchmark_data_smithi043_104001_object81462:head v 278'12623, mlcod=277'12620)", 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:24.604224+0000", 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0024246099999999998, 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002455381, 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604224+0000", 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604224+0000", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604225+0000", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8700000000000005e-07 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604232+0000", 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0639999999999996e-06 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.610 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:24.610 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604241+0000", 2024-04-25T05:42:24.610 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2350000000000005e-06 2024-04-25T05:42:24.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604248+0000", 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1729999999999998e-06 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605242+0000", 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00099382099999999998 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.612 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605260+0000", 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8240000000000002e-05 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:205592 8.1 8:8e17a4f6:::benchmark_data_smithi043_104001_object81468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:24.604611+0000", 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020376349999999999, 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021428770000000001, 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:42:24.613 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 205592 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604611+0000", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604611+0000", 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604612+0000", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0979999999999999e-06 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604616+0000", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.839e-06 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604624+0000", 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6920000000000002e-06 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604631+0000", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0910000000000004e-06 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604871+0000", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000240249 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604986+0000", 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011542799999999999 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605089+0000", 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000102382 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:205593 8.3 8:c513f914:::benchmark_data_smithi043_104001_object81469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:24.604646+0000", 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0020027679999999998, 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002186618, 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:42:24.617 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 205593 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604646+0000", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604646+0000", 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604646+0000", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.34e-07 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604649+0000", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.762e-06 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604655+0000", 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.964e-06 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.604661+0000", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8050000000000003e-06 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.606222+0000", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001560709 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.606340+0000", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118688 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-04-25T05:42:24.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.606453+0000", 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112791 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:205598 8.1 8.33827761 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:24.605694+0000", 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00095468699999999996, 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012173030000000001, 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:42:24.621 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 205598 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605694+0000", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605694+0000", 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605695+0000", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1519999999999999e-06 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605701+0000", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7300000000000002e-06 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605715+0000", 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3817e-05 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605723+0000", 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.49e-06 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.624 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:205589 8.3 8:c58de18a:::benchmark_data_smithi043_104001_object81465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-04-25T05:42:24.625 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:24.601069+0000", 2024-04-25T05:42:24.625 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0055796550000000002, 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059018109999999999, 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 205589 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:42:24.626 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601069+0000", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601069+0000", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601070+0000", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6700000000000002e-07 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601074+0000", 2024-04-25T05:42:24.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.014e-06 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601089+0000", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5056e-05 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601098+0000", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7539999999999995e-06 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601156+0000", 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8068e-05 2024-04-25T05:42:24.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601281+0000", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124881 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 5,6", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.601401+0000", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120265 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.605213+0000", 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038117699999999999 2024-04-25T05:42:24.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:24.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:24.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:24.606188+0000", 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000975189 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 5 2024-04-25T05:42:24.631 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:42:24.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:24.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:24.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:24.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:24.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:42:24.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:42:24.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: pgmap v438: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:24.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/899276270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3895357090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: pgmap v438: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/899276270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.894 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:24 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3895357090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:24.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:24.942 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:24.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: pgmap v438: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/899276270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:42:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:24 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3895357090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:25.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:25.082 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:25.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:25.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:25.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:25.342 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:25.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:25.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:25.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:25.611 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:25.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:25.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:25.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:25.882 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:25.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 6.1e deep-scrub starts 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 6.1e deep-scrub ok 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 4.d deep-scrub starts 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 4.d deep-scrub ok 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:42:25.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: osdmap e278: 8 total, 8 up, 7 in 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[27010]: osdmap e279: 8 total, 8 up, 7 in 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 6.1e deep-scrub starts 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 6.1e deep-scrub ok 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 4.d deep-scrub starts 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 4.d deep-scrub ok 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: osdmap e278: 8 total, 8 up, 7 in 2024-04-25T05:42:25.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:25 smithi043 ceph-mon[37492]: osdmap e279: 8 total, 8 up, 7 in 2024-04-25T05:42:25.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 6.1e deep-scrub starts 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 6.1e deep-scrub ok 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 4.d deep-scrub starts 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 4.d deep-scrub ok 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: osdmap e278: 8 total, 8 up, 7 in 2024-04-25T05:42:25.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:25 smithi060 ceph-mon[30898]: osdmap e279: 8 total, 8 up, 7 in 2024-04-25T05:42:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:26.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:26.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:26.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:26.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:26.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:26.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:26.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:26.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:26.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:26.709 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:26.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:42:26.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:26.823 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:26 smithi043 ceph-mon[27010]: pgmap v441: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-04-25T05:42:26.823 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:26 smithi043 ceph-mon[37492]: pgmap v441: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-04-25T05:42:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.850 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:26.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:26.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:26.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:26 smithi060 ceph-mon[30898]: pgmap v441: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-04-25T05:42:27.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:27.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:27.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:27.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:27.326 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:27.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:27.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:27.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:27.556 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:27.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:27.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:27.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:27.831 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:27.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:27.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:27.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:28.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:28.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:28.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:28.487 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:42:28.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:28.491 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:42:28.524 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:28.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:28.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:42:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.650 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:28.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:28.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:28.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:28.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:28.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:28.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:29.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:28 smithi043 ceph-mon[27010]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-04-25T05:42:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:28 smithi043 ceph-mon[37492]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-04-25T05:42:29.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:29.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:29.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:28 smithi060 ceph-mon[30898]: pgmap v442: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-04-25T05:42:29.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:42:29.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:29.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:29.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:29.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:29.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:29.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:29.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:29.838 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-04-25T05:42:27.826467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2895059507,"num_objects":44397,"num_object_clones":0,"num_object_copies":133191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44397,"num_whiteouts":0,"num_read":1483,"num_read_kb":1203,"num_write":204705,"num_write_kb":7943790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63961,"num_bytes_recovered":4171243691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73214,"ondisk_log_size":73214,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11884152,"kb_used_data":9373696,"kb_used_omap":10,"kb_used_meta":2510197,"kb_avail":644331912,"statfs":{"total":671965249536,"available":659795877888,"internally_reserved":0,"allocated":9598664704,"data_stored":9595180509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2570441835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-444071936,"num_objects":-6776,"num_object_clones":0,"num_object_copies":-20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6776,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002505"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":128,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238065+0000","last_change":"2024-04-25T05:41:35.583008+0000","last_active":"2024-04-25T05:42:16.238065+0000","last_peered":"2024-04-25T05:42:16.238065+0000","last_clean":"2024-04-25T05:42:16.238065+0000","last_became_active":"2024-04-25T05:40:09.734217+0000","last_became_peered":"2024-04-25T05:40:09.734217+0000","last_unstale":"2024-04-25T05:42:16.238065+0000","last_undegraded":"2024-04-25T05:42:16.238065+0000","last_fullsized":"2024-04-25T05:42:16.238065+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:01.911520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239489+0000","last_change":"2024-04-25T05:40:37.473300+0000","last_active":"2024-04-25T05:42:16.239489+0000","last_peered":"2024-04-25T05:42:16.239489+0000","last_clean":"2024-04-25T05:42:16.239489+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:42:16.239489+0000","last_undegraded":"2024-04-25T05:42:16.239489+0000","last_fullsized":"2024-04-25T05:42:16.239489+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:37.473018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.473018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":105,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229013+0000","last_change":"2024-04-25T05:40:55.954113+0000","last_active":"2024-04-25T05:42:16.229013+0000","last_peered":"2024-04-25T05:42:16.229013+0000","last_clean":"2024-04-25T05:42:16.229013+0000","last_became_active":"2024-04-25T05:40:55.953726+0000","last_became_peered":"2024-04-25T05:40:55.953726+0000","last_unstale":"2024-04-25T05:42:16.229013+0000","last_undegraded":"2024-04-25T05:42:16.229013+0000","last_fullsized":"2024-04-25T05:42:16.229013+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":526,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866440+0000","last_change":"2024-04-25T05:42:06.942759+0000","last_active":"2024-04-25T05:42:24.866440+0000","last_peered":"2024-04-25T05:42:24.866440+0000","last_clean":"2024-04-25T05:42:24.866440+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:42:24.866440+0000","last_undegraded":"2024-04-25T05:42:24.866440+0000","last_fullsized":"2024-04-25T05:42:24.866440+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:42:06.942600+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:42:06.942600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:25.919821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":442,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235929+0000","last_change":"2024-04-25T05:39:29.799097+0000","last_active":"2024-04-25T05:42:16.235929+0000","last_peered":"2024-04-25T05:42:16.235929+0000","last_clean":"2024-04-25T05:42:16.235929+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:42:16.235929+0000","last_undegraded":"2024-04-25T05:42:16.235929+0000","last_fullsized":"2024-04-25T05:42:16.235929+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":417,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231509+0000","last_change":"2024-04-25T05:40:13.865954+0000","last_active":"2024-04-25T05:42:16.231509+0000","last_peered":"2024-04-25T05:42:16.231509+0000","last_clean":"2024-04-25T05:42:16.231509+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:42:16.231509+0000","last_undegraded":"2024-04-25T05:42:16.231509+0000","last_fullsized":"2024-04-25T05:42:16.231509+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:13.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:13.865834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":469,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.236222+0000","last_change":"2024-04-25T05:40:55.982555+0000","last_active":"2024-04-25T05:42:16.236222+0000","last_peered":"2024-04-25T05:42:16.236222+0000","last_clean":"2024-04-25T05:42:16.236222+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:42:16.236222+0000","last_undegraded":"2024-04-25T05:42:16.236222+0000","last_fullsized":"2024-04-25T05:42:16.236222+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.485053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":601,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235382+0000","last_change":"2024-04-25T05:40:56.000345+0000","last_active":"2024-04-25T05:42:16.235382+0000","last_peered":"2024-04-25T05:42:16.235382+0000","last_clean":"2024-04-25T05:42:16.235382+0000","last_became_active":"2024-04-25T05:40:55.974452+0000","last_became_peered":"2024-04-25T05:40:55.974452+0000","last_unstale":"2024-04-25T05:42:16.235382+0000","last_undegraded":"2024-04-25T05:42:16.235382+0000","last_fullsized":"2024-04-25T05:42:16.235382+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:39:41.219950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.219950+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229053+0000","last_change":"2024-04-25T05:40:43.441659+0000","last_active":"2024-04-25T05:42:16.229053+0000","last_peered":"2024-04-25T05:42:16.229053+0000","last_clean":"2024-04-25T05:42:16.229053+0000","last_became_active":"2024-04-25T05:40:43.441308+0000","last_became_peered":"2024-04-25T05:40:43.441308+0000","last_unstale":"2024-04-25T05:42:16.229053+0000","last_undegraded":"2024-04-25T05:42:16.229053+0000","last_fullsized":"2024-04-25T05:42:16.229053+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":454,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.234744+0000","last_change":"2024-04-25T05:40:43.453942+0000","last_active":"2024-04-25T05:42:16.234744+0000","last_peered":"2024-04-25T05:42:16.234744+0000","last_clean":"2024-04-25T05:42:16.234744+0000","last_became_active":"2024-04-25T05:40:43.453366+0000","last_became_peered":"2024-04-25T05:40:43.453366+0000","last_unstale":"2024-04-25T05:42:16.234744+0000","last_undegraded":"2024-04-25T05:42:16.234744+0000","last_fullsized":"2024-04-25T05:42:16.234744+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239316+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:42:16.239316+0000","last_peered":"2024-04-25T05:42:16.239316+0000","last_clean":"2024-04-25T05:42:16.239316+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:42:16.239316+0000","last_undegraded":"2024-04-25T05:42:16.239316+0000","last_fullsized":"2024-04-25T05:42:16.239316+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":208,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:19.252205+0000","last_change":"2024-04-25T05:42:19.252275+0000","last_active":"2024-04-25T05:42:19.252205+0000","last_peered":"2024-04-25T05:42:19.252205+0000","last_clean":"2024-04-25T05:42:19.252205+0000","last_became_active":"2024-04-25T05:40:43.433465+0000","last_became_peered":"2024-04-25T05:40:43.433465+0000","last_unstale":"2024-04-25T05:42:19.252205+0000","last_undegraded":"2024-04-25T05:42:19.252205+0000","last_fullsized":"2024-04-25T05:42:19.252205+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.890181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":446,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235174+0000","last_change":"2024-04-25T05:41:31.571478+0000","last_active":"2024-04-25T05:42:16.235174+0000","last_peered":"2024-04-25T05:42:16.235174+0000","last_clean":"2024-04-25T05:42:16.235174+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:42:16.235174+0000","last_undegraded":"2024-04-25T05:42:16.235174+0000","last_fullsized":"2024-04-25T05:42:16.235174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.853234+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228828+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:42:16.228828+0000","last_peered":"2024-04-25T05:42:16.228828+0000","last_clean":"2024-04-25T05:42:16.228828+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:42:16.228828+0000","last_undegraded":"2024-04-25T05:42:16.228828+0000","last_fullsized":"2024-04-25T05:42:16.228828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.487676+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237233+0000","last_change":"2024-04-25T05:40:49.444611+0000","last_active":"2024-04-25T05:42:16.237233+0000","last_peered":"2024-04-25T05:42:16.237233+0000","last_clean":"2024-04-25T05:42:16.237233+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:42:16.237233+0000","last_undegraded":"2024-04-25T05:42:16.237233+0000","last_fullsized":"2024-04-25T05:42:16.237233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:49.444492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.444492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":183,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237885+0000","last_change":"2024-04-25T05:42:13.374107+0000","last_active":"2024-04-25T05:42:16.237885+0000","last_peered":"2024-04-25T05:42:16.237885+0000","last_clean":"2024-04-25T05:42:16.237885+0000","last_became_active":"2024-04-25T05:40:55.984807+0000","last_became_peered":"2024-04-25T05:40:55.984807+0000","last_unstale":"2024-04-25T05:42:16.237885+0000","last_undegraded":"2024-04-25T05:42:16.237885+0000","last_fullsized":"2024-04-25T05:42:16.237885+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.061132+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867637+0000","last_change":"2024-04-25T05:42:24.062360+0000","last_active":"2024-04-25T05:42:24.867637+0000","last_peered":"2024-04-25T05:42:24.867637+0000","last_clean":"2024-04-25T05:42:24.867637+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:42:24.867637+0000","last_undegraded":"2024-04-25T05:42:24.867637+0000","last_fullsized":"2024-04-25T05:42:24.867637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.062132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.623846+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240459+0000","last_change":"2024-04-25T05:40:43.451998+0000","last_active":"2024-04-25T05:42:16.240459+0000","last_peered":"2024-04-25T05:42:16.240459+0000","last_clean":"2024-04-25T05:42:16.240459+0000","last_became_active":"2024-04-25T05:40:43.449712+0000","last_became_peered":"2024-04-25T05:40:43.449712+0000","last_unstale":"2024-04-25T05:42:16.240459+0000","last_undegraded":"2024-04-25T05:42:16.240459+0000","last_fullsized":"2024-04-25T05:42:16.240459+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:41.506653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.506653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231544+0000","last_change":"2024-04-25T05:40:43.435522+0000","last_active":"2024-04-25T05:42:16.231544+0000","last_peered":"2024-04-25T05:42:16.231544+0000","last_clean":"2024-04-25T05:42:16.231544+0000","last_became_active":"2024-04-25T05:40:43.434830+0000","last_became_peered":"2024-04-25T05:40:43.434830+0000","last_unstale":"2024-04-25T05:42:16.231544+0000","last_undegraded":"2024-04-25T05:42:16.231544+0000","last_fullsized":"2024-04-25T05:42:16.231544+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":309,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239573+0000","last_change":"2024-04-25T05:40:56.033623+0000","last_active":"2024-04-25T05:42:16.239573+0000","last_peered":"2024-04-25T05:42:16.239573+0000","last_clean":"2024-04-25T05:42:16.239573+0000","last_became_active":"2024-04-25T05:40:55.991291+0000","last_became_peered":"2024-04-25T05:40:55.991291+0000","last_unstale":"2024-04-25T05:42:16.239573+0000","last_undegraded":"2024-04-25T05:42:16.239573+0000","last_fullsized":"2024-04-25T05:42:16.239573+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:39:42.756928+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.756928+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":443,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238823+0000","last_change":"2024-04-25T05:39:29.799140+0000","last_active":"2024-04-25T05:42:16.238823+0000","last_peered":"2024-04-25T05:42:16.238823+0000","last_clean":"2024-04-25T05:42:16.238823+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:42:16.238823+0000","last_undegraded":"2024-04-25T05:42:16.238823+0000","last_fullsized":"2024-04-25T05:42:16.238823+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":127,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237484+0000","last_change":"2024-04-25T05:40:55.973839+0000","last_active":"2024-04-25T05:42:16.237484+0000","last_peered":"2024-04-25T05:42:16.237484+0000","last_clean":"2024-04-25T05:42:16.237484+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:42:16.237484+0000","last_undegraded":"2024-04-25T05:42:16.237484+0000","last_fullsized":"2024-04-25T05:42:16.237484+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866758+0000","last_change":"2024-04-25T05:42:11.885667+0000","last_active":"2024-04-25T05:42:24.866758+0000","last_peered":"2024-04-25T05:42:24.866758+0000","last_clean":"2024-04-25T05:42:24.866758+0000","last_became_active":"2024-04-25T05:40:55.964565+0000","last_became_peered":"2024-04-25T05:40:55.964565+0000","last_unstale":"2024-04-25T05:42:24.866758+0000","last_undegraded":"2024-04-25T05:42:24.866758+0000","last_fullsized":"2024-04-25T05:42:24.866758+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:11.885545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:11.885545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.830666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":473,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235842+0000","last_change":"2024-04-25T05:40:15.536952+0000","last_active":"2024-04-25T05:42:16.235842+0000","last_peered":"2024-04-25T05:42:16.235842+0000","last_clean":"2024-04-25T05:42:16.235842+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:42:16.235842+0000","last_undegraded":"2024-04-25T05:42:16.235842+0000","last_fullsized":"2024-04-25T05:42:16.235842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:40:15.536802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.536802+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":457,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228827+0000","last_change":"2024-04-25T05:40:55.989126+0000","last_active":"2024-04-25T05:42:16.228827+0000","last_peered":"2024-04-25T05:42:16.228827+0000","last_clean":"2024-04-25T05:42:16.228827+0000","last_became_active":"2024-04-25T05:40:55.974471+0000","last_became_peered":"2024-04-25T05:40:55.974471+0000","last_unstale":"2024-04-25T05:42:16.228827+0000","last_undegraded":"2024-04-25T05:42:16.228827+0000","last_fullsized":"2024-04-25T05:42:16.228827+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:40:10.849562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:10.849562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":204,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.232799+0000","last_change":"2024-04-25T05:39:38.521412+0000","last_active":"2024-04-25T05:42:16.232799+0000","last_peered":"2024-04-25T05:42:16.232799+0000","last_clean":"2024-04-25T05:42:16.232799+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:42:16.232799+0000","last_undegraded":"2024-04-25T05:42:16.232799+0000","last_fullsized":"2024-04-25T05:42:16.232799+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:39:38.521265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.521265+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":459,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239401+0000","last_change":"2024-04-25T05:40:40.500687+0000","last_active":"2024-04-25T05:42:16.239401+0000","last_peered":"2024-04-25T05:42:16.239401+0000","last_clean":"2024-04-25T05:42:16.239401+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:42:16.239401+0000","last_undegraded":"2024-04-25T05:42:16.239401+0000","last_fullsized":"2024-04-25T05:42:16.239401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:40.500174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.500174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":113,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228836+0000","last_change":"2024-04-25T05:40:43.482944+0000","last_active":"2024-04-25T05:42:16.228836+0000","last_peered":"2024-04-25T05:42:16.228836+0000","last_clean":"2024-04-25T05:42:16.228836+0000","last_became_active":"2024-04-25T05:40:43.442720+0000","last_became_peered":"2024-04-25T05:40:43.442720+0000","last_unstale":"2024-04-25T05:42:16.228836+0000","last_undegraded":"2024-04-25T05:42:16.228836+0000","last_fullsized":"2024-04-25T05:42:16.228836+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":438,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240173+0000","last_change":"2024-04-25T05:40:15.703133+0000","last_active":"2024-04-25T05:42:16.240173+0000","last_peered":"2024-04-25T05:42:16.240173+0000","last_clean":"2024-04-25T05:42:16.240173+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:42:16.240173+0000","last_undegraded":"2024-04-25T05:42:16.240173+0000","last_fullsized":"2024-04-25T05:42:16.240173+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.702999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.702999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":105,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229052+0000","last_change":"2024-04-25T05:40:55.967146+0000","last_active":"2024-04-25T05:42:16.229052+0000","last_peered":"2024-04-25T05:42:16.229052+0000","last_clean":"2024-04-25T05:42:16.229052+0000","last_became_active":"2024-04-25T05:40:55.966806+0000","last_became_peered":"2024-04-25T05:40:55.966806+0000","last_unstale":"2024-04-25T05:42:16.229052+0000","last_undegraded":"2024-04-25T05:42:16.229052+0000","last_fullsized":"2024-04-25T05:42:16.229052+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235001+0000","last_change":"2024-04-25T05:40:50.562106+0000","last_active":"2024-04-25T05:42:16.235001+0000","last_peered":"2024-04-25T05:42:16.235001+0000","last_clean":"2024-04-25T05:42:16.235001+0000","last_became_active":"2024-04-25T05:40:43.440363+0000","last_became_peered":"2024-04-25T05:40:43.440363+0000","last_unstale":"2024-04-25T05:42:16.235001+0000","last_undegraded":"2024-04-25T05:42:16.235001+0000","last_fullsized":"2024-04-25T05:42:16.235001+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.560317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.560317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:18.263940+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":180,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.233029+0000","last_change":"2024-04-25T05:39:40.474408+0000","last_active":"2024-04-25T05:42:16.233029+0000","last_peered":"2024-04-25T05:42:16.233029+0000","last_clean":"2024-04-25T05:42:16.233029+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:42:16.233029+0000","last_undegraded":"2024-04-25T05:42:16.233029+0000","last_fullsized":"2024-04-25T05:42:16.233029+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:40.474140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.474140+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":457,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229030+0000","last_change":"2024-04-25T05:41:39.031952+0000","last_active":"2024-04-25T05:42:16.229030+0000","last_peered":"2024-04-25T05:42:16.229030+0000","last_clean":"2024-04-25T05:42:16.229030+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:42:16.229030+0000","last_undegraded":"2024-04-25T05:42:16.229030+0000","last_fullsized":"2024-04-25T05:42:16.229030+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:41:39.031784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.031784+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.769610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":166,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.230995+0000","last_change":"2024-04-25T05:41:40.152573+0000","last_active":"2024-04-25T05:42:16.230995+0000","last_peered":"2024-04-25T05:42:16.230995+0000","last_clean":"2024-04-25T05:42:16.230995+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:42:16.230995+0000","last_undegraded":"2024-04-25T05:42:16.230995+0000","last_fullsized":"2024-04-25T05:42:16.230995+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:40.152411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:40.152411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:02.701355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228920+0000","last_change":"2024-04-25T05:40:43.446312+0000","last_active":"2024-04-25T05:42:16.228920+0000","last_peered":"2024-04-25T05:42:16.228920+0000","last_clean":"2024-04-25T05:42:16.228920+0000","last_became_active":"2024-04-25T05:40:43.445367+0000","last_became_peered":"2024-04-25T05:40:43.445367+0000","last_unstale":"2024-04-25T05:42:16.228920+0000","last_undegraded":"2024-04-25T05:42:16.228920+0000","last_fullsized":"2024-04-25T05:42:16.228920+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":476,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235089+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:42:16.235089+0000","last_peered":"2024-04-25T05:42:16.235089+0000","last_clean":"2024-04-25T05:42:16.235089+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:42:16.235089+0000","last_undegraded":"2024-04-25T05:42:16.235089+0000","last_fullsized":"2024-04-25T05:42:16.235089+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:23.390274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:20.379880+0000","last_change":"2024-04-25T05:42:20.379921+0000","last_active":"2024-04-25T05:42:20.379880+0000","last_peered":"2024-04-25T05:42:20.379880+0000","last_clean":"2024-04-25T05:42:20.379880+0000","last_became_active":"2024-04-25T05:40:43.435420+0000","last_became_peered":"2024-04-25T05:40:43.435420+0000","last_unstale":"2024-04-25T05:42:20.379880+0000","last_undegraded":"2024-04-25T05:42:20.379880+0000","last_fullsized":"2024-04-25T05:42:20.379880+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.364344+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":468,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228245+0000","last_change":"2024-04-25T05:40:53.928517+0000","last_active":"2024-04-25T05:42:16.228245+0000","last_peered":"2024-04-25T05:42:16.228245+0000","last_clean":"2024-04-25T05:42:16.228245+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:42:16.228245+0000","last_undegraded":"2024-04-25T05:42:16.228245+0000","last_fullsized":"2024-04-25T05:42:16.228245+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:53.928372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.928372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237294+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:42:16.237294+0000","last_peered":"2024-04-25T05:42:16.237294+0000","last_clean":"2024-04-25T05:42:16.237294+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:42:16.237294+0000","last_undegraded":"2024-04-25T05:42:16.237294+0000","last_fullsized":"2024-04-25T05:42:16.237294+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":517,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235359+0000","last_change":"2024-04-25T05:40:49.558496+0000","last_active":"2024-04-25T05:42:16.235359+0000","last_peered":"2024-04-25T05:42:16.235359+0000","last_clean":"2024-04-25T05:42:16.235359+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:42:16.235359+0000","last_undegraded":"2024-04-25T05:42:16.235359+0000","last_fullsized":"2024-04-25T05:42:16.235359+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:49.558354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.558354+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228844+0000","last_change":"2024-04-25T05:40:43.446609+0000","last_active":"2024-04-25T05:42:16.228844+0000","last_peered":"2024-04-25T05:42:16.228844+0000","last_clean":"2024-04-25T05:42:16.228844+0000","last_became_active":"2024-04-25T05:40:43.446278+0000","last_became_peered":"2024-04-25T05:40:43.446278+0000","last_unstale":"2024-04-25T05:42:16.228844+0000","last_undegraded":"2024-04-25T05:42:16.228844+0000","last_fullsized":"2024-04-25T05:42:16.228844+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":446,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235803+0000","last_change":"2024-04-25T05:40:43.433578+0000","last_active":"2024-04-25T05:42:16.235803+0000","last_peered":"2024-04-25T05:42:16.235803+0000","last_clean":"2024-04-25T05:42:16.235803+0000","last_became_active":"2024-04-25T05:40:43.433240+0000","last_became_peered":"2024-04-25T05:40:43.433240+0000","last_unstale":"2024-04-25T05:42:16.235803+0000","last_undegraded":"2024-04-25T05:42:16.235803+0000","last_fullsized":"2024-04-25T05:42:16.235803+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.526768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.526768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228527+0000","last_change":"2024-04-25T05:40:17.791243+0000","last_active":"2024-04-25T05:42:16.228527+0000","last_peered":"2024-04-25T05:42:16.228527+0000","last_clean":"2024-04-25T05:42:16.228527+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:42:16.228527+0000","last_undegraded":"2024-04-25T05:42:16.228527+0000","last_fullsized":"2024-04-25T05:42:16.228527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:17.791104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.791104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"257'13","reported_seq":371,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237710+0000","last_change":"2024-04-25T05:41:30.532539+0000","last_active":"2024-04-25T05:42:16.237710+0000","last_peered":"2024-04-25T05:42:16.237710+0000","last_clean":"2024-04-25T05:42:16.237710+0000","last_became_active":"2024-04-25T05:40:55.954865+0000","last_became_peered":"2024-04-25T05:40:55.954865+0000","last_unstale":"2024-04-25T05:42:16.237710+0000","last_undegraded":"2024-04-25T05:42:16.237710+0000","last_fullsized":"2024-04-25T05:42:16.237710+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:59.262752+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235167+0000","last_change":"2024-04-25T05:40:43.434425+0000","last_active":"2024-04-25T05:42:16.235167+0000","last_peered":"2024-04-25T05:42:16.235167+0000","last_clean":"2024-04-25T05:42:16.235167+0000","last_became_active":"2024-04-25T05:40:43.434051+0000","last_became_peered":"2024-04-25T05:40:43.434051+0000","last_unstale":"2024-04-25T05:42:16.235167+0000","last_undegraded":"2024-04-25T05:42:16.235167+0000","last_fullsized":"2024-04-25T05:42:16.235167+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:04.446776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:04.446776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":442,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.243285+0000","last_change":"2024-04-25T05:40:06.644197+0000","last_active":"2024-04-25T05:42:16.243285+0000","last_peered":"2024-04-25T05:42:16.243285+0000","last_clean":"2024-04-25T05:42:16.243285+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:42:16.243285+0000","last_undegraded":"2024-04-25T05:42:16.243285+0000","last_fullsized":"2024-04-25T05:42:16.243285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.644083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.644083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":137,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228573+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:42:16.228573+0000","last_peered":"2024-04-25T05:42:16.228573+0000","last_clean":"2024-04-25T05:42:16.228573+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:42:16.228573+0000","last_undegraded":"2024-04-25T05:42:16.228573+0000","last_fullsized":"2024-04-25T05:42:16.228573+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.152477+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":548,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867767+0000","last_change":"2024-04-25T05:40:56.002007+0000","last_active":"2024-04-25T05:42:24.867767+0000","last_peered":"2024-04-25T05:42:24.867767+0000","last_clean":"2024-04-25T05:42:24.867767+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:42:24.867767+0000","last_undegraded":"2024-04-25T05:42:24.867767+0000","last_fullsized":"2024-04-25T05:42:24.867767+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:40:17.835081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.835081+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229593+0000","last_change":"2024-04-25T05:40:43.443261+0000","last_active":"2024-04-25T05:42:16.229593+0000","last_peered":"2024-04-25T05:42:16.229593+0000","last_clean":"2024-04-25T05:42:16.229593+0000","last_became_active":"2024-04-25T05:40:43.439460+0000","last_became_peered":"2024-04-25T05:40:43.439460+0000","last_unstale":"2024-04-25T05:42:16.229593+0000","last_undegraded":"2024-04-25T05:42:16.229593+0000","last_fullsized":"2024-04-25T05:42:16.229593+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"233'10","reported_seq":580,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239452+0000","last_change":"2024-04-25T05:41:45.481735+0000","last_active":"2024-04-25T05:42:16.239452+0000","last_peered":"2024-04-25T05:42:16.239452+0000","last_clean":"2024-04-25T05:42:16.239452+0000","last_became_active":"2024-04-25T05:40:43.453246+0000","last_became_peered":"2024-04-25T05:40:43.453246+0000","last_unstale":"2024-04-25T05:42:16.239452+0000","last_undegraded":"2024-04-25T05:42:16.239452+0000","last_fullsized":"2024-04-25T05:42:16.239452+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:48.532916+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229301+0000","last_change":"2024-04-25T05:41:37.012236+0000","last_active":"2024-04-25T05:42:16.229301+0000","last_peered":"2024-04-25T05:42:16.229301+0000","last_clean":"2024-04-25T05:42:16.229301+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:42:16.229301+0000","last_undegraded":"2024-04-25T05:42:16.229301+0000","last_fullsized":"2024-04-25T05:42:16.229301+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:37.012159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:37.012159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.873027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":485,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237346+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:42:16.237346+0000","last_peered":"2024-04-25T05:42:16.237346+0000","last_clean":"2024-04-25T05:42:16.237346+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:42:16.237346+0000","last_undegraded":"2024-04-25T05:42:16.237346+0000","last_fullsized":"2024-04-25T05:42:16.237346+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:15.927150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":86,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239012+0000","last_change":"2024-04-25T05:40:55.954634+0000","last_active":"2024-04-25T05:42:16.239012+0000","last_peered":"2024-04-25T05:42:16.239012+0000","last_clean":"2024-04-25T05:42:16.239012+0000","last_became_active":"2024-04-25T05:40:55.954181+0000","last_became_peered":"2024-04-25T05:40:55.954181+0000","last_unstale":"2024-04-25T05:42:16.239012+0000","last_undegraded":"2024-04-25T05:42:16.239012+0000","last_fullsized":"2024-04-25T05:42:16.239012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:09.913324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.913324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":553,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237807+0000","last_change":"2024-04-25T05:40:56.012131+0000","last_active":"2024-04-25T05:42:16.237807+0000","last_peered":"2024-04-25T05:42:16.237807+0000","last_clean":"2024-04-25T05:42:16.237807+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:42:16.237807+0000","last_undegraded":"2024-04-25T05:42:16.237807+0000","last_fullsized":"2024-04-25T05:42:16.237807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:40:37.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.529721+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.226048+0000","last_change":"2024-04-25T05:40:43.434277+0000","last_active":"2024-04-25T05:42:16.226048+0000","last_peered":"2024-04-25T05:42:16.226048+0000","last_clean":"2024-04-25T05:42:16.226048+0000","last_became_active":"2024-04-25T05:40:43.433933+0000","last_became_peered":"2024-04-25T05:40:43.433933+0000","last_unstale":"2024-04-25T05:42:16.226048+0000","last_undegraded":"2024-04-25T05:42:16.226048+0000","last_fullsized":"2024-04-25T05:42:16.226048+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":487,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239659+0000","last_change":"2024-04-25T05:42:08.829436+0000","last_active":"2024-04-25T05:42:16.239659+0000","last_peered":"2024-04-25T05:42:16.239659+0000","last_clean":"2024-04-25T05:42:16.239659+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:42:16.239659+0000","last_undegraded":"2024-04-25T05:42:16.239659+0000","last_fullsized":"2024-04-25T05:42:16.239659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:22.650865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":97,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868158+0000","last_change":"2024-04-25T05:42:10.907844+0000","last_active":"2024-04-25T05:42:24.868158+0000","last_peered":"2024-04-25T05:42:24.868158+0000","last_clean":"2024-04-25T05:42:24.868158+0000","last_became_active":"2024-04-25T05:40:55.953924+0000","last_became_peered":"2024-04-25T05:40:55.953924+0000","last_unstale":"2024-04-25T05:42:24.868158+0000","last_undegraded":"2024-04-25T05:42:24.868158+0000","last_fullsized":"2024-04-25T05:42:24.868158+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:10.907707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.907707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:31.135983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":434,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238711+0000","last_change":"2024-04-25T05:40:38.559014+0000","last_active":"2024-04-25T05:42:16.238711+0000","last_peered":"2024-04-25T05:42:16.238711+0000","last_clean":"2024-04-25T05:42:16.238711+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:42:16.238711+0000","last_undegraded":"2024-04-25T05:42:16.238711+0000","last_fullsized":"2024-04-25T05:42:16.238711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:38.558895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:38.558895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":476,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228365+0000","last_change":"2024-04-25T05:40:50.906267+0000","last_active":"2024-04-25T05:42:16.228365+0000","last_peered":"2024-04-25T05:42:16.228365+0000","last_clean":"2024-04-25T05:42:16.228365+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:42:16.228365+0000","last_undegraded":"2024-04-25T05:42:16.228365+0000","last_fullsized":"2024-04-25T05:42:16.228365+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.905458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.905458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:20.239647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":121,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:21.659457+0000","last_change":"2024-04-25T05:42:21.659494+0000","last_active":"2024-04-25T05:42:21.659457+0000","last_peered":"2024-04-25T05:42:21.659457+0000","last_clean":"2024-04-25T05:42:21.659457+0000","last_became_active":"2024-04-25T05:40:43.441558+0000","last_became_peered":"2024-04-25T05:40:43.441558+0000","last_unstale":"2024-04-25T05:42:21.659457+0000","last_undegraded":"2024-04-25T05:42:21.659457+0000","last_fullsized":"2024-04-25T05:42:21.659457+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.744922+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228385+0000","last_change":"2024-04-25T05:41:27.185161+0000","last_active":"2024-04-25T05:42:16.228385+0000","last_peered":"2024-04-25T05:42:16.228385+0000","last_clean":"2024-04-25T05:42:16.228385+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:42:16.228385+0000","last_undegraded":"2024-04-25T05:42:16.228385+0000","last_fullsized":"2024-04-25T05:42:16.228385+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.185044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.185044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:28.448062+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":442,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237167+0000","last_change":"2024-04-25T05:40:46.378509+0000","last_active":"2024-04-25T05:42:16.237167+0000","last_peered":"2024-04-25T05:42:16.237167+0000","last_clean":"2024-04-25T05:42:16.237167+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:42:16.237167+0000","last_undegraded":"2024-04-25T05:42:16.237167+0000","last_fullsized":"2024-04-25T05:42:16.237167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:46.378394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:40:46.378394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":172,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231119+0000","last_change":"2024-04-25T05:40:43.431845+0000","last_active":"2024-04-25T05:42:16.231119+0000","last_peered":"2024-04-25T05:42:16.231119+0000","last_clean":"2024-04-25T05:42:16.231119+0000","last_became_active":"2024-04-25T05:40:43.431502+0000","last_became_peered":"2024-04-25T05:40:43.431502+0000","last_unstale":"2024-04-25T05:42:16.231119+0000","last_undegraded":"2024-04-25T05:42:16.231119+0000","last_fullsized":"2024-04-25T05:42:16.231119+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.918723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.918723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":467,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238877+0000","last_change":"2024-04-25T05:40:41.507574+0000","last_active":"2024-04-25T05:42:16.238877+0000","last_peered":"2024-04-25T05:42:16.238877+0000","last_clean":"2024-04-25T05:42:16.238877+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:42:16.238877+0000","last_undegraded":"2024-04-25T05:42:16.238877+0000","last_fullsized":"2024-04-25T05:42:16.238877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:41.507400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.507400+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"277'12847","reported_seq":27480,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.837036+0000","last_change":"2024-04-25T05:41:28.270522+0000","last_active":"2024-04-25T05:42:22.837036+0000","last_peered":"2024-04-25T05:42:22.837036+0000","last_clean":"2024-04-25T05:42:22.837036+0000","last_became_active":"2024-04-25T05:41:28.224071+0000","last_became_peered":"2024-04-25T05:41:28.224071+0000","last_unstale":"2024-04-25T05:42:22.837036+0000","last_undegraded":"2024-04-25T05:42:22.837036+0000","last_fullsized":"2024-04-25T05:42:22.837036+0000","mapping_epoch":255,"log_start":"247'8437","ondisk_log_start":"247'8437","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'7237","last_scrub_stamp":"2024-04-25T05:40:43.047218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.047218+0000","objects_scrubbed":7237,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192217088,"num_objects":2933,"num_object_clones":0,"num_object_copies":8799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":504960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7185,"num_bytes_recovered":470810624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238439+0000","last_change":"2024-04-25T05:40:51.460612+0000","last_active":"2024-04-25T05:42:16.238439+0000","last_peered":"2024-04-25T05:42:16.238439+0000","last_clean":"2024-04-25T05:42:16.238439+0000","last_became_active":"2024-04-25T05:40:43.453025+0000","last_became_peered":"2024-04-25T05:40:43.453025+0000","last_unstale":"2024-04-25T05:42:16.238439+0000","last_undegraded":"2024-04-25T05:42:16.238439+0000","last_fullsized":"2024-04-25T05:42:16.238439+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.460330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.460330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":494,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229964+0000","last_change":"2024-04-25T05:41:25.239411+0000","last_active":"2024-04-25T05:42:16.229964+0000","last_peered":"2024-04-25T05:42:16.229964+0000","last_clean":"2024-04-25T05:42:16.229964+0000","last_became_active":"2024-04-25T05:40:43.429608+0000","last_became_peered":"2024-04-25T05:40:43.429608+0000","last_unstale":"2024-04-25T05:42:16.229964+0000","last_undegraded":"2024-04-25T05:42:16.229964+0000","last_fullsized":"2024-04-25T05:42:16.229964+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:41:25.239292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:25.239292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:33.712255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":547,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238121+0000","last_change":"2024-04-25T05:39:30.386374+0000","last_active":"2024-04-25T05:42:16.238121+0000","last_peered":"2024-04-25T05:42:16.238121+0000","last_clean":"2024-04-25T05:42:16.238121+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:42:16.238121+0000","last_undegraded":"2024-04-25T05:42:16.238121+0000","last_fullsized":"2024-04-25T05:42:16.238121+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:39:30.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.386275+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":583,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228636+0000","last_change":"2024-04-25T05:39:33.128642+0000","last_active":"2024-04-25T05:42:16.228636+0000","last_peered":"2024-04-25T05:42:16.228636+0000","last_clean":"2024-04-25T05:42:16.228636+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:42:16.228636+0000","last_undegraded":"2024-04-25T05:42:16.228636+0000","last_fullsized":"2024-04-25T05:42:16.228636+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'10","last_scrub_stamp":"2024-04-25T05:39:33.128477+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.128477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"277'12815","reported_seq":13238,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.775413+0000","last_change":"2024-04-25T05:41:36.031565+0000","last_active":"2024-04-25T05:42:22.775413+0000","last_peered":"2024-04-25T05:42:22.775413+0000","last_clean":"2024-04-25T05:42:22.775413+0000","last_became_active":"2024-04-25T05:39:35.778659+0000","last_became_peered":"2024-04-25T05:39:35.778659+0000","last_unstale":"2024-04-25T05:42:22.775413+0000","last_undegraded":"2024-04-25T05:42:22.775413+0000","last_fullsized":"2024-04-25T05:42:22.775413+0000","mapping_epoch":209,"log_start":"245'8060","ondisk_log_start":"245'8060","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9942","last_scrub_stamp":"2024-04-25T05:41:36.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.031226+0000","objects_scrubbed":5781,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.717941+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189726720,"num_objects":2895,"num_object_clones":0,"num_object_copies":8685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12815,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":442,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.236024+0000","last_change":"2024-04-25T05:40:55.989744+0000","last_active":"2024-04-25T05:42:16.236024+0000","last_peered":"2024-04-25T05:42:16.236024+0000","last_clean":"2024-04-25T05:42:16.236024+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:42:16.236024+0000","last_undegraded":"2024-04-25T05:42:16.236024+0000","last_fullsized":"2024-04-25T05:42:16.236024+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.579078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.579078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229268+0000","last_change":"2024-04-25T05:39:38.030757+0000","last_active":"2024-04-25T05:42:16.229268+0000","last_peered":"2024-04-25T05:42:16.229268+0000","last_clean":"2024-04-25T05:42:16.229268+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:42:16.229268+0000","last_undegraded":"2024-04-25T05:42:16.229268+0000","last_fullsized":"2024-04-25T05:42:16.229268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.030627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.030627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239166+0000","last_change":"2024-04-25T05:40:56.145089+0000","last_active":"2024-04-25T05:42:16.239166+0000","last_peered":"2024-04-25T05:42:16.239166+0000","last_clean":"2024-04-25T05:42:16.239166+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:42:16.239166+0000","last_undegraded":"2024-04-25T05:42:16.239166+0000","last_fullsized":"2024-04-25T05:42:16.239166+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:40:37.943948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.943948+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":448,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228956+0000","last_change":"2024-04-25T05:40:55.988511+0000","last_active":"2024-04-25T05:42:16.228956+0000","last_peered":"2024-04-25T05:42:16.228956+0000","last_clean":"2024-04-25T05:42:16.228956+0000","last_became_active":"2024-04-25T05:40:55.988228+0000","last_became_peered":"2024-04-25T05:40:55.988228+0000","last_unstale":"2024-04-25T05:42:16.228956+0000","last_undegraded":"2024-04-25T05:42:16.228956+0000","last_fullsized":"2024-04-25T05:42:16.228956+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"277'12709","reported_seq":27480,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.257993+0000","last_change":"2024-04-25T05:41:42.929604+0000","last_active":"2024-04-25T05:42:23.257993+0000","last_peered":"2024-04-25T05:42:23.257993+0000","last_clean":"2024-04-25T05:42:23.257993+0000","last_became_active":"2024-04-25T05:41:42.899661+0000","last_became_peered":"2024-04-25T05:41:42.899661+0000","last_unstale":"2024-04-25T05:42:23.257993+0000","last_undegraded":"2024-04-25T05:42:23.257993+0000","last_fullsized":"2024-04-25T05:42:23.257993+0000","mapping_epoch":260,"log_start":"244'7957","ondisk_log_start":"244'7957","created":198,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186449920,"num_objects":2845,"num_object_clones":0,"num_object_copies":8535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12709,"num_write_kb":497728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7289,"num_bytes_recovered":477691904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":444,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.227850+0000","last_change":"2024-04-25T05:40:55.982002+0000","last_active":"2024-04-25T05:42:16.227850+0000","last_peered":"2024-04-25T05:42:16.227850+0000","last_clean":"2024-04-25T05:42:16.227850+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:42:16.227850+0000","last_undegraded":"2024-04-25T05:42:16.227850+0000","last_fullsized":"2024-04-25T05:42:16.227850+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.074043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.074043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231001+0000","last_change":"2024-04-25T05:40:55.953740+0000","last_active":"2024-04-25T05:42:16.231001+0000","last_peered":"2024-04-25T05:42:16.231001+0000","last_clean":"2024-04-25T05:42:16.231001+0000","last_became_active":"2024-04-25T05:40:55.953404+0000","last_became_peered":"2024-04-25T05:40:55.953404+0000","last_unstale":"2024-04-25T05:42:16.231001+0000","last_undegraded":"2024-04-25T05:42:16.231001+0000","last_fullsized":"2024-04-25T05:42:16.231001+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":484,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229241+0000","last_change":"2024-04-25T05:40:08.880333+0000","last_active":"2024-04-25T05:42:16.229241+0000","last_peered":"2024-04-25T05:42:16.229241+0000","last_clean":"2024-04-25T05:42:16.229241+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:42:16.229241+0000","last_undegraded":"2024-04-25T05:42:16.229241+0000","last_fullsized":"2024-04-25T05:42:16.229241+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:08.880207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.880207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":196,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.230858+0000","last_change":"2024-04-25T05:42:15.186498+0000","last_active":"2024-04-25T05:42:16.230858+0000","last_peered":"2024-04-25T05:42:16.230858+0000","last_clean":"2024-04-25T05:42:16.230858+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:42:16.230858+0000","last_undegraded":"2024-04-25T05:42:16.230858+0000","last_fullsized":"2024-04-25T05:42:16.230858+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:15.186321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.186321+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:26.644738+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"277'12759","reported_seq":13894,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.386665+0000","last_change":"2024-04-25T05:40:53.660560+0000","last_active":"2024-04-25T05:42:23.386665+0000","last_peered":"2024-04-25T05:42:23.386665+0000","last_clean":"2024-04-25T05:42:23.386665+0000","last_became_active":"2024-04-25T05:39:35.777746+0000","last_became_peered":"2024-04-25T05:39:35.777746+0000","last_unstale":"2024-04-25T05:42:23.386665+0000","last_undegraded":"2024-04-25T05:42:23.386665+0000","last_fullsized":"2024-04-25T05:42:23.386665+0000","mapping_epoch":209,"log_start":"247'8413","ondisk_log_start":"247'8413","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7624","last_scrub_stamp":"2024-04-25T05:40:53.660173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.660173+0000","objects_scrubbed":7548,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":7947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12759,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":105,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.226675+0000","last_change":"2024-04-25T05:40:55.972738+0000","last_active":"2024-04-25T05:42:16.226675+0000","last_peered":"2024-04-25T05:42:16.226675+0000","last_clean":"2024-04-25T05:42:16.226675+0000","last_became_active":"2024-04-25T05:40:55.972405+0000","last_became_peered":"2024-04-25T05:40:55.972405+0000","last_unstale":"2024-04-25T05:42:16.226675+0000","last_undegraded":"2024-04-25T05:42:16.226675+0000","last_fullsized":"2024-04-25T05:42:16.226675+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":96,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867348+0000","last_change":"2024-04-25T05:42:05.954387+0000","last_active":"2024-04-25T05:42:24.867348+0000","last_peered":"2024-04-25T05:42:24.867348+0000","last_clean":"2024-04-25T05:42:24.867348+0000","last_became_active":"2024-04-25T05:40:55.981082+0000","last_became_peered":"2024-04-25T05:40:55.981082+0000","last_unstale":"2024-04-25T05:42:24.867348+0000","last_undegraded":"2024-04-25T05:42:24.867348+0000","last_fullsized":"2024-04-25T05:42:24.867348+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:05.954259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:05.954259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.753300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":470,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237323+0000","last_change":"2024-04-25T05:41:28.560703+0000","last_active":"2024-04-25T05:42:16.237323+0000","last_peered":"2024-04-25T05:42:16.237323+0000","last_clean":"2024-04-25T05:42:16.237323+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:42:16.237323+0000","last_undegraded":"2024-04-25T05:42:16.237323+0000","last_fullsized":"2024-04-25T05:42:16.237323+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:36.353516+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":111,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231735+0000","last_change":"2024-04-25T05:40:43.470705+0000","last_active":"2024-04-25T05:42:16.231735+0000","last_peered":"2024-04-25T05:42:16.231735+0000","last_clean":"2024-04-25T05:42:16.231735+0000","last_became_active":"2024-04-25T05:40:43.441309+0000","last_became_peered":"2024-04-25T05:40:43.441309+0000","last_unstale":"2024-04-25T05:42:16.231735+0000","last_undegraded":"2024-04-25T05:42:16.231735+0000","last_fullsized":"2024-04-25T05:42:16.231735+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"277'12530","reported_seq":12699,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.820734+0000","last_change":"2024-04-25T05:39:19.233374+0000","last_active":"2024-04-25T05:42:22.820734+0000","last_peered":"2024-04-25T05:42:22.820734+0000","last_clean":"2024-04-25T05:42:22.820734+0000","last_became_active":"2024-04-25T05:39:19.233055+0000","last_became_peered":"2024-04-25T05:39:19.233055+0000","last_unstale":"2024-04-25T05:42:22.820734+0000","last_undegraded":"2024-04-25T05:42:22.820734+0000","last_fullsized":"2024-04-25T05:42:22.820734+0000","mapping_epoch":198,"log_start":"247'8178","ondisk_log_start":"247'8178","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12530,"num_write_kb":488448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":643,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.234936+0000","last_change":"2024-04-25T05:40:58.922339+0000","last_active":"2024-04-25T05:42:16.234936+0000","last_peered":"2024-04-25T05:42:16.234936+0000","last_clean":"2024-04-25T05:42:16.234936+0000","last_became_active":"2024-04-25T05:40:58.921770+0000","last_became_peered":"2024-04-25T05:40:58.921770+0000","last_unstale":"2024-04-25T05:42:16.234936+0000","last_undegraded":"2024-04-25T05:42:16.234936+0000","last_fullsized":"2024-04-25T05:42:16.234936+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:07.411609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:07.411609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":438,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.363473+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:42:16.363473+0000","last_peered":"2024-04-25T05:42:16.363473+0000","last_clean":"2024-04-25T05:42:16.363473+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:42:16.363473+0000","last_undegraded":"2024-04-25T05:42:16.363473+0000","last_fullsized":"2024-04-25T05:42:16.363473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.757268+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":491,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238246+0000","last_change":"2024-04-25T05:39:29.827029+0000","last_active":"2024-04-25T05:42:16.238246+0000","last_peered":"2024-04-25T05:42:16.238246+0000","last_clean":"2024-04-25T05:42:16.238246+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:42:16.238246+0000","last_undegraded":"2024-04-25T05:42:16.238246+0000","last_fullsized":"2024-04-25T05:42:16.238246+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231636+0000","last_change":"2024-04-25T05:40:43.435305+0000","last_active":"2024-04-25T05:42:16.231636+0000","last_peered":"2024-04-25T05:42:16.231636+0000","last_clean":"2024-04-25T05:42:16.231636+0000","last_became_active":"2024-04-25T05:40:43.434803+0000","last_became_peered":"2024-04-25T05:40:43.434803+0000","last_unstale":"2024-04-25T05:42:16.231636+0000","last_undegraded":"2024-04-25T05:42:16.231636+0000","last_fullsized":"2024-04-25T05:42:16.231636+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.849716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.849716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"243'46","reported_seq":649,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:18.844261+0000","last_change":"2024-04-25T05:40:56.036908+0000","last_active":"2024-04-25T05:42:18.844261+0000","last_peered":"2024-04-25T05:42:18.844261+0000","last_clean":"2024-04-25T05:42:18.844261+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:42:18.844261+0000","last_undegraded":"2024-04-25T05:42:18.844261+0000","last_fullsized":"2024-04-25T05:42:18.844261+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:39:42.723973+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.723973+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228499+0000","last_change":"2024-04-25T05:40:43.442114+0000","last_active":"2024-04-25T05:42:16.228499+0000","last_peered":"2024-04-25T05:42:16.228499+0000","last_clean":"2024-04-25T05:42:16.228499+0000","last_became_active":"2024-04-25T05:40:43.441816+0000","last_became_peered":"2024-04-25T05:40:43.441816+0000","last_unstale":"2024-04-25T05:42:16.228499+0000","last_undegraded":"2024-04-25T05:42:16.228499+0000","last_fullsized":"2024-04-25T05:42:16.228499+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"277'12518","reported_seq":12687,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.360900+0000","last_change":"2024-04-25T05:39:19.228876+0000","last_active":"2024-04-25T05:42:23.360900+0000","last_peered":"2024-04-25T05:42:23.360900+0000","last_clean":"2024-04-25T05:42:23.360900+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:42:23.360900+0000","last_undegraded":"2024-04-25T05:42:23.360900+0000","last_fullsized":"2024-04-25T05:42:23.360900+0000","mapping_epoch":198,"log_start":"247'8397","ondisk_log_start":"247'8397","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":8118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":765,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.228931+0000","last_change":"2024-04-25T05:41:44.999875+0000","last_active":"2024-04-25T05:42:16.228931+0000","last_peered":"2024-04-25T05:42:16.228931+0000","last_clean":"2024-04-25T05:42:16.228931+0000","last_became_active":"2024-04-25T05:40:45.848289+0000","last_became_peered":"2024-04-25T05:40:45.848289+0000","last_unstale":"2024-04-25T05:42:16.228931+0000","last_undegraded":"2024-04-25T05:42:16.228931+0000","last_fullsized":"2024-04-25T05:42:16.228931+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.306297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229354+0000","last_change":"2024-04-25T05:40:52.973425+0000","last_active":"2024-04-25T05:42:16.229354+0000","last_peered":"2024-04-25T05:42:16.229354+0000","last_clean":"2024-04-25T05:42:16.229354+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:42:16.229354+0000","last_undegraded":"2024-04-25T05:42:16.229354+0000","last_fullsized":"2024-04-25T05:42:16.229354+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:52.973302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.973302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":483,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237679+0000","last_change":"2024-04-25T05:40:55.966682+0000","last_active":"2024-04-25T05:42:16.237679+0000","last_peered":"2024-04-25T05:42:16.237679+0000","last_clean":"2024-04-25T05:42:16.237679+0000","last_became_active":"2024-04-25T05:40:55.966367+0000","last_became_peered":"2024-04-25T05:40:55.966367+0000","last_unstale":"2024-04-25T05:42:16.237679+0000","last_undegraded":"2024-04-25T05:42:16.237679+0000","last_fullsized":"2024-04-25T05:42:16.237679+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:42.474808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:42.474808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235199+0000","last_change":"2024-04-25T05:40:55.991779+0000","last_active":"2024-04-25T05:42:16.235199+0000","last_peered":"2024-04-25T05:42:16.235199+0000","last_clean":"2024-04-25T05:42:16.235199+0000","last_became_active":"2024-04-25T05:40:55.990929+0000","last_became_peered":"2024-04-25T05:40:55.990929+0000","last_unstale":"2024-04-25T05:42:16.235199+0000","last_undegraded":"2024-04-25T05:42:16.235199+0000","last_fullsized":"2024-04-25T05:42:16.235199+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.764632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.764632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":531,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240880+0000","last_change":"2024-04-25T05:40:43.477675+0000","last_active":"2024-04-25T05:42:16.240880+0000","last_peered":"2024-04-25T05:42:16.240880+0000","last_clean":"2024-04-25T05:42:16.240880+0000","last_became_active":"2024-04-25T05:40:43.450842+0000","last_became_peered":"2024-04-25T05:40:43.450842+0000","last_unstale":"2024-04-25T05:42:16.240880+0000","last_undegraded":"2024-04-25T05:42:16.240880+0000","last_fullsized":"2024-04-25T05:42:16.240880+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.592911+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":84,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237972+0000","last_change":"2024-04-25T05:40:55.910414+0000","last_active":"2024-04-25T05:42:16.237972+0000","last_peered":"2024-04-25T05:42:16.237972+0000","last_clean":"2024-04-25T05:42:16.237972+0000","last_became_active":"2024-04-25T05:40:55.910074+0000","last_became_peered":"2024-04-25T05:40:55.910074+0000","last_unstale":"2024-04-25T05:42:16.237972+0000","last_undegraded":"2024-04-25T05:42:16.237972+0000","last_fullsized":"2024-04-25T05:42:16.237972+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.980299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.980299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"277'12648","reported_seq":27793,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.767330+0000","last_change":"2024-04-25T05:41:32.354405+0000","last_active":"2024-04-25T05:42:22.767330+0000","last_peered":"2024-04-25T05:42:22.767330+0000","last_clean":"2024-04-25T05:42:22.767330+0000","last_became_active":"2024-04-25T05:41:32.341701+0000","last_became_peered":"2024-04-25T05:41:32.341701+0000","last_unstale":"2024-04-25T05:42:22.767330+0000","last_undegraded":"2024-04-25T05:42:22.767330+0000","last_fullsized":"2024-04-25T05:42:22.767330+0000","mapping_epoch":258,"log_start":"245'7927","ondisk_log_start":"245'7927","created":198,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6983","last_scrub_stamp":"2024-04-25T05:40:41.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.157014+0000","objects_scrubbed":6915,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12648,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":487325696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":446,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239120+0000","last_change":"2024-04-25T05:42:15.011589+0000","last_active":"2024-04-25T05:42:16.239120+0000","last_peered":"2024-04-25T05:42:16.239120+0000","last_clean":"2024-04-25T05:42:16.239120+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:42:16.239120+0000","last_undegraded":"2024-04-25T05:42:16.239120+0000","last_fullsized":"2024-04-25T05:42:16.239120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:16.656094+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":767,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:18.235895+0000","last_change":"2024-04-25T05:42:18.235935+0000","last_active":"2024-04-25T05:42:18.235895+0000","last_peered":"2024-04-25T05:42:18.235895+0000","last_clean":"2024-04-25T05:42:18.235895+0000","last_became_active":"2024-04-25T05:40:45.847131+0000","last_became_peered":"2024-04-25T05:40:45.847131+0000","last_unstale":"2024-04-25T05:42:18.235895+0000","last_undegraded":"2024-04-25T05:42:18.235895+0000","last_fullsized":"2024-04-25T05:42:18.235895+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.800850+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":463,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239279+0000","last_change":"2024-04-25T05:42:08.356121+0000","last_active":"2024-04-25T05:42:16.239279+0000","last_peered":"2024-04-25T05:42:16.239279+0000","last_clean":"2024-04-25T05:42:16.239279+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:42:16.239279+0000","last_undegraded":"2024-04-25T05:42:16.239279+0000","last_fullsized":"2024-04-25T05:42:16.239279+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:10.802129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":218,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.233296+0000","last_change":"2024-04-25T05:40:40.957968+0000","last_active":"2024-04-25T05:42:16.233296+0000","last_peered":"2024-04-25T05:42:16.233296+0000","last_clean":"2024-04-25T05:42:16.233296+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:42:16.233296+0000","last_undegraded":"2024-04-25T05:42:16.233296+0000","last_fullsized":"2024-04-25T05:42:16.233296+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:40:40.957665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.957665+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":457,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229003+0000","last_change":"2024-04-25T05:41:20.125777+0000","last_active":"2024-04-25T05:42:16.229003+0000","last_peered":"2024-04-25T05:42:16.229003+0000","last_clean":"2024-04-25T05:42:16.229003+0000","last_became_active":"2024-04-25T05:40:55.953691+0000","last_became_peered":"2024-04-25T05:40:55.953691+0000","last_unstale":"2024-04-25T05:42:16.229003+0000","last_undegraded":"2024-04-25T05:42:16.229003+0000","last_fullsized":"2024-04-25T05:42:16.229003+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:20.125501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:41:20.125501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.894302+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"277'12878","reported_seq":23429,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.370629+0000","last_change":"2024-04-25T05:42:07.903446+0000","last_active":"2024-04-25T05:42:23.370629+0000","last_peered":"2024-04-25T05:42:23.370629+0000","last_clean":"2024-04-25T05:42:23.370629+0000","last_became_active":"2024-04-25T05:42:07.871807+0000","last_became_peered":"2024-04-25T05:42:07.871807+0000","last_unstale":"2024-04-25T05:42:23.370629+0000","last_undegraded":"2024-04-25T05:42:23.370629+0000","last_fullsized":"2024-04-25T05:42:23.370629+0000","mapping_epoch":271,"log_start":"247'8533","ondisk_log_start":"247'8533","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7666","last_scrub_stamp":"2024-04-25T05:40:52.458924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.458924+0000","objects_scrubbed":7586,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181927936,"num_objects":2776,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12878,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5139,"num_bytes_recovered":336789504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":640,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.242095+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:42:16.242095+0000","last_peered":"2024-04-25T05:42:16.242095+0000","last_clean":"2024-04-25T05:42:16.242095+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:42:16.242095+0000","last_undegraded":"2024-04-25T05:42:16.242095+0000","last_fullsized":"2024-04-25T05:42:16.242095+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.357654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":446,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239396+0000","last_change":"2024-04-25T05:40:47.469910+0000","last_active":"2024-04-25T05:42:16.239396+0000","last_peered":"2024-04-25T05:42:16.239396+0000","last_clean":"2024-04-25T05:42:16.239396+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:42:16.239396+0000","last_undegraded":"2024-04-25T05:42:16.239396+0000","last_fullsized":"2024-04-25T05:42:16.239396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.226752+0000","last_change":"2024-04-25T05:40:43.435255+0000","last_active":"2024-04-25T05:42:16.226752+0000","last_peered":"2024-04-25T05:42:16.226752+0000","last_clean":"2024-04-25T05:42:16.226752+0000","last_became_active":"2024-04-25T05:40:43.434912+0000","last_became_peered":"2024-04-25T05:40:43.434912+0000","last_unstale":"2024-04-25T05:42:16.226752+0000","last_undegraded":"2024-04-25T05:42:16.226752+0000","last_fullsized":"2024-04-25T05:42:16.226752+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.785355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.785355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":987,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:21.608416+0000","last_change":"2024-04-25T05:41:36.018017+0000","last_active":"2024-04-25T05:42:21.608416+0000","last_peered":"2024-04-25T05:42:21.608416+0000","last_clean":"2024-04-25T05:42:21.608416+0000","last_became_active":"2024-04-25T05:40:43.431415+0000","last_became_peered":"2024-04-25T05:40:43.431415+0000","last_unstale":"2024-04-25T05:42:21.608416+0000","last_undegraded":"2024-04-25T05:42:21.608416+0000","last_fullsized":"2024-04-25T05:42:21.608416+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:41:36.017857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.017857+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:39.489209+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":491,"num_read_kb":486,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":285,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237114+0000","last_change":"2024-04-25T05:40:12.370708+0000","last_active":"2024-04-25T05:42:16.237114+0000","last_peered":"2024-04-25T05:42:16.237114+0000","last_clean":"2024-04-25T05:42:16.237114+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:42:16.237114+0000","last_undegraded":"2024-04-25T05:42:16.237114+0000","last_fullsized":"2024-04-25T05:42:16.237114+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239523+0000","last_change":"2024-04-25T05:40:55.418588+0000","last_active":"2024-04-25T05:42:16.239523+0000","last_peered":"2024-04-25T05:42:16.239523+0000","last_clean":"2024-04-25T05:42:16.239523+0000","last_became_active":"2024-04-25T05:40:43.454300+0000","last_became_peered":"2024-04-25T05:40:43.454300+0000","last_unstale":"2024-04-25T05:42:16.239523+0000","last_undegraded":"2024-04-25T05:42:16.239523+0000","last_fullsized":"2024-04-25T05:42:16.239523+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:55.418499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.418499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"277'13026","reported_seq":26199,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.402036+0000","last_change":"2024-04-25T05:41:48.020080+0000","last_active":"2024-04-25T05:42:23.402036+0000","last_peered":"2024-04-25T05:42:23.402036+0000","last_clean":"2024-04-25T05:42:23.402036+0000","last_became_active":"2024-04-25T05:41:47.997545+0000","last_became_peered":"2024-04-25T05:41:47.997545+0000","last_unstale":"2024-04-25T05:42:23.402036+0000","last_undegraded":"2024-04-25T05:42:23.402036+0000","last_fullsized":"2024-04-25T05:42:23.402036+0000","mapping_epoch":262,"log_start":"248'8892","ondisk_log_start":"248'8892","created":198,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13026,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6494,"num_bytes_recovered":425590784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":638,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.233285+0000","last_change":"2024-04-25T05:40:45.845890+0000","last_active":"2024-04-25T05:42:16.233285+0000","last_peered":"2024-04-25T05:42:16.233285+0000","last_clean":"2024-04-25T05:42:16.233285+0000","last_became_active":"2024-04-25T05:40:45.845535+0000","last_became_peered":"2024-04-25T05:40:45.845535+0000","last_unstale":"2024-04-25T05:42:16.233285+0000","last_undegraded":"2024-04-25T05:42:16.233285+0000","last_fullsized":"2024-04-25T05:42:16.233285+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:02.832193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.832193+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":464,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.119346+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:42:23.119346+0000","last_peered":"2024-04-25T05:42:23.119346+0000","last_clean":"2024-04-25T05:42:23.119346+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:42:23.119346+0000","last_undegraded":"2024-04-25T05:42:23.119346+0000","last_fullsized":"2024-04-25T05:42:23.119346+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.278820+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":465,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239720+0000","last_change":"2024-04-25T05:40:55.977907+0000","last_active":"2024-04-25T05:42:16.239720+0000","last_peered":"2024-04-25T05:42:16.239720+0000","last_clean":"2024-04-25T05:42:16.239720+0000","last_became_active":"2024-04-25T05:40:55.977574+0000","last_became_peered":"2024-04-25T05:40:55.977574+0000","last_unstale":"2024-04-25T05:42:16.239720+0000","last_undegraded":"2024-04-25T05:42:16.239720+0000","last_fullsized":"2024-04-25T05:42:16.239720+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:16.651620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:16.651620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":127,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.674279+0000","last_change":"2024-04-25T05:42:22.674343+0000","last_active":"2024-04-25T05:42:22.674279+0000","last_peered":"2024-04-25T05:42:22.674279+0000","last_clean":"2024-04-25T05:42:22.674279+0000","last_became_active":"2024-04-25T05:40:43.446526+0000","last_became_peered":"2024-04-25T05:40:43.446526+0000","last_unstale":"2024-04-25T05:42:22.674279+0000","last_undegraded":"2024-04-25T05:42:22.674279+0000","last_fullsized":"2024-04-25T05:42:22.674279+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:34.867005+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":180,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231079+0000","last_change":"2024-04-25T05:42:14.225323+0000","last_active":"2024-04-25T05:42:16.231079+0000","last_peered":"2024-04-25T05:42:16.231079+0000","last_clean":"2024-04-25T05:42:16.231079+0000","last_became_active":"2024-04-25T05:40:43.429667+0000","last_became_peered":"2024-04-25T05:40:43.429667+0000","last_unstale":"2024-04-25T05:42:16.231079+0000","last_undegraded":"2024-04-25T05:42:16.231079+0000","last_fullsized":"2024-04-25T05:42:16.231079+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.225203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.225203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.751205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"277'12545","reported_seq":21651,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.267078+0000","last_change":"2024-04-25T05:42:12.209425+0000","last_active":"2024-04-25T05:42:23.267078+0000","last_peered":"2024-04-25T05:42:23.267078+0000","last_clean":"2024-04-25T05:42:23.267078+0000","last_became_active":"2024-04-25T05:42:12.177948+0000","last_became_peered":"2024-04-25T05:42:12.177948+0000","last_unstale":"2024-04-25T05:42:23.267078+0000","last_undegraded":"2024-04-25T05:42:23.267078+0000","last_fullsized":"2024-04-25T05:42:23.267078+0000","mapping_epoch":274,"log_start":"245'7839","ondisk_log_start":"245'7839","created":198,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":999,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181862400,"num_objects":2775,"num_object_clones":0,"num_object_copies":8325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":490240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4450,"num_bytes_recovered":291635200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":629,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.232857+0000","last_change":"2024-04-25T05:41:31.093059+0000","last_active":"2024-04-25T05:42:16.232857+0000","last_peered":"2024-04-25T05:42:16.232857+0000","last_clean":"2024-04-25T05:42:16.232857+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:42:16.232857+0000","last_undegraded":"2024-04-25T05:42:16.232857+0000","last_fullsized":"2024-04-25T05:42:16.232857+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:31.092923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.092923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.069935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":113,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229218+0000","last_change":"2024-04-25T05:42:14.731871+0000","last_active":"2024-04-25T05:42:16.229218+0000","last_peered":"2024-04-25T05:42:16.229218+0000","last_clean":"2024-04-25T05:42:16.229218+0000","last_became_active":"2024-04-25T05:40:55.970110+0000","last_became_peered":"2024-04-25T05:40:55.970110+0000","last_unstale":"2024-04-25T05:42:16.229218+0000","last_undegraded":"2024-04-25T05:42:16.229218+0000","last_fullsized":"2024-04-25T05:42:16.229218+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.632461+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":457,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239549+0000","last_change":"2024-04-25T05:40:55.975334+0000","last_active":"2024-04-25T05:42:16.239549+0000","last_peered":"2024-04-25T05:42:16.239549+0000","last_clean":"2024-04-25T05:42:16.239549+0000","last_became_active":"2024-04-25T05:40:55.974871+0000","last_became_peered":"2024-04-25T05:40:55.974871+0000","last_unstale":"2024-04-25T05:42:16.239549+0000","last_undegraded":"2024-04-25T05:42:16.239549+0000","last_fullsized":"2024-04-25T05:42:16.239549+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":105,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:18.092148+0000","last_change":"2024-04-25T05:42:18.092205+0000","last_active":"2024-04-25T05:42:18.092148+0000","last_peered":"2024-04-25T05:42:18.092148+0000","last_clean":"2024-04-25T05:42:18.092148+0000","last_became_active":"2024-04-25T05:40:55.964721+0000","last_became_peered":"2024-04-25T05:40:55.964721+0000","last_unstale":"2024-04-25T05:42:18.092148+0000","last_undegraded":"2024-04-25T05:42:18.092148+0000","last_fullsized":"2024-04-25T05:42:18.092148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:30.000606+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"233'12","reported_seq":123,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.231368+0000","last_change":"2024-04-25T05:40:43.475139+0000","last_active":"2024-04-25T05:42:16.231368+0000","last_peered":"2024-04-25T05:42:16.231368+0000","last_clean":"2024-04-25T05:42:16.231368+0000","last_became_active":"2024-04-25T05:40:43.445833+0000","last_became_peered":"2024-04-25T05:40:43.445833+0000","last_unstale":"2024-04-25T05:42:16.231368+0000","last_undegraded":"2024-04-25T05:42:16.231368+0000","last_fullsized":"2024-04-25T05:42:16.231368+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:40:06.822122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.822122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"277'12756","reported_seq":27001,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.820573+0000","last_change":"2024-04-25T05:41:28.231423+0000","last_active":"2024-04-25T05:42:22.820573+0000","last_peered":"2024-04-25T05:42:22.820573+0000","last_clean":"2024-04-25T05:42:22.820573+0000","last_became_active":"2024-04-25T05:41:28.213582+0000","last_became_peered":"2024-04-25T05:41:28.213582+0000","last_unstale":"2024-04-25T05:42:22.820573+0000","last_undegraded":"2024-04-25T05:42:22.820573+0000","last_fullsized":"2024-04-25T05:42:22.820573+0000","mapping_epoch":255,"log_start":"246'8329","ondisk_log_start":"246'8329","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4427,"log_dups_size":0,"ondisk_log_size":4427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184352792,"num_objects":2814,"num_object_clones":0,"num_object_copies":8442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12756,"num_write_kb":498177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7030,"num_bytes_recovered":460652568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":625,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:22.285343+0000","last_change":"2024-04-25T05:42:22.285373+0000","last_active":"2024-04-25T05:42:22.285343+0000","last_peered":"2024-04-25T05:42:22.285343+0000","last_clean":"2024-04-25T05:42:22.285343+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:42:22.285343+0000","last_undegraded":"2024-04-25T05:42:22.285343+0000","last_fullsized":"2024-04-25T05:42:22.285343+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:29.371504+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229209+0000","last_change":"2024-04-25T05:40:55.981504+0000","last_active":"2024-04-25T05:42:16.229209+0000","last_peered":"2024-04-25T05:42:16.229209+0000","last_clean":"2024-04-25T05:42:16.229209+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:42:16.229209+0000","last_undegraded":"2024-04-25T05:42:16.229209+0000","last_fullsized":"2024-04-25T05:42:16.229209+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.781719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.781719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":103,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.229489+0000","last_change":"2024-04-25T05:40:43.463116+0000","last_active":"2024-04-25T05:42:16.229489+0000","last_peered":"2024-04-25T05:42:16.229489+0000","last_clean":"2024-04-25T05:42:16.229489+0000","last_became_active":"2024-04-25T05:40:43.443728+0000","last_became_peered":"2024-04-25T05:40:43.443728+0000","last_unstale":"2024-04-25T05:42:16.229489+0000","last_undegraded":"2024-04-25T05:42:16.229489+0000","last_fullsized":"2024-04-25T05:42:16.229489+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":181,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235129+0000","last_change":"2024-04-25T05:40:48.391768+0000","last_active":"2024-04-25T05:42:16.235129+0000","last_peered":"2024-04-25T05:42:16.235129+0000","last_clean":"2024-04-25T05:42:16.235129+0000","last_became_active":"2024-04-25T05:40:09.733866+0000","last_became_peered":"2024-04-25T05:40:09.733866+0000","last_unstale":"2024-04-25T05:42:16.235129+0000","last_undegraded":"2024-04-25T05:42:16.235129+0000","last_fullsized":"2024-04-25T05:42:16.235129+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:40:48.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.391383+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":552,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235370+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:42:16.235370+0000","last_peered":"2024-04-25T05:42:16.235370+0000","last_clean":"2024-04-25T05:42:16.235370+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:42:16.235370+0000","last_undegraded":"2024-04-25T05:42:16.235370+0000","last_fullsized":"2024-04-25T05:42:16.235370+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:17.034593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"277'12796","reported_seq":22992,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.273960+0000","last_change":"2024-04-25T05:42:06.908338+0000","last_active":"2024-04-25T05:42:23.273960+0000","last_peered":"2024-04-25T05:42:23.273960+0000","last_clean":"2024-04-25T05:42:23.273960+0000","last_became_active":"2024-04-25T05:42:06.870659+0000","last_became_peered":"2024-04-25T05:42:06.870659+0000","last_unstale":"2024-04-25T05:42:23.273960+0000","last_undegraded":"2024-04-25T05:42:23.273960+0000","last_fullsized":"2024-04-25T05:42:23.273960+0000","mapping_epoch":270,"log_start":"245'8068","ondisk_log_start":"245'8068","created":198,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12796,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5005,"num_bytes_recovered":328007680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":647,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867458+0000","last_change":"2024-04-25T05:40:58.920790+0000","last_active":"2024-04-25T05:42:24.867458+0000","last_peered":"2024-04-25T05:42:24.867458+0000","last_clean":"2024-04-25T05:42:24.867458+0000","last_became_active":"2024-04-25T05:40:58.920400+0000","last_became_peered":"2024-04-25T05:40:58.920400+0000","last_unstale":"2024-04-25T05:42:24.867458+0000","last_undegraded":"2024-04-25T05:42:24.867458+0000","last_fullsized":"2024-04-25T05:42:24.867458+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:37.793923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.793923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":127,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239553+0000","last_change":"2024-04-25T05:40:55.975104+0000","last_active":"2024-04-25T05:42:16.239553+0000","last_peered":"2024-04-25T05:42:16.239553+0000","last_clean":"2024-04-25T05:42:16.239553+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:42:16.239553+0000","last_undegraded":"2024-04-25T05:42:16.239553+0000","last_fullsized":"2024-04-25T05:42:16.239553+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.719427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.719427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238530+0000","last_change":"2024-04-25T05:40:43.516969+0000","last_active":"2024-04-25T05:42:16.238530+0000","last_peered":"2024-04-25T05:42:16.238530+0000","last_clean":"2024-04-25T05:42:16.238530+0000","last_became_active":"2024-04-25T05:40:43.440253+0000","last_became_peered":"2024-04-25T05:40:43.440253+0000","last_unstale":"2024-04-25T05:42:16.238530+0000","last_undegraded":"2024-04-25T05:42:16.238530+0000","last_fullsized":"2024-04-25T05:42:16.238530+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:43.516897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.516897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":559,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235280+0000","last_change":"2024-04-25T05:40:48.519454+0000","last_active":"2024-04-25T05:42:16.235280+0000","last_peered":"2024-04-25T05:42:16.235280+0000","last_clean":"2024-04-25T05:42:16.235280+0000","last_became_active":"2024-04-25T05:40:43.440689+0000","last_became_peered":"2024-04-25T05:40:43.440689+0000","last_unstale":"2024-04-25T05:42:16.235280+0000","last_undegraded":"2024-04-25T05:42:16.235280+0000","last_fullsized":"2024-04-25T05:42:16.235280+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:40:48.519288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.519288+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866729+0000","last_change":"2024-04-25T05:42:19.974041+0000","last_active":"2024-04-25T05:42:24.866729+0000","last_peered":"2024-04-25T05:42:24.866729+0000","last_clean":"2024-04-25T05:42:24.866729+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:42:24.866729+0000","last_undegraded":"2024-04-25T05:42:24.866729+0000","last_fullsized":"2024-04-25T05:42:24.866729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.973911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.973911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.078951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"277'12550","reported_seq":13504,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.370750+0000","last_change":"2024-04-25T05:42:21.320689+0000","last_active":"2024-04-25T05:42:23.370750+0000","last_peered":"2024-04-25T05:42:23.370750+0000","last_clean":"2024-04-25T05:42:23.370750+0000","last_became_active":"2024-04-25T05:39:34.780422+0000","last_became_peered":"2024-04-25T05:39:34.780422+0000","last_unstale":"2024-04-25T05:42:23.370750+0000","last_undegraded":"2024-04-25T05:42:23.370750+0000","last_fullsized":"2024-04-25T05:42:23.370750+0000","mapping_epoch":208,"log_start":"247'8169","ondisk_log_start":"247'8169","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'12380","last_scrub_stamp":"2024-04-25T05:42:21.320336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.320336+0000","objects_scrubbed":2986,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:44.979593+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":490368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":24117248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":635,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237958+0000","last_change":"2024-04-25T05:40:58.924826+0000","last_active":"2024-04-25T05:42:16.237958+0000","last_peered":"2024-04-25T05:42:16.237958+0000","last_clean":"2024-04-25T05:42:16.237958+0000","last_became_active":"2024-04-25T05:40:58.923753+0000","last_became_peered":"2024-04-25T05:40:58.923753+0000","last_unstale":"2024-04-25T05:42:16.237958+0000","last_undegraded":"2024-04-25T05:42:16.237958+0000","last_fullsized":"2024-04-25T05:42:16.237958+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":458,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239218+0000","last_change":"2024-04-25T05:40:50.460093+0000","last_active":"2024-04-25T05:42:16.239218+0000","last_peered":"2024-04-25T05:42:16.239218+0000","last_clean":"2024-04-25T05:42:16.239218+0000","last_became_active":"2024-04-25T05:40:43.454117+0000","last_became_peered":"2024-04-25T05:40:43.454117+0000","last_unstale":"2024-04-25T05:42:16.239218+0000","last_undegraded":"2024-04-25T05:42:16.239218+0000","last_fullsized":"2024-04-25T05:42:16.239218+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"233'11","reported_seq":190,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237753+0000","last_change":"2024-04-25T05:41:38.552749+0000","last_active":"2024-04-25T05:42:16.237753+0000","last_peered":"2024-04-25T05:42:16.237753+0000","last_clean":"2024-04-25T05:42:16.237753+0000","last_became_active":"2024-04-25T05:40:43.453268+0000","last_became_peered":"2024-04-25T05:40:43.453268+0000","last_unstale":"2024-04-25T05:42:16.237753+0000","last_undegraded":"2024-04-25T05:42:16.237753+0000","last_fullsized":"2024-04-25T05:42:16.237753+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:52.068035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":450,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.235180+0000","last_change":"2024-04-25T05:40:51.592334+0000","last_active":"2024-04-25T05:42:16.235180+0000","last_peered":"2024-04-25T05:42:16.235180+0000","last_clean":"2024-04-25T05:42:16.235180+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:42:16.235180+0000","last_undegraded":"2024-04-25T05:42:16.235180+0000","last_fullsized":"2024-04-25T05:42:16.235180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.592111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.592111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":478,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240981+0000","last_change":"2024-04-25T05:39:39.713342+0000","last_active":"2024-04-25T05:42:16.240981+0000","last_peered":"2024-04-25T05:42:16.240981+0000","last_clean":"2024-04-25T05:42:16.240981+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:42:16.240981+0000","last_undegraded":"2024-04-25T05:42:16.240981+0000","last_fullsized":"2024-04-25T05:42:16.240981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.713165+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"279'13026","reported_seq":27563,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.753917+0000","last_change":"2024-04-25T05:41:28.229751+0000","last_active":"2024-04-25T05:42:27.753917+0000","last_peered":"2024-04-25T05:42:27.753917+0000","last_clean":"2024-04-25T05:42:27.753917+0000","last_became_active":"2024-04-25T05:41:28.214541+0000","last_became_peered":"2024-04-25T05:41:28.214541+0000","last_unstale":"2024-04-25T05:42:27.753917+0000","last_undegraded":"2024-04-25T05:42:27.753917+0000","last_fullsized":"2024-04-25T05:42:27.753917+0000","mapping_epoch":255,"log_start":"250'8718","ondisk_log_start":"250'8718","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6914","last_scrub_stamp":"2024-04-25T05:40:40.935916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.935916+0000","objects_scrubbed":6834,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.992837+0000","scrub_duration":2117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157024256,"num_objects":2396,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13026,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7133,"num_bytes_recovered":467468288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":442,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237262+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:42:16.237262+0000","last_peered":"2024-04-25T05:42:16.237262+0000","last_clean":"2024-04-25T05:42:16.237262+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:42:16.237262+0000","last_undegraded":"2024-04-25T05:42:16.237262+0000","last_fullsized":"2024-04-25T05:42:16.237262+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.239432+0000","last_change":"2024-04-25T05:41:27.501946+0000","last_active":"2024-04-25T05:42:16.239432+0000","last_peered":"2024-04-25T05:42:16.239432+0000","last_clean":"2024-04-25T05:42:16.239432+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:42:16.239432+0000","last_undegraded":"2024-04-25T05:42:16.239432+0000","last_fullsized":"2024-04-25T05:42:16.239432+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.501828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.501828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.447143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866814+0000","last_change":"2024-04-25T05:40:55.964874+0000","last_active":"2024-04-25T05:42:24.866814+0000","last_peered":"2024-04-25T05:42:24.866814+0000","last_clean":"2024-04-25T05:42:24.866814+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:42:24.866814+0000","last_undegraded":"2024-04-25T05:42:24.866814+0000","last_fullsized":"2024-04-25T05:42:24.866814+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:39.870293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:39.870293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":466,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.232476+0000","last_change":"2024-04-25T05:40:09.839273+0000","last_active":"2024-04-25T05:42:16.232476+0000","last_peered":"2024-04-25T05:42:16.232476+0000","last_clean":"2024-04-25T05:42:16.232476+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:42:16.232476+0000","last_undegraded":"2024-04-25T05:42:16.232476+0000","last_fullsized":"2024-04-25T05:42:16.232476+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:09.838661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.838661+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"277'13034","reported_seq":23779,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.358800+0000","last_change":"2024-04-25T05:42:15.166676+0000","last_active":"2024-04-25T05:42:23.358800+0000","last_peered":"2024-04-25T05:42:23.358800+0000","last_clean":"2024-04-25T05:42:23.358800+0000","last_became_active":"2024-04-25T05:42:07.869166+0000","last_became_peered":"2024-04-25T05:42:07.869166+0000","last_unstale":"2024-04-25T05:42:23.358800+0000","last_undegraded":"2024-04-25T05:42:23.358800+0000","last_fullsized":"2024-04-25T05:42:23.358800+0000","mapping_epoch":271,"log_start":"247'8652","ondisk_log_start":"247'8652","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'12381","last_scrub_stamp":"2024-04-25T05:42:15.166473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.166473+0000","objects_scrubbed":3559,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:26.495367+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186908672,"num_objects":2852,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13034,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5255,"num_bytes_recovered":344391680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":630,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238007+0000","last_change":"2024-04-25T05:40:58.929257+0000","last_active":"2024-04-25T05:42:16.238007+0000","last_peered":"2024-04-25T05:42:16.238007+0000","last_clean":"2024-04-25T05:42:16.238007+0000","last_became_active":"2024-04-25T05:40:58.926534+0000","last_became_peered":"2024-04-25T05:40:58.926534+0000","last_unstale":"2024-04-25T05:42:16.238007+0000","last_undegraded":"2024-04-25T05:42:16.238007+0000","last_fullsized":"2024-04-25T05:42:16.238007+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":85,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.238397+0000","last_change":"2024-04-25T05:40:55.974556+0000","last_active":"2024-04-25T05:42:16.238397+0000","last_peered":"2024-04-25T05:42:16.238397+0000","last_clean":"2024-04-25T05:42:16.238397+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:42:16.238397+0000","last_undegraded":"2024-04-25T05:42:16.238397+0000","last_fullsized":"2024-04-25T05:42:16.238397+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.019675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.019675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":468,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240010+0000","last_change":"2024-04-25T05:40:44.537276+0000","last_active":"2024-04-25T05:42:16.240010+0000","last_peered":"2024-04-25T05:42:16.240010+0000","last_clean":"2024-04-25T05:42:16.240010+0000","last_became_active":"2024-04-25T05:40:43.449807+0000","last_became_peered":"2024-04-25T05:40:43.449807+0000","last_unstale":"2024-04-25T05:42:16.240010+0000","last_undegraded":"2024-04-25T05:42:16.240010+0000","last_fullsized":"2024-04-25T05:42:16.240010+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":174,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:19.197545+0000","last_change":"2024-04-25T05:42:19.197580+0000","last_active":"2024-04-25T05:42:19.197545+0000","last_peered":"2024-04-25T05:42:19.197545+0000","last_clean":"2024-04-25T05:42:19.197545+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:42:19.197545+0000","last_undegraded":"2024-04-25T05:42:19.197545+0000","last_fullsized":"2024-04-25T05:42:19.197545+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.197467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.197467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:47.403991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":149,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.240271+0000","last_change":"2024-04-25T05:40:56.017339+0000","last_active":"2024-04-25T05:42:16.240271+0000","last_peered":"2024-04-25T05:42:16.240271+0000","last_clean":"2024-04-25T05:42:16.240271+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:42:16.240271+0000","last_undegraded":"2024-04-25T05:42:16.240271+0000","last_fullsized":"2024-04-25T05:42:16.240271+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:34.605704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.605704+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"277'12640","reported_seq":12809,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:23.414109+0000","last_change":"2024-04-25T05:39:19.231808+0000","last_active":"2024-04-25T05:42:23.414109+0000","last_peered":"2024-04-25T05:42:23.414109+0000","last_clean":"2024-04-25T05:42:23.414109+0000","last_became_active":"2024-04-25T05:39:19.231525+0000","last_became_peered":"2024-04-25T05:39:19.231525+0000","last_unstale":"2024-04-25T05:42:23.414109+0000","last_undegraded":"2024-04-25T05:42:23.414109+0000","last_fullsized":"2024-04-25T05:42:23.414109+0000","mapping_epoch":198,"log_start":"247'8451","ondisk_log_start":"247'8451","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176947200,"num_objects":2700,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12640,"num_write_kb":490880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":765,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:20.218703+0000","last_change":"2024-04-25T05:42:20.218748+0000","last_active":"2024-04-25T05:42:20.218703+0000","last_peered":"2024-04-25T05:42:20.218703+0000","last_clean":"2024-04-25T05:42:20.218703+0000","last_became_active":"2024-04-25T05:40:58.930089+0000","last_became_peered":"2024-04-25T05:40:58.930089+0000","last_unstale":"2024-04-25T05:42:20.218703+0000","last_undegraded":"2024-04-25T05:42:20.218703+0000","last_fullsized":"2024-04-25T05:42:20.218703+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.920803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":126,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.242354+0000","last_change":"2024-04-25T05:40:55.964673+0000","last_active":"2024-04-25T05:42:16.242354+0000","last_peered":"2024-04-25T05:42:16.242354+0000","last_clean":"2024-04-25T05:42:16.242354+0000","last_became_active":"2024-04-25T05:40:55.964302+0000","last_became_peered":"2024-04-25T05:40:55.964302+0000","last_unstale":"2024-04-25T05:42:16.242354+0000","last_undegraded":"2024-04-25T05:42:16.242354+0000","last_fullsized":"2024-04-25T05:42:16.242354+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:02.813078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.813078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"243'10","reported_seq":211,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:21.213519+0000","last_change":"2024-04-25T05:42:21.213565+0000","last_active":"2024-04-25T05:42:21.213519+0000","last_peered":"2024-04-25T05:42:21.213519+0000","last_clean":"2024-04-25T05:42:21.213519+0000","last_became_active":"2024-04-25T05:40:55.955721+0000","last_became_peered":"2024-04-25T05:40:55.955721+0000","last_unstale":"2024-04-25T05:42:21.213519+0000","last_undegraded":"2024-04-25T05:42:21.213519+0000","last_fullsized":"2024-04-25T05:42:21.213519+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.386980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":462,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:17.311372+0000","last_change":"2024-04-25T05:42:17.311411+0000","last_active":"2024-04-25T05:42:17.311372+0000","last_peered":"2024-04-25T05:42:17.311372+0000","last_clean":"2024-04-25T05:42:17.311372+0000","last_became_active":"2024-04-25T05:40:43.432950+0000","last_became_peered":"2024-04-25T05:40:43.432950+0000","last_unstale":"2024-04-25T05:42:17.311372+0000","last_undegraded":"2024-04-25T05:42:17.311372+0000","last_fullsized":"2024-04-25T05:42:17.311372+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.857445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":162,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-04-25T05:42:16.237567+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:42:16.237567+0000","last_peered":"2024-04-25T05:42:16.237567+0000","last_clean":"2024-04-25T05:42:16.237567+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:42:16.237567+0000","last_undegraded":"2024-04-25T05:42:16.237567+0000","last_fullsized":"2024-04-25T05:42:16.237567+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.683068+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2894463000,"num_objects":44167,"num_object_clones":0,"num_object_copies":132501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44167,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204077,"num_write_kb":7943745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63646,"num_bytes_recovered":4170973208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9895751680,"data_stored":9895739464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71057,"ondisk_log_size":71057,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":491,"num_read_kb":486,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":974,"num_read_kb":699,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":5396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":652,"ondisk_log_size":652,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070570,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1416148,"kb_used_data":1105932,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92329004,"statfs":{"total":95995035648,"available":94544900096,"internally_reserved":0,"allocated":1132474368,"data_stored":1131981832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317651402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364910,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1620116,"kb_used_data":1291788,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92125036,"statfs":{"total":95995035648,"available":94336036864,"internally_reserved":0,"allocated":1322790912,"data_stored":1322298290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1642348,"kb_used_data":1245988,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92102804,"statfs":{"total":95995035648,"available":94313271296,"internally_reserved":0,"allocated":1275891712,"data_stored":1275373724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490000,"kb_used_data":1089480,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92255152,"statfs":{"total":95995035648,"available":94469275648,"internally_reserved":0,"allocated":1115627520,"data_stored":1115137543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2216864,"kb_used_data":1784472,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91528288,"statfs":{"total":95995035648,"available":93724966912,"internally_reserved":0,"allocated":1827299328,"data_stored":1826793804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":442759623},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986296,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1273964,"kb_used_data":959588,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92471188,"statfs":{"total":95995035648,"available":94690496512,"internally_reserved":0,"allocated":982618112,"data_stored":982113780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691955,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2224712,"kb_used_data":1896448,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91520440,"statfs":{"total":95995035648,"available":93716930560,"internally_reserved":0,"allocated":1941962752,"data_stored":1941481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1096744960,"data_stored":1096744960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1262616576,"data_stored":1262616576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":463273984,"data_stored":463273984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1820000256,"data_stored":1820000256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":954597376,"data_stored":954597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1891569664,"data_stored":1891565592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1297420288,"data_stored":1297416216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1109528576,"data_stored":1109524504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:42:29.845 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:42:29.845 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:42:29.845 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-04-25T05:42:29.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:42:29.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:29.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:29.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:29.943 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:42:29.943 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:42:29.943 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:42:29.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:42:29.943 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:42:29.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:29 smithi043 ceph-mon[37492]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:29.944 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:29 smithi043 ceph-mon[27010]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:29.944 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:29.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:30.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:30.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:30.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:30.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:29 smithi060 ceph-mon[30898]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:30.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:30.297 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:30.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:30.434 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:30.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:30.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:30.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:30.706 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:30.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:30.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:30.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:30.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:30.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:31.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:42:31.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:31.089 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:31.111 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:31.170 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:31.190 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:31.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:31.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:31.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:30 smithi060 ceph-mon[30898]: pgmap v443: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-04-25T05:42:31.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:30 smithi043 ceph-mon[37492]: pgmap v443: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-04-25T05:42:31.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:30 smithi043 ceph-mon[27010]: pgmap v443: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-04-25T05:42:31.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:31.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:31.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.528 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:31.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:31.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:31.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:31.789 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:31.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:31.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:31.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:32.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:32.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:32.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:32.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:32.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:32.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:32.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:32.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:32.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:32.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:32.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:32.884 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:32.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:32.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:33.018 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: pgmap v444: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s 2024-04-25T05:42:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-04-25T05:42:33.055 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]: dispatch 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: pgmap v444: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-04-25T05:42:33.056 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]: dispatch 2024-04-25T05:42:33.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:33.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: pgmap v444: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 851 op/s 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]: dispatch 2024-04-25T05:42:33.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]: dispatch 2024-04-25T05:42:33.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:33.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:33.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:33.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:33.454 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:42:33.458 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:42:33.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:33.564 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:33.577 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:42:33.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:33.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:33.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:33.840 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:33.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:33.979 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:33.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:34.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:34.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/706990686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: osdmap e280: 8 total, 8 up, 7 in 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3185398566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/706990686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: osdmap e280: 8 total, 8 up, 7 in 2024-04-25T05:42:34.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3185398566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:34.164 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:34.164 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:34.164 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:34.165 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:34.166 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:34.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/706990686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-04-25T05:42:34.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 6]}]': finished 2024-04-25T05:42:34.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.9", "id": [3, 1]}]': finished 2024-04-25T05:42:34.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:34.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: osdmap e280: 8 total, 8 up, 7 in 2024-04-25T05:42:34.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3185398566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.251 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:34.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:34.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:34.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:34.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:34.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:34.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:34.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:34.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:34.778 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:34.880 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:34.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:35.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:35.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:35.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:35.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[27010]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:35.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[27010]: pgmap v446: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:35.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[27010]: osdmap e281: 8 total, 8 up, 7 in 2024-04-25T05:42:35.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[37492]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:35.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[37492]: pgmap v446: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:35.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:34 smithi043 ceph-mon[37492]: osdmap e281: 8 total, 8 up, 7 in 2024-04-25T05:42:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.195 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:35.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:34 smithi060 ceph-mon[30898]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:42:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:34 smithi060 ceph-mon[30898]: pgmap v446: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:42:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:34 smithi060 ceph-mon[30898]: osdmap e281: 8 total, 8 up, 7 in 2024-04-25T05:42:35.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:35.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:35.529 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:35.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:35.600 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:35.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:35.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:35.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:36.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:36.023 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-04-25T05:42:31.827865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2475825715,"num_objects":38000,"num_object_clones":0,"num_object_copies":114000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38000,"num_whiteouts":0,"num_read":1488,"num_read_kb":1208,"num_write":211102,"num_write_kb":7943790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63961,"num_bytes_recovered":4171243691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73019,"ondisk_log_size":73019,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10743568,"kb_used_data":8194128,"kb_used_omap":10,"kb_used_meta":2549109,"kb_avail":645472496,"statfs":{"total":671965249536,"available":660963835904,"internally_reserved":0,"allocated":8390787072,"data_stored":8387237741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2610287723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,6,6,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-446234624,"num_objects":-6809,"num_object_clones":0,"num_object_copies":-20427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6809,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6809,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002877"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870678+0000","last_change":"2024-04-25T05:41:35.583008+0000","last_active":"2024-04-25T05:42:24.870678+0000","last_peered":"2024-04-25T05:42:24.870678+0000","last_clean":"2024-04-25T05:42:24.870678+0000","last_became_active":"2024-04-25T05:40:09.734217+0000","last_became_peered":"2024-04-25T05:40:09.734217+0000","last_unstale":"2024-04-25T05:42:24.870678+0000","last_undegraded":"2024-04-25T05:42:24.870678+0000","last_fullsized":"2024-04-25T05:42:24.870678+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:01.911520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867029+0000","last_change":"2024-04-25T05:40:37.473300+0000","last_active":"2024-04-25T05:42:24.867029+0000","last_peered":"2024-04-25T05:42:24.867029+0000","last_clean":"2024-04-25T05:42:24.867029+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:42:24.867029+0000","last_undegraded":"2024-04-25T05:42:24.867029+0000","last_fullsized":"2024-04-25T05:42:24.867029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:37.473018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.473018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":109,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874232+0000","last_change":"2024-04-25T05:40:55.954113+0000","last_active":"2024-04-25T05:42:24.874232+0000","last_peered":"2024-04-25T05:42:24.874232+0000","last_clean":"2024-04-25T05:42:24.874232+0000","last_became_active":"2024-04-25T05:40:55.953726+0000","last_became_peered":"2024-04-25T05:40:55.953726+0000","last_unstale":"2024-04-25T05:42:24.874232+0000","last_undegraded":"2024-04-25T05:42:24.874232+0000","last_fullsized":"2024-04-25T05:42:24.874232+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:07.379459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:07.379459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.673923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":526,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866440+0000","last_change":"2024-04-25T05:42:06.942759+0000","last_active":"2024-04-25T05:42:24.866440+0000","last_peered":"2024-04-25T05:42:24.866440+0000","last_clean":"2024-04-25T05:42:24.866440+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:42:24.866440+0000","last_undegraded":"2024-04-25T05:42:24.866440+0000","last_fullsized":"2024-04-25T05:42:24.866440+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:42:06.942600+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:42:06.942600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:25.919821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866673+0000","last_change":"2024-04-25T05:39:29.799097+0000","last_active":"2024-04-25T05:42:24.866673+0000","last_peered":"2024-04-25T05:42:24.866673+0000","last_clean":"2024-04-25T05:42:24.866673+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:42:24.866673+0000","last_undegraded":"2024-04-25T05:42:24.866673+0000","last_fullsized":"2024-04-25T05:42:24.866673+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:09.117207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:09.117207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":421,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870438+0000","last_change":"2024-04-25T05:40:13.865954+0000","last_active":"2024-04-25T05:42:24.870438+0000","last_peered":"2024-04-25T05:42:24.870438+0000","last_clean":"2024-04-25T05:42:24.870438+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:42:24.870438+0000","last_undegraded":"2024-04-25T05:42:24.870438+0000","last_fullsized":"2024-04-25T05:42:24.870438+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:13.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:13.865834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":473,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866578+0000","last_change":"2024-04-25T05:40:55.982555+0000","last_active":"2024-04-25T05:42:24.866578+0000","last_peered":"2024-04-25T05:42:24.866578+0000","last_clean":"2024-04-25T05:42:24.866578+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:42:24.866578+0000","last_undegraded":"2024-04-25T05:42:24.866578+0000","last_fullsized":"2024-04-25T05:42:24.866578+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.485053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":605,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870733+0000","last_change":"2024-04-25T05:40:56.000345+0000","last_active":"2024-04-25T05:42:24.870733+0000","last_peered":"2024-04-25T05:42:24.870733+0000","last_clean":"2024-04-25T05:42:24.870733+0000","last_became_active":"2024-04-25T05:40:55.974452+0000","last_became_peered":"2024-04-25T05:40:55.974452+0000","last_unstale":"2024-04-25T05:42:24.870733+0000","last_undegraded":"2024-04-25T05:42:24.870733+0000","last_fullsized":"2024-04-25T05:42:24.870733+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:39:41.219950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.219950+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874283+0000","last_change":"2024-04-25T05:40:43.441659+0000","last_active":"2024-04-25T05:42:24.874283+0000","last_peered":"2024-04-25T05:42:24.874283+0000","last_clean":"2024-04-25T05:42:24.874283+0000","last_became_active":"2024-04-25T05:40:43.441308+0000","last_became_peered":"2024-04-25T05:40:43.441308+0000","last_unstale":"2024-04-25T05:42:24.874283+0000","last_undegraded":"2024-04-25T05:42:24.874283+0000","last_fullsized":"2024-04-25T05:42:24.874283+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:59.924352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.924352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.673820+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872356+0000","last_change":"2024-04-25T05:40:43.453942+0000","last_active":"2024-04-25T05:42:24.872356+0000","last_peered":"2024-04-25T05:42:24.872356+0000","last_clean":"2024-04-25T05:42:24.872356+0000","last_became_active":"2024-04-25T05:40:43.453366+0000","last_became_peered":"2024-04-25T05:40:43.453366+0000","last_unstale":"2024-04-25T05:42:24.872356+0000","last_undegraded":"2024-04-25T05:42:24.872356+0000","last_fullsized":"2024-04-25T05:42:24.872356+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":474,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873234+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:42:24.873234+0000","last_peered":"2024-04-25T05:42:24.873234+0000","last_clean":"2024-04-25T05:42:24.873234+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:42:24.873234+0000","last_undegraded":"2024-04-25T05:42:24.873234+0000","last_fullsized":"2024-04-25T05:42:24.873234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":212,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867472+0000","last_change":"2024-04-25T05:42:19.252275+0000","last_active":"2024-04-25T05:42:24.867472+0000","last_peered":"2024-04-25T05:42:24.867472+0000","last_clean":"2024-04-25T05:42:24.867472+0000","last_became_active":"2024-04-25T05:40:43.433465+0000","last_became_peered":"2024-04-25T05:40:43.433465+0000","last_unstale":"2024-04-25T05:42:24.867472+0000","last_undegraded":"2024-04-25T05:42:24.867472+0000","last_fullsized":"2024-04-25T05:42:24.867472+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.890181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":450,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871568+0000","last_change":"2024-04-25T05:41:31.571478+0000","last_active":"2024-04-25T05:42:24.871568+0000","last_peered":"2024-04-25T05:42:24.871568+0000","last_clean":"2024-04-25T05:42:24.871568+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:42:24.871568+0000","last_undegraded":"2024-04-25T05:42:24.871568+0000","last_fullsized":"2024-04-25T05:42:24.871568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.853234+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.882804+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:42:24.882804+0000","last_peered":"2024-04-25T05:42:24.882804+0000","last_clean":"2024-04-25T05:42:24.882804+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:42:24.882804+0000","last_undegraded":"2024-04-25T05:42:24.882804+0000","last_fullsized":"2024-04-25T05:42:24.882804+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.487676+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874048+0000","last_change":"2024-04-25T05:40:49.444611+0000","last_active":"2024-04-25T05:42:24.874048+0000","last_peered":"2024-04-25T05:42:24.874048+0000","last_clean":"2024-04-25T05:42:24.874048+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:42:24.874048+0000","last_undegraded":"2024-04-25T05:42:24.874048+0000","last_fullsized":"2024-04-25T05:42:24.874048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:49.444492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.444492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":187,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866473+0000","last_change":"2024-04-25T05:42:13.374107+0000","last_active":"2024-04-25T05:42:24.866473+0000","last_peered":"2024-04-25T05:42:24.866473+0000","last_clean":"2024-04-25T05:42:24.866473+0000","last_became_active":"2024-04-25T05:40:55.984807+0000","last_became_peered":"2024-04-25T05:40:55.984807+0000","last_unstale":"2024-04-25T05:42:24.866473+0000","last_undegraded":"2024-04-25T05:42:24.866473+0000","last_fullsized":"2024-04-25T05:42:24.866473+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.061132+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867637+0000","last_change":"2024-04-25T05:42:24.062360+0000","last_active":"2024-04-25T05:42:24.867637+0000","last_peered":"2024-04-25T05:42:24.867637+0000","last_clean":"2024-04-25T05:42:24.867637+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:42:24.867637+0000","last_undegraded":"2024-04-25T05:42:24.867637+0000","last_fullsized":"2024-04-25T05:42:24.867637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.062132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.623846+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867534+0000","last_change":"2024-04-25T05:40:43.451998+0000","last_active":"2024-04-25T05:42:24.867534+0000","last_peered":"2024-04-25T05:42:24.867534+0000","last_clean":"2024-04-25T05:42:24.867534+0000","last_became_active":"2024-04-25T05:40:43.449712+0000","last_became_peered":"2024-04-25T05:40:43.449712+0000","last_unstale":"2024-04-25T05:42:24.867534+0000","last_undegraded":"2024-04-25T05:42:24.867534+0000","last_fullsized":"2024-04-25T05:42:24.867534+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:41.506653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.506653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.878290+0000","last_change":"2024-04-25T05:40:43.435522+0000","last_active":"2024-04-25T05:42:24.878290+0000","last_peered":"2024-04-25T05:42:24.878290+0000","last_clean":"2024-04-25T05:42:24.878290+0000","last_became_active":"2024-04-25T05:40:43.434830+0000","last_became_peered":"2024-04-25T05:40:43.434830+0000","last_unstale":"2024-04-25T05:42:24.878290+0000","last_undegraded":"2024-04-25T05:42:24.878290+0000","last_fullsized":"2024-04-25T05:42:24.878290+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":313,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867441+0000","last_change":"2024-04-25T05:40:56.033623+0000","last_active":"2024-04-25T05:42:24.867441+0000","last_peered":"2024-04-25T05:42:24.867441+0000","last_clean":"2024-04-25T05:42:24.867441+0000","last_became_active":"2024-04-25T05:40:55.991291+0000","last_became_peered":"2024-04-25T05:40:55.991291+0000","last_unstale":"2024-04-25T05:42:24.867441+0000","last_undegraded":"2024-04-25T05:42:24.867441+0000","last_fullsized":"2024-04-25T05:42:24.867441+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:39:42.756928+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.756928+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":447,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868138+0000","last_change":"2024-04-25T05:39:29.799140+0000","last_active":"2024-04-25T05:42:24.868138+0000","last_peered":"2024-04-25T05:42:24.868138+0000","last_clean":"2024-04-25T05:42:24.868138+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:42:24.868138+0000","last_undegraded":"2024-04-25T05:42:24.868138+0000","last_fullsized":"2024-04-25T05:42:24.868138+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.146572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.146572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":131,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867022+0000","last_change":"2024-04-25T05:40:55.973839+0000","last_active":"2024-04-25T05:42:24.867022+0000","last_peered":"2024-04-25T05:42:24.867022+0000","last_clean":"2024-04-25T05:42:24.867022+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:42:24.867022+0000","last_undegraded":"2024-04-25T05:42:24.867022+0000","last_fullsized":"2024-04-25T05:42:24.867022+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.632789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.632789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866758+0000","last_change":"2024-04-25T05:42:11.885667+0000","last_active":"2024-04-25T05:42:24.866758+0000","last_peered":"2024-04-25T05:42:24.866758+0000","last_clean":"2024-04-25T05:42:24.866758+0000","last_became_active":"2024-04-25T05:40:55.964565+0000","last_became_peered":"2024-04-25T05:40:55.964565+0000","last_unstale":"2024-04-25T05:42:24.866758+0000","last_undegraded":"2024-04-25T05:42:24.866758+0000","last_fullsized":"2024-04-25T05:42:24.866758+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:11.885545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:11.885545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.830666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":477,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865978+0000","last_change":"2024-04-25T05:40:15.536952+0000","last_active":"2024-04-25T05:42:24.865978+0000","last_peered":"2024-04-25T05:42:24.865978+0000","last_clean":"2024-04-25T05:42:24.865978+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:42:24.865978+0000","last_undegraded":"2024-04-25T05:42:24.865978+0000","last_fullsized":"2024-04-25T05:42:24.865978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:40:15.536802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.536802+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":461,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883690+0000","last_change":"2024-04-25T05:40:55.989126+0000","last_active":"2024-04-25T05:42:24.883690+0000","last_peered":"2024-04-25T05:42:24.883690+0000","last_clean":"2024-04-25T05:42:24.883690+0000","last_became_active":"2024-04-25T05:40:55.974471+0000","last_became_peered":"2024-04-25T05:40:55.974471+0000","last_unstale":"2024-04-25T05:42:24.883690+0000","last_undegraded":"2024-04-25T05:42:24.883690+0000","last_fullsized":"2024-04-25T05:42:24.883690+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:40:10.849562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:10.849562+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":217,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874579+0000","last_change":"2024-04-25T05:42:24.210053+0000","last_active":"2024-04-25T05:42:24.874579+0000","last_peered":"2024-04-25T05:42:24.874579+0000","last_clean":"2024-04-25T05:42:24.874579+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:42:24.874579+0000","last_undegraded":"2024-04-25T05:42:24.874579+0000","last_fullsized":"2024-04-25T05:42:24.874579+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:42:24.209923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.209923+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:51.982376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":463,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865959+0000","last_change":"2024-04-25T05:40:40.500687+0000","last_active":"2024-04-25T05:42:24.865959+0000","last_peered":"2024-04-25T05:42:24.865959+0000","last_clean":"2024-04-25T05:42:24.865959+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:42:24.865959+0000","last_undegraded":"2024-04-25T05:42:24.865959+0000","last_fullsized":"2024-04-25T05:42:24.865959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:40.500174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.500174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":117,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873976+0000","last_change":"2024-04-25T05:40:43.482944+0000","last_active":"2024-04-25T05:42:24.873976+0000","last_peered":"2024-04-25T05:42:24.873976+0000","last_clean":"2024-04-25T05:42:24.873976+0000","last_became_active":"2024-04-25T05:40:43.442720+0000","last_became_peered":"2024-04-25T05:40:43.442720+0000","last_unstale":"2024-04-25T05:42:24.873976+0000","last_undegraded":"2024-04-25T05:42:24.873976+0000","last_fullsized":"2024-04-25T05:42:24.873976+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865547+0000","last_change":"2024-04-25T05:40:15.703133+0000","last_active":"2024-04-25T05:42:24.865547+0000","last_peered":"2024-04-25T05:42:24.865547+0000","last_clean":"2024-04-25T05:42:24.865547+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:42:24.865547+0000","last_undegraded":"2024-04-25T05:42:24.865547+0000","last_fullsized":"2024-04-25T05:42:24.865547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.702999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.702999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":109,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874233+0000","last_change":"2024-04-25T05:40:55.967146+0000","last_active":"2024-04-25T05:42:24.874233+0000","last_peered":"2024-04-25T05:42:24.874233+0000","last_clean":"2024-04-25T05:42:24.874233+0000","last_became_active":"2024-04-25T05:40:55.966806+0000","last_became_peered":"2024-04-25T05:40:55.966806+0000","last_unstale":"2024-04-25T05:42:24.874233+0000","last_undegraded":"2024-04-25T05:42:24.874233+0000","last_fullsized":"2024-04-25T05:42:24.874233+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:04.107938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:04.107938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.673859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867041+0000","last_change":"2024-04-25T05:40:50.562106+0000","last_active":"2024-04-25T05:42:24.867041+0000","last_peered":"2024-04-25T05:42:24.867041+0000","last_clean":"2024-04-25T05:42:24.867041+0000","last_became_active":"2024-04-25T05:40:43.440363+0000","last_became_peered":"2024-04-25T05:40:43.440363+0000","last_unstale":"2024-04-25T05:42:24.867041+0000","last_undegraded":"2024-04-25T05:42:24.867041+0000","last_fullsized":"2024-04-25T05:42:24.867041+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.560317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.560317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'17","reported_seq":184,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874761+0000","last_change":"2024-04-25T05:39:40.474408+0000","last_active":"2024-04-25T05:42:24.874761+0000","last_peered":"2024-04-25T05:42:24.874761+0000","last_clean":"2024-04-25T05:42:24.874761+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:42:24.874761+0000","last_undegraded":"2024-04-25T05:42:24.874761+0000","last_fullsized":"2024-04-25T05:42:24.874761+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:39:40.474140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.474140+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":461,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883015+0000","last_change":"2024-04-25T05:41:39.031952+0000","last_active":"2024-04-25T05:42:24.883015+0000","last_peered":"2024-04-25T05:42:24.883015+0000","last_clean":"2024-04-25T05:42:24.883015+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:42:24.883015+0000","last_undegraded":"2024-04-25T05:42:24.883015+0000","last_fullsized":"2024-04-25T05:42:24.883015+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:41:39.031784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.031784+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.769610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":170,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872268+0000","last_change":"2024-04-25T05:41:40.152573+0000","last_active":"2024-04-25T05:42:24.872268+0000","last_peered":"2024-04-25T05:42:24.872268+0000","last_clean":"2024-04-25T05:42:24.872268+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:42:24.872268+0000","last_undegraded":"2024-04-25T05:42:24.872268+0000","last_fullsized":"2024-04-25T05:42:24.872268+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:40.152411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:40.152411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:02.701355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874146+0000","last_change":"2024-04-25T05:40:43.446312+0000","last_active":"2024-04-25T05:42:24.874146+0000","last_peered":"2024-04-25T05:42:24.874146+0000","last_clean":"2024-04-25T05:42:24.874146+0000","last_became_active":"2024-04-25T05:40:43.445367+0000","last_became_peered":"2024-04-25T05:40:43.445367+0000","last_unstale":"2024-04-25T05:42:24.874146+0000","last_undegraded":"2024-04-25T05:42:24.874146+0000","last_fullsized":"2024-04-25T05:42:24.874146+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":480,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867146+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:42:24.867146+0000","last_peered":"2024-04-25T05:42:24.867146+0000","last_clean":"2024-04-25T05:42:24.867146+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:42:24.867146+0000","last_undegraded":"2024-04-25T05:42:24.867146+0000","last_fullsized":"2024-04-25T05:42:24.867146+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866925+0000","last_change":"2024-04-25T05:42:20.379921+0000","last_active":"2024-04-25T05:42:24.866925+0000","last_peered":"2024-04-25T05:42:24.866925+0000","last_clean":"2024-04-25T05:42:24.866925+0000","last_became_active":"2024-04-25T05:40:43.435420+0000","last_became_peered":"2024-04-25T05:40:43.435420+0000","last_unstale":"2024-04-25T05:42:24.866925+0000","last_undegraded":"2024-04-25T05:42:24.866925+0000","last_fullsized":"2024-04-25T05:42:24.866925+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.364344+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":472,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885779+0000","last_change":"2024-04-25T05:40:53.928517+0000","last_active":"2024-04-25T05:42:24.885779+0000","last_peered":"2024-04-25T05:42:24.885779+0000","last_clean":"2024-04-25T05:42:24.885779+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:42:24.885779+0000","last_undegraded":"2024-04-25T05:42:24.885779+0000","last_fullsized":"2024-04-25T05:42:24.885779+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:53.928372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.928372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872544+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:42:24.872544+0000","last_peered":"2024-04-25T05:42:24.872544+0000","last_clean":"2024-04-25T05:42:24.872544+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:42:24.872544+0000","last_undegraded":"2024-04-25T05:42:24.872544+0000","last_fullsized":"2024-04-25T05:42:24.872544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":521,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870564+0000","last_change":"2024-04-25T05:40:49.558496+0000","last_active":"2024-04-25T05:42:24.870564+0000","last_peered":"2024-04-25T05:42:24.870564+0000","last_clean":"2024-04-25T05:42:24.870564+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:42:24.870564+0000","last_undegraded":"2024-04-25T05:42:24.870564+0000","last_fullsized":"2024-04-25T05:42:24.870564+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:49.558354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.558354+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874093+0000","last_change":"2024-04-25T05:40:43.446609+0000","last_active":"2024-04-25T05:42:24.874093+0000","last_peered":"2024-04-25T05:42:24.874093+0000","last_clean":"2024-04-25T05:42:24.874093+0000","last_became_active":"2024-04-25T05:40:43.446278+0000","last_became_peered":"2024-04-25T05:40:43.446278+0000","last_unstale":"2024-04-25T05:42:24.874093+0000","last_undegraded":"2024-04-25T05:42:24.874093+0000","last_fullsized":"2024-04-25T05:42:24.874093+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:38:54.429462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:38:54.429462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.673892+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":450,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866642+0000","last_change":"2024-04-25T05:40:43.433578+0000","last_active":"2024-04-25T05:42:24.866642+0000","last_peered":"2024-04-25T05:42:24.866642+0000","last_clean":"2024-04-25T05:42:24.866642+0000","last_became_active":"2024-04-25T05:40:43.433240+0000","last_became_peered":"2024-04-25T05:40:43.433240+0000","last_unstale":"2024-04-25T05:42:24.866642+0000","last_undegraded":"2024-04-25T05:42:24.866642+0000","last_fullsized":"2024-04-25T05:42:24.866642+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.526768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.526768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885389+0000","last_change":"2024-04-25T05:40:17.791243+0000","last_active":"2024-04-25T05:42:24.885389+0000","last_peered":"2024-04-25T05:42:24.885389+0000","last_clean":"2024-04-25T05:42:24.885389+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:42:24.885389+0000","last_undegraded":"2024-04-25T05:42:24.885389+0000","last_fullsized":"2024-04-25T05:42:24.885389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:17.791104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.791104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"257'13","reported_seq":375,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870433+0000","last_change":"2024-04-25T05:41:30.532539+0000","last_active":"2024-04-25T05:42:24.870433+0000","last_peered":"2024-04-25T05:42:24.870433+0000","last_clean":"2024-04-25T05:42:24.870433+0000","last_became_active":"2024-04-25T05:40:55.954865+0000","last_became_peered":"2024-04-25T05:40:55.954865+0000","last_unstale":"2024-04-25T05:42:24.870433+0000","last_undegraded":"2024-04-25T05:42:24.870433+0000","last_fullsized":"2024-04-25T05:42:24.870433+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:59.262752+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870053+0000","last_change":"2024-04-25T05:40:43.434425+0000","last_active":"2024-04-25T05:42:24.870053+0000","last_peered":"2024-04-25T05:42:24.870053+0000","last_clean":"2024-04-25T05:42:24.870053+0000","last_became_active":"2024-04-25T05:40:43.434051+0000","last_became_peered":"2024-04-25T05:40:43.434051+0000","last_unstale":"2024-04-25T05:42:24.870053+0000","last_undegraded":"2024-04-25T05:42:24.870053+0000","last_fullsized":"2024-04-25T05:42:24.870053+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:04.446776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:04.446776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865766+0000","last_change":"2024-04-25T05:40:06.644197+0000","last_active":"2024-04-25T05:42:24.865766+0000","last_peered":"2024-04-25T05:42:24.865766+0000","last_clean":"2024-04-25T05:42:24.865766+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:42:24.865766+0000","last_undegraded":"2024-04-25T05:42:24.865766+0000","last_fullsized":"2024-04-25T05:42:24.865766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.644083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.644083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":141,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883529+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:42:24.883529+0000","last_peered":"2024-04-25T05:42:24.883529+0000","last_clean":"2024-04-25T05:42:24.883529+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:42:24.883529+0000","last_undegraded":"2024-04-25T05:42:24.883529+0000","last_fullsized":"2024-04-25T05:42:24.883529+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.152477+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":548,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867767+0000","last_change":"2024-04-25T05:40:56.002007+0000","last_active":"2024-04-25T05:42:24.867767+0000","last_peered":"2024-04-25T05:42:24.867767+0000","last_clean":"2024-04-25T05:42:24.867767+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:42:24.867767+0000","last_undegraded":"2024-04-25T05:42:24.867767+0000","last_fullsized":"2024-04-25T05:42:24.867767+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:40:17.835081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:17.835081+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874434+0000","last_change":"2024-04-25T05:40:43.443261+0000","last_active":"2024-04-25T05:42:24.874434+0000","last_peered":"2024-04-25T05:42:24.874434+0000","last_clean":"2024-04-25T05:42:24.874434+0000","last_became_active":"2024-04-25T05:40:43.439460+0000","last_became_peered":"2024-04-25T05:40:43.439460+0000","last_unstale":"2024-04-25T05:42:24.874434+0000","last_undegraded":"2024-04-25T05:42:24.874434+0000","last_fullsized":"2024-04-25T05:42:24.874434+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"233'10","reported_seq":585,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873315+0000","last_change":"2024-04-25T05:41:45.481735+0000","last_active":"2024-04-25T05:42:24.873315+0000","last_peered":"2024-04-25T05:42:24.873315+0000","last_clean":"2024-04-25T05:42:24.873315+0000","last_became_active":"2024-04-25T05:40:43.453246+0000","last_became_peered":"2024-04-25T05:40:43.453246+0000","last_unstale":"2024-04-25T05:42:24.873315+0000","last_undegraded":"2024-04-25T05:42:24.873315+0000","last_fullsized":"2024-04-25T05:42:24.873315+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:48.532916+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":468,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883693+0000","last_change":"2024-04-25T05:41:37.012236+0000","last_active":"2024-04-25T05:42:24.883693+0000","last_peered":"2024-04-25T05:42:24.883693+0000","last_clean":"2024-04-25T05:42:24.883693+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:42:24.883693+0000","last_undegraded":"2024-04-25T05:42:24.883693+0000","last_fullsized":"2024-04-25T05:42:24.883693+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:37.012159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:37.012159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.873027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":489,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866865+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:42:24.866865+0000","last_peered":"2024-04-25T05:42:24.866865+0000","last_clean":"2024-04-25T05:42:24.866865+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:42:24.866865+0000","last_undegraded":"2024-04-25T05:42:24.866865+0000","last_fullsized":"2024-04-25T05:42:24.866865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:15.927150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":90,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872960+0000","last_change":"2024-04-25T05:40:55.954634+0000","last_active":"2024-04-25T05:42:24.872960+0000","last_peered":"2024-04-25T05:42:24.872960+0000","last_clean":"2024-04-25T05:42:24.872960+0000","last_became_active":"2024-04-25T05:40:55.954181+0000","last_became_peered":"2024-04-25T05:40:55.954181+0000","last_unstale":"2024-04-25T05:42:24.872960+0000","last_undegraded":"2024-04-25T05:42:24.872960+0000","last_fullsized":"2024-04-25T05:42:24.872960+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:09.913324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.913324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":558,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867216+0000","last_change":"2024-04-25T05:40:56.012131+0000","last_active":"2024-04-25T05:42:24.867216+0000","last_peered":"2024-04-25T05:42:24.867216+0000","last_clean":"2024-04-25T05:42:24.867216+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:42:24.867216+0000","last_undegraded":"2024-04-25T05:42:24.867216+0000","last_fullsized":"2024-04-25T05:42:24.867216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:40:37.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.529721+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873886+0000","last_change":"2024-04-25T05:40:43.434277+0000","last_active":"2024-04-25T05:42:24.873886+0000","last_peered":"2024-04-25T05:42:24.873886+0000","last_clean":"2024-04-25T05:42:24.873886+0000","last_became_active":"2024-04-25T05:40:43.433933+0000","last_became_peered":"2024-04-25T05:40:43.433933+0000","last_unstale":"2024-04-25T05:42:24.873886+0000","last_undegraded":"2024-04-25T05:42:24.873886+0000","last_fullsized":"2024-04-25T05:42:24.873886+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":491,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873534+0000","last_change":"2024-04-25T05:42:08.829436+0000","last_active":"2024-04-25T05:42:24.873534+0000","last_peered":"2024-04-25T05:42:24.873534+0000","last_clean":"2024-04-25T05:42:24.873534+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:42:24.873534+0000","last_undegraded":"2024-04-25T05:42:24.873534+0000","last_fullsized":"2024-04-25T05:42:24.873534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:22.650865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":97,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868158+0000","last_change":"2024-04-25T05:42:10.907844+0000","last_active":"2024-04-25T05:42:24.868158+0000","last_peered":"2024-04-25T05:42:24.868158+0000","last_clean":"2024-04-25T05:42:24.868158+0000","last_became_active":"2024-04-25T05:40:55.953924+0000","last_became_peered":"2024-04-25T05:40:55.953924+0000","last_unstale":"2024-04-25T05:42:24.868158+0000","last_undegraded":"2024-04-25T05:42:24.868158+0000","last_fullsized":"2024-04-25T05:42:24.868158+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:10.907707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.907707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:31.135983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":438,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868046+0000","last_change":"2024-04-25T05:40:38.559014+0000","last_active":"2024-04-25T05:42:24.868046+0000","last_peered":"2024-04-25T05:42:24.868046+0000","last_clean":"2024-04-25T05:42:24.868046+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:42:24.868046+0000","last_undegraded":"2024-04-25T05:42:24.868046+0000","last_fullsized":"2024-04-25T05:42:24.868046+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:38.558895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:38.558895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":480,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883081+0000","last_change":"2024-04-25T05:40:50.906267+0000","last_active":"2024-04-25T05:42:24.883081+0000","last_peered":"2024-04-25T05:42:24.883081+0000","last_clean":"2024-04-25T05:42:24.883081+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:42:24.883081+0000","last_undegraded":"2024-04-25T05:42:24.883081+0000","last_fullsized":"2024-04-25T05:42:24.883081+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.905458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.905458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":125,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874108+0000","last_change":"2024-04-25T05:42:21.659494+0000","last_active":"2024-04-25T05:42:24.874108+0000","last_peered":"2024-04-25T05:42:24.874108+0000","last_clean":"2024-04-25T05:42:24.874108+0000","last_became_active":"2024-04-25T05:40:43.441558+0000","last_became_peered":"2024-04-25T05:40:43.441558+0000","last_unstale":"2024-04-25T05:42:24.874108+0000","last_undegraded":"2024-04-25T05:42:24.874108+0000","last_fullsized":"2024-04-25T05:42:24.874108+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.744922+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.884422+0000","last_change":"2024-04-25T05:41:27.185161+0000","last_active":"2024-04-25T05:42:24.884422+0000","last_peered":"2024-04-25T05:42:24.884422+0000","last_clean":"2024-04-25T05:42:24.884422+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:42:24.884422+0000","last_undegraded":"2024-04-25T05:42:24.884422+0000","last_fullsized":"2024-04-25T05:42:24.884422+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.185044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.185044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:28.448062+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872466+0000","last_change":"2024-04-25T05:40:46.378509+0000","last_active":"2024-04-25T05:42:24.872466+0000","last_peered":"2024-04-25T05:42:24.872466+0000","last_clean":"2024-04-25T05:42:24.872466+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:42:24.872466+0000","last_undegraded":"2024-04-25T05:42:24.872466+0000","last_fullsized":"2024-04-25T05:42:24.872466+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:46.378394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:40:46.378394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":176,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870175+0000","last_change":"2024-04-25T05:40:43.431845+0000","last_active":"2024-04-25T05:42:24.870175+0000","last_peered":"2024-04-25T05:42:24.870175+0000","last_clean":"2024-04-25T05:42:24.870175+0000","last_became_active":"2024-04-25T05:40:43.431502+0000","last_became_peered":"2024-04-25T05:40:43.431502+0000","last_unstale":"2024-04-25T05:42:24.870175+0000","last_undegraded":"2024-04-25T05:42:24.870175+0000","last_fullsized":"2024-04-25T05:42:24.870175+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.918723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.918723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":471,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868235+0000","last_change":"2024-04-25T05:40:41.507574+0000","last_active":"2024-04-25T05:42:24.868235+0000","last_peered":"2024-04-25T05:42:24.868235+0000","last_clean":"2024-04-25T05:42:24.868235+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:42:24.868235+0000","last_undegraded":"2024-04-25T05:42:24.868235+0000","last_fullsized":"2024-04-25T05:42:24.868235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:41.507400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.507400+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"279'13298","reported_seq":27934,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.871388+0000","last_change":"2024-04-25T05:41:28.270522+0000","last_active":"2024-04-25T05:42:27.871388+0000","last_peered":"2024-04-25T05:42:27.871388+0000","last_clean":"2024-04-25T05:42:27.871388+0000","last_became_active":"2024-04-25T05:41:28.224071+0000","last_became_peered":"2024-04-25T05:41:28.224071+0000","last_unstale":"2024-04-25T05:42:27.871388+0000","last_undegraded":"2024-04-25T05:42:27.871388+0000","last_fullsized":"2024-04-25T05:42:27.871388+0000","mapping_epoch":255,"log_start":"250'8937","ondisk_log_start":"250'8937","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'7237","last_scrub_stamp":"2024-04-25T05:40:43.047218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.047218+0000","objects_scrubbed":7237,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:30.005688+0000","scrub_duration":822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162660352,"num_objects":2482,"num_object_clones":0,"num_object_copies":7446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13298,"num_write_kb":504960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7185,"num_bytes_recovered":470810624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":474,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871556+0000","last_change":"2024-04-25T05:40:51.460612+0000","last_active":"2024-04-25T05:42:24.871556+0000","last_peered":"2024-04-25T05:42:24.871556+0000","last_clean":"2024-04-25T05:42:24.871556+0000","last_became_active":"2024-04-25T05:40:43.453025+0000","last_became_peered":"2024-04-25T05:40:43.453025+0000","last_unstale":"2024-04-25T05:42:24.871556+0000","last_undegraded":"2024-04-25T05:42:24.871556+0000","last_fullsized":"2024-04-25T05:42:24.871556+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.460330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.460330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":498,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885085+0000","last_change":"2024-04-25T05:41:25.239411+0000","last_active":"2024-04-25T05:42:24.885085+0000","last_peered":"2024-04-25T05:42:24.885085+0000","last_clean":"2024-04-25T05:42:24.885085+0000","last_became_active":"2024-04-25T05:40:43.429608+0000","last_became_peered":"2024-04-25T05:40:43.429608+0000","last_unstale":"2024-04-25T05:42:24.885085+0000","last_undegraded":"2024-04-25T05:42:24.885085+0000","last_fullsized":"2024-04-25T05:42:24.885085+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:41:25.239292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:25.239292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:33.712255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":551,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871165+0000","last_change":"2024-04-25T05:39:30.386374+0000","last_active":"2024-04-25T05:42:24.871165+0000","last_peered":"2024-04-25T05:42:24.871165+0000","last_clean":"2024-04-25T05:42:24.871165+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:42:24.871165+0000","last_undegraded":"2024-04-25T05:42:24.871165+0000","last_fullsized":"2024-04-25T05:42:24.871165+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:39:30.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:39:30.386275+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":588,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883380+0000","last_change":"2024-04-25T05:39:33.128642+0000","last_active":"2024-04-25T05:42:24.883380+0000","last_peered":"2024-04-25T05:42:24.883380+0000","last_clean":"2024-04-25T05:42:24.883380+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:42:24.883380+0000","last_undegraded":"2024-04-25T05:42:24.883380+0000","last_fullsized":"2024-04-25T05:42:24.883380+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'10","last_scrub_stamp":"2024-04-25T05:39:33.128477+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:39:33.128477+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"279'13218","reported_seq":13644,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.865123+0000","last_change":"2024-04-25T05:41:36.031565+0000","last_active":"2024-04-25T05:42:27.865123+0000","last_peered":"2024-04-25T05:42:27.865123+0000","last_clean":"2024-04-25T05:42:27.865123+0000","last_became_active":"2024-04-25T05:39:35.778659+0000","last_became_peered":"2024-04-25T05:39:35.778659+0000","last_unstale":"2024-04-25T05:42:27.865123+0000","last_undegraded":"2024-04-25T05:42:27.865123+0000","last_fullsized":"2024-04-25T05:42:27.865123+0000","mapping_epoch":209,"log_start":"247'8460","ondisk_log_start":"247'8460","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'9942","last_scrub_stamp":"2024-04-25T05:41:36.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.031226+0000","objects_scrubbed":5781,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:45.717941+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866796+0000","last_change":"2024-04-25T05:40:55.989744+0000","last_active":"2024-04-25T05:42:24.866796+0000","last_peered":"2024-04-25T05:42:24.866796+0000","last_clean":"2024-04-25T05:42:24.866796+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:42:24.866796+0000","last_undegraded":"2024-04-25T05:42:24.866796+0000","last_fullsized":"2024-04-25T05:42:24.866796+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.579078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.579078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883715+0000","last_change":"2024-04-25T05:39:38.030757+0000","last_active":"2024-04-25T05:42:24.883715+0000","last_peered":"2024-04-25T05:42:24.883715+0000","last_clean":"2024-04-25T05:42:24.883715+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:42:24.883715+0000","last_undegraded":"2024-04-25T05:42:24.883715+0000","last_fullsized":"2024-04-25T05:42:24.883715+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.030627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.030627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865355+0000","last_change":"2024-04-25T05:40:56.145089+0000","last_active":"2024-04-25T05:42:24.865355+0000","last_peered":"2024-04-25T05:42:24.865355+0000","last_clean":"2024-04-25T05:42:24.865355+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:42:24.865355+0000","last_undegraded":"2024-04-25T05:42:24.865355+0000","last_fullsized":"2024-04-25T05:42:24.865355+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:40:37.943948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.943948+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":452,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874202+0000","last_change":"2024-04-25T05:40:55.988511+0000","last_active":"2024-04-25T05:42:24.874202+0000","last_peered":"2024-04-25T05:42:24.874202+0000","last_clean":"2024-04-25T05:42:24.874202+0000","last_became_active":"2024-04-25T05:40:55.988228+0000","last_became_peered":"2024-04-25T05:40:55.988228+0000","last_unstale":"2024-04-25T05:42:24.874202+0000","last_undegraded":"2024-04-25T05:42:24.874202+0000","last_fullsized":"2024-04-25T05:42:24.874202+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674437+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"279'13147","reported_seq":27922,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.247336+0000","last_change":"2024-04-25T05:41:42.929604+0000","last_active":"2024-04-25T05:42:28.247336+0000","last_peered":"2024-04-25T05:42:28.247336+0000","last_clean":"2024-04-25T05:42:28.247336+0000","last_became_active":"2024-04-25T05:41:42.899661+0000","last_became_peered":"2024-04-25T05:41:42.899661+0000","last_unstale":"2024-04-25T05:42:28.247336+0000","last_undegraded":"2024-04-25T05:42:28.247336+0000","last_fullsized":"2024-04-25T05:42:28.247336+0000","mapping_epoch":260,"log_start":"247'8457","ondisk_log_start":"247'8457","created":198,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":7221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13147,"num_write_kb":497728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7289,"num_bytes_recovered":477691904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":448,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885354+0000","last_change":"2024-04-25T05:40:55.982002+0000","last_active":"2024-04-25T05:42:24.885354+0000","last_peered":"2024-04-25T05:42:24.885354+0000","last_clean":"2024-04-25T05:42:24.885354+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:42:24.885354+0000","last_undegraded":"2024-04-25T05:42:24.885354+0000","last_fullsized":"2024-04-25T05:42:24.885354+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:36.074043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:36.074043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":451,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873575+0000","last_change":"2024-04-25T05:40:55.953740+0000","last_active":"2024-04-25T05:42:24.873575+0000","last_peered":"2024-04-25T05:42:24.873575+0000","last_clean":"2024-04-25T05:42:24.873575+0000","last_became_active":"2024-04-25T05:40:55.953404+0000","last_became_peered":"2024-04-25T05:40:55.953404+0000","last_unstale":"2024-04-25T05:42:24.873575+0000","last_undegraded":"2024-04-25T05:42:24.873575+0000","last_fullsized":"2024-04-25T05:42:24.873575+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":488,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885877+0000","last_change":"2024-04-25T05:40:08.880333+0000","last_active":"2024-04-25T05:42:24.885877+0000","last_peered":"2024-04-25T05:42:24.885877+0000","last_clean":"2024-04-25T05:42:24.885877+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:42:24.885877+0000","last_undegraded":"2024-04-25T05:42:24.885877+0000","last_fullsized":"2024-04-25T05:42:24.885877+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:08.880207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.880207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":200,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870997+0000","last_change":"2024-04-25T05:42:15.186498+0000","last_active":"2024-04-25T05:42:24.870997+0000","last_peered":"2024-04-25T05:42:24.870997+0000","last_clean":"2024-04-25T05:42:24.870997+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:42:24.870997+0000","last_undegraded":"2024-04-25T05:42:24.870997+0000","last_fullsized":"2024-04-25T05:42:24.870997+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:15.186321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.186321+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:26.644738+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"279'13162","reported_seq":14300,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.378518+0000","last_change":"2024-04-25T05:40:53.660560+0000","last_active":"2024-04-25T05:42:28.378518+0000","last_peered":"2024-04-25T05:42:28.378518+0000","last_clean":"2024-04-25T05:42:28.378518+0000","last_became_active":"2024-04-25T05:39:35.777746+0000","last_became_peered":"2024-04-25T05:39:35.777746+0000","last_unstale":"2024-04-25T05:42:28.378518+0000","last_undegraded":"2024-04-25T05:42:28.378518+0000","last_fullsized":"2024-04-25T05:42:28.378518+0000","mapping_epoch":209,"log_start":"250'8813","ondisk_log_start":"250'8813","created":198,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7624","last_scrub_stamp":"2024-04-25T05:40:53.660173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.660173+0000","objects_scrubbed":7548,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:32.034578+0000","scrub_duration":3156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147193856,"num_objects":2246,"num_object_clones":0,"num_object_copies":6738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13162,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":109,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.881697+0000","last_change":"2024-04-25T05:40:55.972738+0000","last_active":"2024-04-25T05:42:24.881697+0000","last_peered":"2024-04-25T05:42:24.881697+0000","last_clean":"2024-04-25T05:42:24.881697+0000","last_became_active":"2024-04-25T05:40:55.972405+0000","last_became_peered":"2024-04-25T05:40:55.972405+0000","last_unstale":"2024-04-25T05:42:24.881697+0000","last_undegraded":"2024-04-25T05:42:24.881697+0000","last_fullsized":"2024-04-25T05:42:24.881697+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:06.050260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:06.050260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":96,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867348+0000","last_change":"2024-04-25T05:42:05.954387+0000","last_active":"2024-04-25T05:42:24.867348+0000","last_peered":"2024-04-25T05:42:24.867348+0000","last_clean":"2024-04-25T05:42:24.867348+0000","last_became_active":"2024-04-25T05:40:55.981082+0000","last_became_peered":"2024-04-25T05:40:55.981082+0000","last_unstale":"2024-04-25T05:42:24.867348+0000","last_undegraded":"2024-04-25T05:42:24.867348+0000","last_fullsized":"2024-04-25T05:42:24.867348+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:05.954259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:05.954259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.753300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":474,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874182+0000","last_change":"2024-04-25T05:41:28.560703+0000","last_active":"2024-04-25T05:42:24.874182+0000","last_peered":"2024-04-25T05:42:24.874182+0000","last_clean":"2024-04-25T05:42:24.874182+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:42:24.874182+0000","last_undegraded":"2024-04-25T05:42:24.874182+0000","last_fullsized":"2024-04-25T05:42:24.874182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:36.353516+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":115,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.878471+0000","last_change":"2024-04-25T05:40:43.470705+0000","last_active":"2024-04-25T05:42:24.878471+0000","last_peered":"2024-04-25T05:42:24.878471+0000","last_clean":"2024-04-25T05:42:24.878471+0000","last_became_active":"2024-04-25T05:40:43.441309+0000","last_became_peered":"2024-04-25T05:40:43.441309+0000","last_unstale":"2024-04-25T05:42:24.878471+0000","last_undegraded":"2024-04-25T05:42:24.878471+0000","last_fullsized":"2024-04-25T05:42:24.878471+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674352+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"279'12951","reported_seq":13124,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.854345+0000","last_change":"2024-04-25T05:39:19.233374+0000","last_active":"2024-04-25T05:42:27.854345+0000","last_peered":"2024-04-25T05:42:27.854345+0000","last_clean":"2024-04-25T05:42:27.854345+0000","last_became_active":"2024-04-25T05:39:19.233055+0000","last_became_peered":"2024-04-25T05:39:19.233055+0000","last_unstale":"2024-04-25T05:42:27.854345+0000","last_undegraded":"2024-04-25T05:42:27.854345+0000","last_fullsized":"2024-04-25T05:42:27.854345+0000","mapping_epoch":198,"log_start":"248'8578","ondisk_log_start":"248'8578","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:30.005343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151584768,"num_objects":2313,"num_object_clones":0,"num_object_copies":6939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12951,"num_write_kb":488448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":647,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871353+0000","last_change":"2024-04-25T05:40:58.922339+0000","last_active":"2024-04-25T05:42:24.871353+0000","last_peered":"2024-04-25T05:42:24.871353+0000","last_clean":"2024-04-25T05:42:24.871353+0000","last_became_active":"2024-04-25T05:40:58.921770+0000","last_became_peered":"2024-04-25T05:40:58.921770+0000","last_unstale":"2024-04-25T05:42:24.871353+0000","last_undegraded":"2024-04-25T05:42:24.871353+0000","last_fullsized":"2024-04-25T05:42:24.871353+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:07.411609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:07.411609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":442,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867310+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:42:24.867310+0000","last_peered":"2024-04-25T05:42:24.867310+0000","last_clean":"2024-04-25T05:42:24.867310+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:42:24.867310+0000","last_undegraded":"2024-04-25T05:42:24.867310+0000","last_fullsized":"2024-04-25T05:42:24.867310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.757268+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":495,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870826+0000","last_change":"2024-04-25T05:39:29.827029+0000","last_active":"2024-04-25T05:42:24.870826+0000","last_peered":"2024-04-25T05:42:24.870826+0000","last_clean":"2024-04-25T05:42:24.870826+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:42:24.870826+0000","last_undegraded":"2024-04-25T05:42:24.870826+0000","last_fullsized":"2024-04-25T05:42:24.870826+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:39:08.351553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:08.351553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.878378+0000","last_change":"2024-04-25T05:40:43.435305+0000","last_active":"2024-04-25T05:42:24.878378+0000","last_peered":"2024-04-25T05:42:24.878378+0000","last_clean":"2024-04-25T05:42:24.878378+0000","last_became_active":"2024-04-25T05:40:43.434803+0000","last_became_peered":"2024-04-25T05:40:43.434803+0000","last_unstale":"2024-04-25T05:42:24.878378+0000","last_undegraded":"2024-04-25T05:42:24.878378+0000","last_fullsized":"2024-04-25T05:42:24.878378+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.849716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.849716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674574+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"243'46","reported_seq":654,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865987+0000","last_change":"2024-04-25T05:40:56.036908+0000","last_active":"2024-04-25T05:42:24.865987+0000","last_peered":"2024-04-25T05:42:24.865987+0000","last_clean":"2024-04-25T05:42:24.865987+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:42:24.865987+0000","last_undegraded":"2024-04-25T05:42:24.865987+0000","last_fullsized":"2024-04-25T05:42:24.865987+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'45","last_scrub_stamp":"2024-04-25T05:39:42.723973+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:39:42.723973+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874145+0000","last_change":"2024-04-25T05:40:43.442114+0000","last_active":"2024-04-25T05:42:24.874145+0000","last_peered":"2024-04-25T05:42:24.874145+0000","last_clean":"2024-04-25T05:42:24.874145+0000","last_became_active":"2024-04-25T05:40:43.441816+0000","last_became_peered":"2024-04-25T05:40:43.441816+0000","last_unstale":"2024-04-25T05:42:24.874145+0000","last_undegraded":"2024-04-25T05:42:24.874145+0000","last_fullsized":"2024-04-25T05:42:24.874145+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674628+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"279'12924","reported_seq":13097,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.406494+0000","last_change":"2024-04-25T05:39:19.228876+0000","last_active":"2024-04-25T05:42:28.406494+0000","last_peered":"2024-04-25T05:42:28.406494+0000","last_clean":"2024-04-25T05:42:28.406494+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:42:28.406494+0000","last_undegraded":"2024-04-25T05:42:28.406494+0000","last_fullsized":"2024-04-25T05:42:28.406494+0000","mapping_epoch":198,"log_start":"251'8797","ondisk_log_start":"251'8797","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.325381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150732800,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12924,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":769,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.882898+0000","last_change":"2024-04-25T05:41:44.999875+0000","last_active":"2024-04-25T05:42:24.882898+0000","last_peered":"2024-04-25T05:42:24.882898+0000","last_clean":"2024-04-25T05:42:24.882898+0000","last_became_active":"2024-04-25T05:40:45.848289+0000","last_became_peered":"2024-04-25T05:40:45.848289+0000","last_unstale":"2024-04-25T05:42:24.882898+0000","last_undegraded":"2024-04-25T05:42:24.882898+0000","last_fullsized":"2024-04-25T05:42:24.882898+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.306297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.886044+0000","last_change":"2024-04-25T05:40:52.973425+0000","last_active":"2024-04-25T05:42:24.886044+0000","last_peered":"2024-04-25T05:42:24.886044+0000","last_clean":"2024-04-25T05:42:24.886044+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:42:24.886044+0000","last_undegraded":"2024-04-25T05:42:24.886044+0000","last_fullsized":"2024-04-25T05:42:24.886044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:52.973302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.973302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":487,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868365+0000","last_change":"2024-04-25T05:40:55.966682+0000","last_active":"2024-04-25T05:42:24.868365+0000","last_peered":"2024-04-25T05:42:24.868365+0000","last_clean":"2024-04-25T05:42:24.868365+0000","last_became_active":"2024-04-25T05:40:55.966367+0000","last_became_peered":"2024-04-25T05:40:55.966367+0000","last_unstale":"2024-04-25T05:42:24.868365+0000","last_undegraded":"2024-04-25T05:42:24.868365+0000","last_fullsized":"2024-04-25T05:42:24.868365+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:42.474808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:42.474808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":468,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870165+0000","last_change":"2024-04-25T05:40:55.991779+0000","last_active":"2024-04-25T05:42:24.870165+0000","last_peered":"2024-04-25T05:42:24.870165+0000","last_clean":"2024-04-25T05:42:24.870165+0000","last_became_active":"2024-04-25T05:40:55.990929+0000","last_became_peered":"2024-04-25T05:40:55.990929+0000","last_unstale":"2024-04-25T05:42:24.870165+0000","last_undegraded":"2024-04-25T05:42:24.870165+0000","last_fullsized":"2024-04-25T05:42:24.870165+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.764632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.764632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":535,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868055+0000","last_change":"2024-04-25T05:40:43.477675+0000","last_active":"2024-04-25T05:42:24.868055+0000","last_peered":"2024-04-25T05:42:24.868055+0000","last_clean":"2024-04-25T05:42:24.868055+0000","last_became_active":"2024-04-25T05:40:43.450842+0000","last_became_peered":"2024-04-25T05:40:43.450842+0000","last_unstale":"2024-04-25T05:42:24.868055+0000","last_undegraded":"2024-04-25T05:42:24.868055+0000","last_fullsized":"2024-04-25T05:42:24.868055+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.592911+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":88,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872730+0000","last_change":"2024-04-25T05:40:55.910414+0000","last_active":"2024-04-25T05:42:24.872730+0000","last_peered":"2024-04-25T05:42:24.872730+0000","last_clean":"2024-04-25T05:42:24.872730+0000","last_became_active":"2024-04-25T05:40:55.910074+0000","last_became_peered":"2024-04-25T05:40:55.910074+0000","last_unstale":"2024-04-25T05:42:24.872730+0000","last_undegraded":"2024-04-25T05:42:24.872730+0000","last_fullsized":"2024-04-25T05:42:24.872730+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.980299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.980299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"279'13051","reported_seq":28199,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.833841+0000","last_change":"2024-04-25T05:41:32.354405+0000","last_active":"2024-04-25T05:42:27.833841+0000","last_peered":"2024-04-25T05:42:27.833841+0000","last_clean":"2024-04-25T05:42:27.833841+0000","last_became_active":"2024-04-25T05:41:32.341701+0000","last_became_peered":"2024-04-25T05:41:32.341701+0000","last_unstale":"2024-04-25T05:42:27.833841+0000","last_undegraded":"2024-04-25T05:42:27.833841+0000","last_fullsized":"2024-04-25T05:42:27.833841+0000","mapping_epoch":258,"log_start":"247'8327","ondisk_log_start":"247'8327","created":198,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6983","last_scrub_stamp":"2024-04-25T05:40:41.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.157014+0000","objects_scrubbed":6915,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:30.251304+0000","scrub_duration":2134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153288704,"num_objects":2339,"num_object_clones":0,"num_object_copies":7017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":487325696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":450,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872963+0000","last_change":"2024-04-25T05:42:15.011589+0000","last_active":"2024-04-25T05:42:24.872963+0000","last_peered":"2024-04-25T05:42:24.872963+0000","last_clean":"2024-04-25T05:42:24.872963+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:42:24.872963+0000","last_undegraded":"2024-04-25T05:42:24.872963+0000","last_fullsized":"2024-04-25T05:42:24.872963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:16.656094+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":771,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885536+0000","last_change":"2024-04-25T05:42:18.235935+0000","last_active":"2024-04-25T05:42:24.885536+0000","last_peered":"2024-04-25T05:42:24.885536+0000","last_clean":"2024-04-25T05:42:24.885536+0000","last_became_active":"2024-04-25T05:40:45.847131+0000","last_became_peered":"2024-04-25T05:40:45.847131+0000","last_unstale":"2024-04-25T05:42:24.885536+0000","last_undegraded":"2024-04-25T05:42:24.885536+0000","last_fullsized":"2024-04-25T05:42:24.885536+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.800850+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867308+0000","last_change":"2024-04-25T05:42:08.356121+0000","last_active":"2024-04-25T05:42:24.867308+0000","last_peered":"2024-04-25T05:42:24.867308+0000","last_clean":"2024-04-25T05:42:24.867308+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:42:24.867308+0000","last_undegraded":"2024-04-25T05:42:24.867308+0000","last_fullsized":"2024-04-25T05:42:24.867308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:10.802129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":223,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874914+0000","last_change":"2024-04-25T05:40:40.957968+0000","last_active":"2024-04-25T05:42:24.874914+0000","last_peered":"2024-04-25T05:42:24.874914+0000","last_clean":"2024-04-25T05:42:24.874914+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:42:24.874914+0000","last_undegraded":"2024-04-25T05:42:24.874914+0000","last_fullsized":"2024-04-25T05:42:24.874914+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:40:40.957665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.957665+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":461,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883794+0000","last_change":"2024-04-25T05:41:20.125777+0000","last_active":"2024-04-25T05:42:24.883794+0000","last_peered":"2024-04-25T05:42:24.883794+0000","last_clean":"2024-04-25T05:42:24.883794+0000","last_became_active":"2024-04-25T05:40:55.953691+0000","last_became_peered":"2024-04-25T05:40:55.953691+0000","last_unstale":"2024-04-25T05:42:24.883794+0000","last_undegraded":"2024-04-25T05:42:24.883794+0000","last_fullsized":"2024-04-25T05:42:24.883794+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:20.125501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:41:20.125501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.894302+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"279'13306","reported_seq":23860,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.401344+0000","last_change":"2024-04-25T05:42:07.903446+0000","last_active":"2024-04-25T05:42:28.401344+0000","last_peered":"2024-04-25T05:42:28.401344+0000","last_clean":"2024-04-25T05:42:28.401344+0000","last_became_active":"2024-04-25T05:42:07.871807+0000","last_became_peered":"2024-04-25T05:42:07.871807+0000","last_unstale":"2024-04-25T05:42:28.401344+0000","last_undegraded":"2024-04-25T05:42:28.401344+0000","last_fullsized":"2024-04-25T05:42:28.401344+0000","mapping_epoch":271,"log_start":"250'8933","ondisk_log_start":"250'8933","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7666","last_scrub_stamp":"2024-04-25T05:40:52.458924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.458924+0000","objects_scrubbed":7586,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:32.033105+0000","scrub_duration":2997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153878528,"num_objects":2348,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13306,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5139,"num_bytes_recovered":336789504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":644,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866762+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:42:24.866762+0000","last_peered":"2024-04-25T05:42:24.866762+0000","last_clean":"2024-04-25T05:42:24.866762+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:42:24.866762+0000","last_undegraded":"2024-04-25T05:42:24.866762+0000","last_fullsized":"2024-04-25T05:42:24.866762+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.357654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":451,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865323+0000","last_change":"2024-04-25T05:40:47.469910+0000","last_active":"2024-04-25T05:42:24.865323+0000","last_peered":"2024-04-25T05:42:24.865323+0000","last_clean":"2024-04-25T05:42:24.865323+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:42:24.865323+0000","last_undegraded":"2024-04-25T05:42:24.865323+0000","last_fullsized":"2024-04-25T05:42:24.865323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:32.019522+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":103,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.881794+0000","last_change":"2024-04-25T05:40:43.435255+0000","last_active":"2024-04-25T05:42:24.881794+0000","last_peered":"2024-04-25T05:42:24.881794+0000","last_clean":"2024-04-25T05:42:24.881794+0000","last_became_active":"2024-04-25T05:40:43.434912+0000","last_became_peered":"2024-04-25T05:40:43.434912+0000","last_unstale":"2024-04-25T05:42:24.881794+0000","last_undegraded":"2024-04-25T05:42:24.881794+0000","last_fullsized":"2024-04-25T05:42:24.881794+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.785355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.785355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":996,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:26.623493+0000","last_change":"2024-04-25T05:41:36.018017+0000","last_active":"2024-04-25T05:42:26.623493+0000","last_peered":"2024-04-25T05:42:26.623493+0000","last_clean":"2024-04-25T05:42:26.623493+0000","last_became_active":"2024-04-25T05:40:43.431415+0000","last_became_peered":"2024-04-25T05:40:43.431415+0000","last_unstale":"2024-04-25T05:42:26.623493+0000","last_undegraded":"2024-04-25T05:42:26.623493+0000","last_fullsized":"2024-04-25T05:42:26.623493+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:41:36.017857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.017857+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:39.489209+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":496,"num_read_kb":491,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":289,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872367+0000","last_change":"2024-04-25T05:40:12.370708+0000","last_active":"2024-04-25T05:42:24.872367+0000","last_peered":"2024-04-25T05:42:24.872367+0000","last_clean":"2024-04-25T05:42:24.872367+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:42:24.872367+0000","last_undegraded":"2024-04-25T05:42:24.872367+0000","last_fullsized":"2024-04-25T05:42:24.872367+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874934+0000","last_change":"2024-04-25T05:40:55.418588+0000","last_active":"2024-04-25T05:42:24.874934+0000","last_peered":"2024-04-25T05:42:24.874934+0000","last_clean":"2024-04-25T05:42:24.874934+0000","last_became_active":"2024-04-25T05:40:43.454300+0000","last_became_peered":"2024-04-25T05:40:43.454300+0000","last_unstale":"2024-04-25T05:42:24.874934+0000","last_undegraded":"2024-04-25T05:42:24.874934+0000","last_fullsized":"2024-04-25T05:42:24.874934+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:55.418499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.418499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"279'13461","reported_seq":26638,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.398828+0000","last_change":"2024-04-25T05:41:48.020080+0000","last_active":"2024-04-25T05:42:28.398828+0000","last_peered":"2024-04-25T05:42:28.398828+0000","last_clean":"2024-04-25T05:42:28.398828+0000","last_became_active":"2024-04-25T05:41:47.997545+0000","last_became_peered":"2024-04-25T05:41:47.997545+0000","last_unstale":"2024-04-25T05:42:28.398828+0000","last_undegraded":"2024-04-25T05:42:28.398828+0000","last_fullsized":"2024-04-25T05:42:28.398828+0000","mapping_epoch":262,"log_start":"252'9292","ondisk_log_start":"252'9292","created":198,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.325296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13461,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6494,"num_bytes_recovered":425590784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":642,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873233+0000","last_change":"2024-04-25T05:40:45.845890+0000","last_active":"2024-04-25T05:42:24.873233+0000","last_peered":"2024-04-25T05:42:24.873233+0000","last_clean":"2024-04-25T05:42:24.873233+0000","last_became_active":"2024-04-25T05:40:45.845535+0000","last_became_peered":"2024-04-25T05:40:45.845535+0000","last_unstale":"2024-04-25T05:42:24.873233+0000","last_undegraded":"2024-04-25T05:42:24.873233+0000","last_fullsized":"2024-04-25T05:42:24.873233+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:02.832193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.832193+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":469,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867162+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:42:24.867162+0000","last_peered":"2024-04-25T05:42:24.867162+0000","last_clean":"2024-04-25T05:42:24.867162+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:42:24.867162+0000","last_undegraded":"2024-04-25T05:42:24.867162+0000","last_fullsized":"2024-04-25T05:42:24.867162+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.278820+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":469,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865659+0000","last_change":"2024-04-25T05:40:55.977907+0000","last_active":"2024-04-25T05:42:24.865659+0000","last_peered":"2024-04-25T05:42:24.865659+0000","last_clean":"2024-04-25T05:42:24.865659+0000","last_became_active":"2024-04-25T05:40:55.977574+0000","last_became_peered":"2024-04-25T05:40:55.977574+0000","last_unstale":"2024-04-25T05:42:24.865659+0000","last_undegraded":"2024-04-25T05:42:24.865659+0000","last_fullsized":"2024-04-25T05:42:24.865659+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:16.651620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:16.651620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":131,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.878187+0000","last_change":"2024-04-25T05:42:22.674343+0000","last_active":"2024-04-25T05:42:24.878187+0000","last_peered":"2024-04-25T05:42:24.878187+0000","last_clean":"2024-04-25T05:42:24.878187+0000","last_became_active":"2024-04-25T05:40:43.446526+0000","last_became_peered":"2024-04-25T05:40:43.446526+0000","last_unstale":"2024-04-25T05:42:24.878187+0000","last_undegraded":"2024-04-25T05:42:24.878187+0000","last_fullsized":"2024-04-25T05:42:24.878187+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:34.867005+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":184,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871192+0000","last_change":"2024-04-25T05:42:14.225323+0000","last_active":"2024-04-25T05:42:24.871192+0000","last_peered":"2024-04-25T05:42:24.871192+0000","last_clean":"2024-04-25T05:42:24.871192+0000","last_became_active":"2024-04-25T05:40:43.429667+0000","last_became_peered":"2024-04-25T05:40:43.429667+0000","last_unstale":"2024-04-25T05:42:24.871192+0000","last_undegraded":"2024-04-25T05:42:24.871192+0000","last_fullsized":"2024-04-25T05:42:24.871192+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.225203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.225203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.751205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"279'12941","reported_seq":22052,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.247539+0000","last_change":"2024-04-25T05:42:12.209425+0000","last_active":"2024-04-25T05:42:28.247539+0000","last_peered":"2024-04-25T05:42:28.247539+0000","last_clean":"2024-04-25T05:42:28.247539+0000","last_became_active":"2024-04-25T05:42:12.177948+0000","last_became_peered":"2024-04-25T05:42:12.177948+0000","last_unstale":"2024-04-25T05:42:28.247539+0000","last_undegraded":"2024-04-25T05:42:28.247539+0000","last_fullsized":"2024-04-25T05:42:28.247539+0000","mapping_epoch":274,"log_start":"247'8239","ondisk_log_start":"247'8239","created":198,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":999,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155910144,"num_objects":2379,"num_object_clones":0,"num_object_copies":7137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12941,"num_write_kb":490240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4450,"num_bytes_recovered":291635200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":633,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874851+0000","last_change":"2024-04-25T05:41:31.093059+0000","last_active":"2024-04-25T05:42:24.874851+0000","last_peered":"2024-04-25T05:42:24.874851+0000","last_clean":"2024-04-25T05:42:24.874851+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:42:24.874851+0000","last_undegraded":"2024-04-25T05:42:24.874851+0000","last_fullsized":"2024-04-25T05:42:24.874851+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:31.092923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.092923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.069935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":117,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874375+0000","last_change":"2024-04-25T05:42:14.731871+0000","last_active":"2024-04-25T05:42:24.874375+0000","last_peered":"2024-04-25T05:42:24.874375+0000","last_clean":"2024-04-25T05:42:24.874375+0000","last_became_active":"2024-04-25T05:40:55.970110+0000","last_became_peered":"2024-04-25T05:40:55.970110+0000","last_unstale":"2024-04-25T05:42:24.874375+0000","last_undegraded":"2024-04-25T05:42:24.874375+0000","last_fullsized":"2024-04-25T05:42:24.874375+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.632461+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":461,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873402+0000","last_change":"2024-04-25T05:40:55.975334+0000","last_active":"2024-04-25T05:42:24.873402+0000","last_peered":"2024-04-25T05:42:24.873402+0000","last_clean":"2024-04-25T05:42:24.873402+0000","last_became_active":"2024-04-25T05:40:55.974871+0000","last_became_peered":"2024-04-25T05:40:55.974871+0000","last_unstale":"2024-04-25T05:42:24.873402+0000","last_undegraded":"2024-04-25T05:42:24.873402+0000","last_fullsized":"2024-04-25T05:42:24.873402+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":109,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865662+0000","last_change":"2024-04-25T05:42:18.092205+0000","last_active":"2024-04-25T05:42:24.865662+0000","last_peered":"2024-04-25T05:42:24.865662+0000","last_clean":"2024-04-25T05:42:24.865662+0000","last_became_active":"2024-04-25T05:40:55.964721+0000","last_became_peered":"2024-04-25T05:40:55.964721+0000","last_unstale":"2024-04-25T05:42:24.865662+0000","last_undegraded":"2024-04-25T05:42:24.865662+0000","last_fullsized":"2024-04-25T05:42:24.865662+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:30.000606+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"233'12","reported_seq":128,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874396+0000","last_change":"2024-04-25T05:40:43.475139+0000","last_active":"2024-04-25T05:42:24.874396+0000","last_peered":"2024-04-25T05:42:24.874396+0000","last_clean":"2024-04-25T05:42:24.874396+0000","last_became_active":"2024-04-25T05:40:43.445833+0000","last_became_peered":"2024-04-25T05:40:43.445833+0000","last_unstale":"2024-04-25T05:42:24.874396+0000","last_undegraded":"2024-04-25T05:42:24.874396+0000","last_fullsized":"2024-04-25T05:42:24.874396+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:40:06.822122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.822122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674499+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"279'13188","reported_seq":27437,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.862547+0000","last_change":"2024-04-25T05:41:28.231423+0000","last_active":"2024-04-25T05:42:27.862547+0000","last_peered":"2024-04-25T05:42:27.862547+0000","last_clean":"2024-04-25T05:42:27.862547+0000","last_became_active":"2024-04-25T05:41:28.213582+0000","last_became_peered":"2024-04-25T05:41:28.213582+0000","last_unstale":"2024-04-25T05:42:27.862547+0000","last_undegraded":"2024-04-25T05:42:27.862547+0000","last_fullsized":"2024-04-25T05:42:27.862547+0000","mapping_epoch":255,"log_start":"250'8829","ondisk_log_start":"250'8829","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:30.005434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156041240,"num_objects":2382,"num_object_clones":0,"num_object_copies":7146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13188,"num_write_kb":498177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7030,"num_bytes_recovered":460652568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":629,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866581+0000","last_change":"2024-04-25T05:42:22.285373+0000","last_active":"2024-04-25T05:42:24.866581+0000","last_peered":"2024-04-25T05:42:24.866581+0000","last_clean":"2024-04-25T05:42:24.866581+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:42:24.866581+0000","last_undegraded":"2024-04-25T05:42:24.866581+0000","last_fullsized":"2024-04-25T05:42:24.866581+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:29.371504+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":87,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.883582+0000","last_change":"2024-04-25T05:40:55.981504+0000","last_active":"2024-04-25T05:42:24.883582+0000","last_peered":"2024-04-25T05:42:24.883582+0000","last_clean":"2024-04-25T05:42:24.883582+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:42:24.883582+0000","last_undegraded":"2024-04-25T05:42:24.883582+0000","last_fullsized":"2024-04-25T05:42:24.883582+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.781719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.781719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":107,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874409+0000","last_change":"2024-04-25T05:40:43.463116+0000","last_active":"2024-04-25T05:42:24.874409+0000","last_peered":"2024-04-25T05:42:24.874409+0000","last_clean":"2024-04-25T05:42:24.874409+0000","last_became_active":"2024-04-25T05:40:43.443728+0000","last_became_peered":"2024-04-25T05:40:43.443728+0000","last_unstale":"2024-04-25T05:42:24.874409+0000","last_undegraded":"2024-04-25T05:42:24.874409+0000","last_fullsized":"2024-04-25T05:42:24.874409+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":185,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871464+0000","last_change":"2024-04-25T05:40:48.391768+0000","last_active":"2024-04-25T05:42:24.871464+0000","last_peered":"2024-04-25T05:42:24.871464+0000","last_clean":"2024-04-25T05:42:24.871464+0000","last_became_active":"2024-04-25T05:40:09.733866+0000","last_became_peered":"2024-04-25T05:40:09.733866+0000","last_unstale":"2024-04-25T05:42:24.871464+0000","last_undegraded":"2024-04-25T05:42:24.871464+0000","last_fullsized":"2024-04-25T05:42:24.871464+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:40:48.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.391383+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":557,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867478+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:42:24.867478+0000","last_peered":"2024-04-25T05:42:24.867478+0000","last_clean":"2024-04-25T05:42:24.867478+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:42:24.867478+0000","last_undegraded":"2024-04-25T05:42:24.867478+0000","last_fullsized":"2024-04-25T05:42:24.867478+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:17.034593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"279'13243","reported_seq":23440,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.272125+0000","last_change":"2024-04-25T05:42:06.908338+0000","last_active":"2024-04-25T05:42:28.272125+0000","last_peered":"2024-04-25T05:42:28.272125+0000","last_clean":"2024-04-25T05:42:28.272125+0000","last_became_active":"2024-04-25T05:42:06.870659+0000","last_became_peered":"2024-04-25T05:42:06.870659+0000","last_unstale":"2024-04-25T05:42:28.272125+0000","last_undegraded":"2024-04-25T05:42:28.272125+0000","last_fullsized":"2024-04-25T05:42:28.272125+0000","mapping_epoch":270,"log_start":"247'8468","ondisk_log_start":"247'8468","created":198,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:29.674230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152371200,"num_objects":2325,"num_object_clones":0,"num_object_copies":6975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13243,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5005,"num_bytes_recovered":328007680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":647,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867458+0000","last_change":"2024-04-25T05:40:58.920790+0000","last_active":"2024-04-25T05:42:24.867458+0000","last_peered":"2024-04-25T05:42:24.867458+0000","last_clean":"2024-04-25T05:42:24.867458+0000","last_became_active":"2024-04-25T05:40:58.920400+0000","last_became_peered":"2024-04-25T05:40:58.920400+0000","last_unstale":"2024-04-25T05:42:24.867458+0000","last_undegraded":"2024-04-25T05:42:24.867458+0000","last_fullsized":"2024-04-25T05:42:24.867458+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:37.793923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.793923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":131,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867648+0000","last_change":"2024-04-25T05:40:55.975104+0000","last_active":"2024-04-25T05:42:24.867648+0000","last_peered":"2024-04-25T05:42:24.867648+0000","last_clean":"2024-04-25T05:42:24.867648+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:42:24.867648+0000","last_undegraded":"2024-04-25T05:42:24.867648+0000","last_fullsized":"2024-04-25T05:42:24.867648+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.719427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.719427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867533+0000","last_change":"2024-04-25T05:40:43.516969+0000","last_active":"2024-04-25T05:42:24.867533+0000","last_peered":"2024-04-25T05:42:24.867533+0000","last_clean":"2024-04-25T05:42:24.867533+0000","last_became_active":"2024-04-25T05:40:43.440253+0000","last_became_peered":"2024-04-25T05:40:43.440253+0000","last_unstale":"2024-04-25T05:42:24.867533+0000","last_undegraded":"2024-04-25T05:42:24.867533+0000","last_fullsized":"2024-04-25T05:42:24.867533+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:43.516897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.516897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":563,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867365+0000","last_change":"2024-04-25T05:40:48.519454+0000","last_active":"2024-04-25T05:42:24.867365+0000","last_peered":"2024-04-25T05:42:24.867365+0000","last_clean":"2024-04-25T05:42:24.867365+0000","last_became_active":"2024-04-25T05:40:43.440689+0000","last_became_peered":"2024-04-25T05:40:43.440689+0000","last_unstale":"2024-04-25T05:42:24.867365+0000","last_undegraded":"2024-04-25T05:42:24.867365+0000","last_fullsized":"2024-04-25T05:42:24.867365+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:40:48.519288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.519288+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866729+0000","last_change":"2024-04-25T05:42:19.974041+0000","last_active":"2024-04-25T05:42:24.866729+0000","last_peered":"2024-04-25T05:42:24.866729+0000","last_clean":"2024-04-25T05:42:24.866729+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:42:24.866729+0000","last_undegraded":"2024-04-25T05:42:24.866729+0000","last_fullsized":"2024-04-25T05:42:24.866729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.973911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.973911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.078951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"279'13011","reported_seq":13969,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.397962+0000","last_change":"2024-04-25T05:42:21.320689+0000","last_active":"2024-04-25T05:42:28.397962+0000","last_peered":"2024-04-25T05:42:28.397962+0000","last_clean":"2024-04-25T05:42:28.397962+0000","last_became_active":"2024-04-25T05:39:34.780422+0000","last_became_peered":"2024-04-25T05:39:34.780422+0000","last_unstale":"2024-04-25T05:42:28.397962+0000","last_undegraded":"2024-04-25T05:42:28.397962+0000","last_fullsized":"2024-04-25T05:42:28.397962+0000","mapping_epoch":208,"log_start":"249'8669","ondisk_log_start":"249'8669","created":198,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'12380","last_scrub_stamp":"2024-04-25T05:42:21.320336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.320336+0000","objects_scrubbed":2986,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:44.979593+0000","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151584768,"num_objects":2313,"num_object_clones":0,"num_object_copies":6939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":490368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":24117248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":639,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.870556+0000","last_change":"2024-04-25T05:40:58.924826+0000","last_active":"2024-04-25T05:42:24.870556+0000","last_peered":"2024-04-25T05:42:24.870556+0000","last_clean":"2024-04-25T05:42:24.870556+0000","last_became_active":"2024-04-25T05:40:58.923753+0000","last_became_peered":"2024-04-25T05:40:58.923753+0000","last_unstale":"2024-04-25T05:42:24.870556+0000","last_undegraded":"2024-04-25T05:42:24.870556+0000","last_fullsized":"2024-04-25T05:42:24.870556+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:57.405761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:57.405761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":462,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.873148+0000","last_change":"2024-04-25T05:40:50.460093+0000","last_active":"2024-04-25T05:42:24.873148+0000","last_peered":"2024-04-25T05:42:24.873148+0000","last_clean":"2024-04-25T05:42:24.873148+0000","last_became_active":"2024-04-25T05:40:43.454117+0000","last_became_peered":"2024-04-25T05:40:43.454117+0000","last_unstale":"2024-04-25T05:42:24.873148+0000","last_undegraded":"2024-04-25T05:42:24.873148+0000","last_fullsized":"2024-04-25T05:42:24.873148+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"233'11","reported_seq":196,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.872676+0000","last_change":"2024-04-25T05:41:38.552749+0000","last_active":"2024-04-25T05:42:24.872676+0000","last_peered":"2024-04-25T05:42:24.872676+0000","last_clean":"2024-04-25T05:42:24.872676+0000","last_became_active":"2024-04-25T05:40:43.453268+0000","last_became_peered":"2024-04-25T05:40:43.453268+0000","last_unstale":"2024-04-25T05:42:24.872676+0000","last_undegraded":"2024-04-25T05:42:24.872676+0000","last_fullsized":"2024-04-25T05:42:24.872676+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:52.068035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":454,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867682+0000","last_change":"2024-04-25T05:40:51.592334+0000","last_active":"2024-04-25T05:42:24.867682+0000","last_peered":"2024-04-25T05:42:24.867682+0000","last_clean":"2024-04-25T05:42:24.867682+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:42:24.867682+0000","last_undegraded":"2024-04-25T05:42:24.867682+0000","last_fullsized":"2024-04-25T05:42:24.867682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.592111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.592111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":482,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.868158+0000","last_change":"2024-04-25T05:39:39.713342+0000","last_active":"2024-04-25T05:42:24.868158+0000","last_peered":"2024-04-25T05:42:24.868158+0000","last_clean":"2024-04-25T05:42:24.868158+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:42:24.868158+0000","last_undegraded":"2024-04-25T05:42:24.868158+0000","last_fullsized":"2024-04-25T05:42:24.868158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.713165+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"279'13026","reported_seq":27563,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:27.753917+0000","last_change":"2024-04-25T05:41:28.229751+0000","last_active":"2024-04-25T05:42:27.753917+0000","last_peered":"2024-04-25T05:42:27.753917+0000","last_clean":"2024-04-25T05:42:27.753917+0000","last_became_active":"2024-04-25T05:41:28.214541+0000","last_became_peered":"2024-04-25T05:41:28.214541+0000","last_unstale":"2024-04-25T05:42:27.753917+0000","last_undegraded":"2024-04-25T05:42:27.753917+0000","last_fullsized":"2024-04-25T05:42:27.753917+0000","mapping_epoch":255,"log_start":"250'8718","ondisk_log_start":"250'8718","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'6914","last_scrub_stamp":"2024-04-25T05:40:40.935916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.935916+0000","objects_scrubbed":6834,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.992837+0000","scrub_duration":2117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157024256,"num_objects":2396,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13026,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7133,"num_bytes_recovered":467468288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":446,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866752+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:42:24.866752+0000","last_peered":"2024-04-25T05:42:24.866752+0000","last_clean":"2024-04-25T05:42:24.866752+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:42:24.866752+0000","last_undegraded":"2024-04-25T05:42:24.866752+0000","last_fullsized":"2024-04-25T05:42:24.866752+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.874841+0000","last_change":"2024-04-25T05:41:27.501946+0000","last_active":"2024-04-25T05:42:24.874841+0000","last_peered":"2024-04-25T05:42:24.874841+0000","last_clean":"2024-04-25T05:42:24.874841+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:42:24.874841+0000","last_undegraded":"2024-04-25T05:42:24.874841+0000","last_fullsized":"2024-04-25T05:42:24.874841+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.501828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.501828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:35.447143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":467,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866814+0000","last_change":"2024-04-25T05:40:55.964874+0000","last_active":"2024-04-25T05:42:24.866814+0000","last_peered":"2024-04-25T05:42:24.866814+0000","last_clean":"2024-04-25T05:42:24.866814+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:42:24.866814+0000","last_undegraded":"2024-04-25T05:42:24.866814+0000","last_fullsized":"2024-04-25T05:42:24.866814+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:39.870293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:39.870293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":471,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871749+0000","last_change":"2024-04-25T05:40:09.839273+0000","last_active":"2024-04-25T05:42:24.871749+0000","last_peered":"2024-04-25T05:42:24.871749+0000","last_clean":"2024-04-25T05:42:24.871749+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:42:24.871749+0000","last_undegraded":"2024-04-25T05:42:24.871749+0000","last_fullsized":"2024-04-25T05:42:24.871749+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:09.838661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.838661+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:30.251145+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"279'13480","reported_seq":24229,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.389404+0000","last_change":"2024-04-25T05:42:15.166676+0000","last_active":"2024-04-25T05:42:28.389404+0000","last_peered":"2024-04-25T05:42:28.389404+0000","last_clean":"2024-04-25T05:42:28.389404+0000","last_became_active":"2024-04-25T05:42:07.869166+0000","last_became_peered":"2024-04-25T05:42:07.869166+0000","last_unstale":"2024-04-25T05:42:28.389404+0000","last_undegraded":"2024-04-25T05:42:28.389404+0000","last_fullsized":"2024-04-25T05:42:28.389404+0000","mapping_epoch":271,"log_start":"251'9152","ondisk_log_start":"251'9152","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'12381","last_scrub_stamp":"2024-04-25T05:42:15.166473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.166473+0000","objects_scrubbed":3559,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:26.495367+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157679616,"num_objects":2406,"num_object_clones":0,"num_object_copies":7218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":508352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5255,"num_bytes_recovered":344391680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":634,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871162+0000","last_change":"2024-04-25T05:40:58.929257+0000","last_active":"2024-04-25T05:42:24.871162+0000","last_peered":"2024-04-25T05:42:24.871162+0000","last_clean":"2024-04-25T05:42:24.871162+0000","last_became_active":"2024-04-25T05:40:58.926534+0000","last_became_peered":"2024-04-25T05:40:58.926534+0000","last_unstale":"2024-04-25T05:42:24.871162+0000","last_undegraded":"2024-04-25T05:42:24.871162+0000","last_fullsized":"2024-04-25T05:42:24.871162+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:38:59.374826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:38:59.374826+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867058+0000","last_change":"2024-04-25T05:40:55.974556+0000","last_active":"2024-04-25T05:42:24.867058+0000","last_peered":"2024-04-25T05:42:24.867058+0000","last_clean":"2024-04-25T05:42:24.867058+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:42:24.867058+0000","last_undegraded":"2024-04-25T05:42:24.867058+0000","last_fullsized":"2024-04-25T05:42:24.867058+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.019675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.019675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":472,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866157+0000","last_change":"2024-04-25T05:40:44.537276+0000","last_active":"2024-04-25T05:42:24.866157+0000","last_peered":"2024-04-25T05:42:24.866157+0000","last_clean":"2024-04-25T05:42:24.866157+0000","last_became_active":"2024-04-25T05:40:43.449807+0000","last_became_peered":"2024-04-25T05:40:43.449807+0000","last_unstale":"2024-04-25T05:42:24.866157+0000","last_undegraded":"2024-04-25T05:42:24.866157+0000","last_fullsized":"2024-04-25T05:42:24.866157+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":178,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.871102+0000","last_change":"2024-04-25T05:42:19.197580+0000","last_active":"2024-04-25T05:42:24.871102+0000","last_peered":"2024-04-25T05:42:24.871102+0000","last_clean":"2024-04-25T05:42:24.871102+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:42:24.871102+0000","last_undegraded":"2024-04-25T05:42:24.871102+0000","last_fullsized":"2024-04-25T05:42:24.871102+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.197467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.197467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:47.403991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"277'25","reported_seq":169,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865659+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:42:24.865659+0000","last_peered":"2024-04-25T05:42:24.865659+0000","last_clean":"2024-04-25T05:42:24.865659+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:42:24.865659+0000","last_undegraded":"2024-04-25T05:42:24.865659+0000","last_fullsized":"2024-04-25T05:42:24.865659+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:42.840863+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"279'13067","reported_seq":13240,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:28.398901+0000","last_change":"2024-04-25T05:39:19.231808+0000","last_active":"2024-04-25T05:42:28.398901+0000","last_peered":"2024-04-25T05:42:28.398901+0000","last_clean":"2024-04-25T05:42:28.398901+0000","last_became_active":"2024-04-25T05:39:19.231525+0000","last_became_peered":"2024-04-25T05:39:19.231525+0000","last_unstale":"2024-04-25T05:42:28.398901+0000","last_undegraded":"2024-04-25T05:42:28.398901+0000","last_fullsized":"2024-04-25T05:42:28.398901+0000","mapping_epoch":198,"log_start":"251'8951","ondisk_log_start":"251'8951","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4116,"log_dups_size":0,"ondisk_log_size":4116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:31.325257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148963328,"num_objects":2273,"num_object_clones":0,"num_object_copies":6819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13067,"num_write_kb":490880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":769,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.882708+0000","last_change":"2024-04-25T05:42:20.218748+0000","last_active":"2024-04-25T05:42:24.882708+0000","last_peered":"2024-04-25T05:42:24.882708+0000","last_clean":"2024-04-25T05:42:24.882708+0000","last_became_active":"2024-04-25T05:40:58.930089+0000","last_became_peered":"2024-04-25T05:40:58.930089+0000","last_unstale":"2024-04-25T05:42:24.882708+0000","last_undegraded":"2024-04-25T05:42:24.882708+0000","last_fullsized":"2024-04-25T05:42:24.882708+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.920803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":131,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.865869+0000","last_change":"2024-04-25T05:40:55.964673+0000","last_active":"2024-04-25T05:42:24.865869+0000","last_peered":"2024-04-25T05:42:24.865869+0000","last_clean":"2024-04-25T05:42:24.865869+0000","last_became_active":"2024-04-25T05:40:55.964302+0000","last_became_peered":"2024-04-25T05:40:55.964302+0000","last_unstale":"2024-04-25T05:42:24.865869+0000","last_undegraded":"2024-04-25T05:42:24.865869+0000","last_fullsized":"2024-04-25T05:42:24.865869+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:02.813078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.813078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:32.019275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"243'10","reported_seq":216,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.885660+0000","last_change":"2024-04-25T05:42:21.213565+0000","last_active":"2024-04-25T05:42:24.885660+0000","last_peered":"2024-04-25T05:42:24.885660+0000","last_clean":"2024-04-25T05:42:24.885660+0000","last_became_active":"2024-04-25T05:40:55.955721+0000","last_became_peered":"2024-04-25T05:40:55.955721+0000","last_unstale":"2024-04-25T05:42:24.885660+0000","last_undegraded":"2024-04-25T05:42:24.885660+0000","last_fullsized":"2024-04-25T05:42:24.885660+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.386980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":466,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.866577+0000","last_change":"2024-04-25T05:42:17.311411+0000","last_active":"2024-04-25T05:42:24.866577+0000","last_peered":"2024-04-25T05:42:24.866577+0000","last_clean":"2024-04-25T05:42:24.866577+0000","last_became_active":"2024-04-25T05:40:43.432950+0000","last_became_peered":"2024-04-25T05:40:43.432950+0000","last_unstale":"2024-04-25T05:42:24.866577+0000","last_undegraded":"2024-04-25T05:42:24.866577+0000","last_fullsized":"2024-04-25T05:42:24.866577+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.857445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":167,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-04-25T05:42:24.867150+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:42:24.867150+0000","last_peered":"2024-04-25T05:42:24.867150+0000","last_clean":"2024-04-25T05:42:24.867150+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:42:24.867150+0000","last_undegraded":"2024-04-25T05:42:24.867150+0000","last_fullsized":"2024-04-25T05:42:24.867150+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.683068+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2475229208,"num_objects":37770,"num_object_clones":0,"num_object_copies":113310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210474,"num_write_kb":7943745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63646,"num_bytes_recovered":4170973208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8686743552,"data_stored":8686731336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70854,"ondisk_log_size":70854,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":496,"num_read_kb":491,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":974,"num_read_kb":699,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":5396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":660,"ondisk_log_size":660,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070571,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265964,"kb_used_data":941860,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92479188,"statfs":{"total":95995035648,"available":94698688512,"internally_reserved":0,"allocated":964464640,"data_stored":963961248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364911,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417524,"kb_used_data":1097572,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92327628,"statfs":{"total":95995035648,"available":94543491072,"internally_reserved":0,"allocated":1123913728,"data_stored":1123410250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449476,"kb_used_data":1057276,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92295676,"statfs":{"total":95995035648,"available":94510772224,"internally_reserved":0,"allocated":1082650624,"data_stored":1082121780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322152,"kb_used_data":930016,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92423000,"statfs":{"total":95995035648,"available":94641152000,"internally_reserved":0,"allocated":952336384,"data_stored":951835551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2036536,"kb_used_data":1576176,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91708616,"statfs":{"total":95995035648,"available":93909622784,"internally_reserved":0,"allocated":1614004224,"data_stored":1613487844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":471202247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986296,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1273964,"kb_used_data":959588,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92471188,"statfs":{"total":95995035648,"available":94690496512,"internally_reserved":0,"allocated":982618112,"data_stored":982113780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691956,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1977952,"kb_used_data":1631640,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91767200,"statfs":{"total":95995035648,"available":93969612800,"internally_reserved":0,"allocated":1670799360,"data_stored":1670307288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":929562624,"data_stored":929562624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1065418752,"data_stored":1065418752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":463273984,"data_stored":463273984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1605238784,"data_stored":1605238784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":954597376,"data_stored":954597376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1625559040,"data_stored":1625554968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1099173888,"data_stored":1099169816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":943919104,"data_stored":943915032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:42:36.029 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-25T05:42:36.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 26 2024-04-25T05:42:36.049 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:36.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.196 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:36.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:36.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:36.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:36.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:36.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:36.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:36.663 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:36.664 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:42:36.664 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:42:36.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:36.738 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:36.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:36.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:36.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:36.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:37.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:42:37.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:37.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:36 smithi043 ceph-mon[27010]: pgmap v448: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:42:37.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:36 smithi043 ceph-mon[37492]: pgmap v448: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:42:37.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:36 smithi060 ceph-mon[30898]: pgmap v448: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-04-25T05:42:37.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:37.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:37.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:37.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:37.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:37.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:42:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:37.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:37.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:37.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:37.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:37.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:37.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:37.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:38.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:38.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:38.296 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:38.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:42:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:38.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:38.332 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:38.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:38.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:38.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:38.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:38.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:38.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:38.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:38.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 8 pg_num to 26 2024-04-25T05:42:38.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:39.036 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:39.063 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:39.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:38 smithi043 ceph-mon[27010]: pgmap v449: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-04-25T05:42:39.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:42:39.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:38 smithi043 ceph-mon[37492]: pgmap v449: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-04-25T05:42:39.091 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:42:39.109 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:39.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:39.148 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.168 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:39.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:39.187 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:39.232 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:39.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:38 smithi060 ceph-mon[30898]: pgmap v449: 158 pgs: 158 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-04-25T05:42:39.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:42:39.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:39.285 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:39.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:39.330 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:39.363 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:39.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:39.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:39.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:39.581 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:39.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:39.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:39.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:39.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:39.926 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:39.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:39.991 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:39.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:40.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.143 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:40.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:40.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:40.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:42:40.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: osdmap e282: 8 total, 8 up, 7 in 2024-04-25T05:42:40.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3749441504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:40.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:40.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:42:40.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:39 smithi060 ceph-mon[30898]: osdmap e283: 8 total, 8 up, 7 in 2024-04-25T05:42:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.266 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:40.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:40.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:42:40.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: osdmap e282: 8 total, 8 up, 7 in 2024-04-25T05:42:40.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3749441504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:40.287 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[37492]: osdmap e283: 8 total, 8 up, 7 in 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4168048817' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: osdmap e282: 8 total, 8 up, 7 in 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3749441504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:42:40.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:39 smithi043 ceph-mon[27010]: osdmap e283: 8 total, 8 up, 7 in 2024-04-25T05:42:40.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:40.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:40.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:40.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:40.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:40.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:40.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:40.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:40.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:40.953 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:42:40.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:41.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:41.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:41.158 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:40 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:40 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:40 smithi060 ceph-mon[30898]: pgmap v451: 158 pgs: 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 816 B/s rd, 0 B/s wr, 1.10k op/s; 17 B/s, 0 objects/s recovering 2024-04-25T05:42:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3093926094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.295 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[27010]: pgmap v451: 158 pgs: 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 816 B/s rd, 0 B/s wr, 1.10k op/s; 17 B/s, 0 objects/s recovering 2024-04-25T05:42:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3093926094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[37492]: pgmap v451: 158 pgs: 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 816 B/s rd, 0 B/s wr, 1.10k op/s; 17 B/s, 0 objects/s recovering 2024-04-25T05:42:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3093926094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:42:41.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:41.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:41.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:41.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:41.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:41.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:41.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:41.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:41.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:42.047 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: osdmap e284: 8 total, 8 up, 7 in 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.8 scrub starts 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.2 scrub starts 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.0 scrub starts 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.9 scrub starts 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.2 scrub ok 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.0 scrub ok 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: 8.9 scrub ok 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:42.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: osdmap e284: 8 total, 8 up, 7 in 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.8 scrub starts 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.2 scrub starts 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.0 scrub starts 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.9 scrub starts 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.2 scrub ok 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.0 scrub ok 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: 8.9 scrub ok 2024-04-25T05:42:42.049 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:41 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:42.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:42.093 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:42:42.093 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:42:42.093 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:42:42.093 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:42:42.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:42.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:42.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: osdmap e284: 8 total, 8 up, 7 in 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.8 scrub starts 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.2 scrub starts 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.0 scrub starts 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.9 scrub starts 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.2 scrub ok 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.0 scrub ok 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: 8.9 scrub ok 2024-04-25T05:42:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:41 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:42:42.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:42.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:42.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.469 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:42.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:42.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:42:42.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:42.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:42.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:42.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:42.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:42.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:42.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:42.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:43.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:43.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:43.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.8 scrub ok 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.6 scrub starts 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.6 scrub ok 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: pgmap v454: 168 pgs: 10 unknown, 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.3 scrub starts 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:42:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: osdmap e285: 8 total, 8 up, 7 in 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.3 scrub ok 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.4 scrub starts 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.4 scrub ok 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.f scrub starts 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.5 scrub starts 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.5 scrub ok 2024-04-25T05:42:43.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:42 smithi060 ceph-mon[30898]: 8.f scrub ok 2024-04-25T05:42:43.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:43.330 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:43.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.8 scrub ok 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.6 scrub starts 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.6 scrub ok 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: pgmap v454: 168 pgs: 10 unknown, 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.3 scrub starts 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: osdmap e285: 8 total, 8 up, 7 in 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.3 scrub ok 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.4 scrub starts 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.4 scrub ok 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.f scrub starts 2024-04-25T05:42:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.5 scrub starts 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.5 scrub ok 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[27010]: 8.f scrub ok 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.8 scrub ok 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.6 scrub starts 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.6 scrub ok 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: pgmap v454: 168 pgs: 10 unknown, 158 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.3 scrub starts 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: osdmap e285: 8 total, 8 up, 7 in 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.3 scrub ok 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.4 scrub starts 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:42:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:42:43.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.4 scrub ok 2024-04-25T05:42:43.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.f scrub starts 2024-04-25T05:42:43.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.5 scrub starts 2024-04-25T05:42:43.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.5 scrub ok 2024-04-25T05:42:43.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:42 smithi043 ceph-mon[37492]: 8.f scrub ok 2024-04-25T05:42:43.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:43.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:43.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.582 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:43.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:43.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:43.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:43.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:43.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:43.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:43.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:42:43.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:43.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:44.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:44.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:44.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: 8.1 scrub starts 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: 8.1 scrub ok 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: osdmap e286: 8 total, 8 up, 7 in 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: 8.b scrub starts 2024-04-25T05:42:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:43 smithi060 ceph-mon[30898]: 8.b scrub ok 2024-04-25T05:42:44.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:44.360 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: 8.1 scrub starts 2024-04-25T05:42:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: 8.1 scrub ok 2024-04-25T05:42:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: osdmap e286: 8 total, 8 up, 7 in 2024-04-25T05:42:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: 8.b scrub starts 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[27010]: 8.b scrub ok 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: 8.1 scrub starts 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: 8.1 scrub ok 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: osdmap e286: 8 total, 8 up, 7 in 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: 8.b scrub starts 2024-04-25T05:42:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:43 smithi043 ceph-mon[37492]: 8.b scrub ok 2024-04-25T05:42:44.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:44.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:42:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.499 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:44.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:44.636 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:44.708 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:44.714 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:44.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:44.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:44.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:44.908 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:44.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:45.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:45.045 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:45.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: pgmap v457: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:42:45.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.7 scrub starts 2024-04-25T05:42:45.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 284'13984 2024-04-25T05:42:45.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: osdmap e287: 8 total, 8 up, 7 in 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14019 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 284'14343 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14110 2024-04-25T05:42:45.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.137 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 8.7 scrub ok 2024-04-25T05:42:45.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:42:45.138 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: pgmap v457: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.7 scrub starts 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 284'13984 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: osdmap e287: 8 total, 8 up, 7 in 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14019 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.139 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 284'14343 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14110 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.140 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 8.7 scrub ok 2024-04-25T05:42:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:42:45.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:44 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:42:45.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.183 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:45.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: pgmap v457: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.7 scrub starts 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 284'13984 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: osdmap e287: 8 total, 8 up, 7 in 2024-04-25T05:42:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14019 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 284'14343 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14110 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14439 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14052 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14508 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 284'14206 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14048 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 284'14160 2024-04-25T05:42:45.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 8.7 scrub ok 2024-04-25T05:42:45.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:42:45.251 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:44 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:42:45.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:45.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:45.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.462 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:45.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:45.564 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:45.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:45.651 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:45.691 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:45.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:45.732 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:45.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:45.777 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:45.831 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:45.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:45.879 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:45.880 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:45.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:45.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:45.929 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:45.984 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:45.990 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:46.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:46.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:46.138 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 6, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:42:46.138 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:42:46.138 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T05:42:46.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:42:46.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:46.178 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:46.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:42:46.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:42:46.187 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4098685313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:46.188 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:42:46.188 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:42:46.188 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4098685313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:45 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:42:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:45 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:42:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4098685313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:46.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:46.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:46.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:46.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:46.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:46.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:46.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:46.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:46.754 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:42:46.754 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:42:46.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:46.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:46.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:46.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:47.018 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:42:47.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:47.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:47.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:47.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:47.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[27010]: pgmap v459: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[27010]: osdmap e288: 8 total, 8 up, 7 in 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/742052587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[37492]: pgmap v459: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[37492]: osdmap e288: 8 total, 8 up, 7 in 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/742052587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:47.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:47.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:47 smithi060 ceph-mon[30898]: pgmap v459: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:42:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:47 smithi060 ceph-mon[30898]: osdmap e288: 8 total, 8 up, 7 in 2024-04-25T05:42:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/742052587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:42:47.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:47.576 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:47.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.719 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:47.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:47.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:47.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:47.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:47.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:47.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:48.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:48.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:48.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.249 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:48.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:48.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:48.373 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:42:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:48.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: osdmap e289: 8 total, 8 up, 7 in 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: osdmap e289: 8 total, 8 up, 7 in 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:48.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:48.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:42:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:42:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:42:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: osdmap e289: 8 total, 8 up, 7 in 2024-04-25T05:42:48.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:42:48.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:42:48.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:48.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:48.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:42:48.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:48.751 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:48.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:48.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:48.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:48.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:48.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:49.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:49.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:49.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:49.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:49.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:49.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:49.285 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:49.342 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:42:49.364 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: pgmap v462: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail 2024-04-25T05:42:49.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: osdmap e290: 8 total, 8 up, 7 in 2024-04-25T05:42:49.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: 5.1c deep-scrub starts 2024-04-25T05:42:49.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: 5.1c deep-scrub ok 2024-04-25T05:42:49.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:42:49.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:42:49.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: pgmap v462: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail 2024-04-25T05:42:49.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: osdmap e290: 8 total, 8 up, 7 in 2024-04-25T05:42:49.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: 5.1c deep-scrub starts 2024-04-25T05:42:49.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: 5.1c deep-scrub ok 2024-04-25T05:42:49.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:42:49.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:42:49.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:49.400 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: pgmap v462: 168 pgs: 6 remapped+peering, 4 unknown, 158 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: osdmap e290: 8 total, 8 up, 7 in 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: 5.1c deep-scrub starts 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: 5.1c deep-scrub ok 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:42:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:49 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:42:49.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:49.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:49.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:49.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:49.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:49.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:49.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:49.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:49.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:49.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:50.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:50.044 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:50.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.154 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:50.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:50.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:50.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.10 scrub starts 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.10 scrub ok 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.11 scrub starts 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: osdmap e291: 8 total, 8 up, 7 in 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.17 scrub starts 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 4.13 deep-scrub starts 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 4.13 deep-scrub ok 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.13 scrub starts 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: 8.13 scrub ok 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[37492]: osdmap e292: 8 total, 8 up, 7 in 2024-04-25T05:42:50.343 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.10 scrub starts 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.10 scrub ok 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.11 scrub starts 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: osdmap e291: 8 total, 8 up, 7 in 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.17 scrub starts 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 4.13 deep-scrub starts 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 4.13 deep-scrub ok 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.13 scrub starts 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: 8.13 scrub ok 2024-04-25T05:42:50.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:50 smithi043 ceph-mon[27010]: osdmap e292: 8 total, 8 up, 7 in 2024-04-25T05:42:50.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:50.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.10 scrub starts 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.10 scrub ok 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.11 scrub starts 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: osdmap e291: 8 total, 8 up, 7 in 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.17 scrub starts 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 4.13 deep-scrub starts 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 4.13 deep-scrub ok 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.13 scrub starts 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: 8.13 scrub ok 2024-04-25T05:42:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:50 smithi060 ceph-mon[30898]: osdmap e292: 8 total, 8 up, 7 in 2024-04-25T05:42:50.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:50.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:50.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:50.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:50.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:50.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:50.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:50.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:51.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:51.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:51.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:51.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:51.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:42:51.276 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:42:51.276 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.17 scrub ok 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: pgmap v465: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 32 MiB/s, 512 objects/s recovering 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.19 scrub starts 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.10 scrub starts 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.19 scrub ok 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.14 scrub starts 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[27010]: 8.14 scrub ok 2024-04-25T05:42:51.313 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.17 scrub ok 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: pgmap v465: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 32 MiB/s, 512 objects/s recovering 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.19 scrub starts 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.10 scrub starts 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.19 scrub ok 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.14 scrub starts 2024-04-25T05:42:51.314 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:51 smithi043 ceph-mon[37492]: 8.14 scrub ok 2024-04-25T05:42:51.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:51.376 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:51.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.17 scrub ok 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: pgmap v465: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 32 MiB/s, 512 objects/s recovering 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.19 scrub starts 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.10 scrub starts 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.19 scrub ok 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.14 scrub starts 2024-04-25T05:42:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:51 smithi060 ceph-mon[30898]: 8.14 scrub ok 2024-04-25T05:42:51.509 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:51.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:51.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:51.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:51.779 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:51.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:51.912 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:51.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:52.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:52.049 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:52.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:52.183 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:52.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:52.316 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 8.10 scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 4.f scrub starts 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 4.f scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 8.11 scrub starts 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 3.e deep-scrub starts 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 3.e deep-scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 8.11 scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 8.10 scrub ok 2024-04-25T05:42:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 4.f scrub starts 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 4.f scrub ok 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 8.11 scrub starts 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 3.e deep-scrub starts 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 3.e deep-scrub ok 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 8.11 scrub ok 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:42:52.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:52 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:42:52.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:52.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 8.10 scrub ok 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 4.f scrub starts 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 4.f scrub ok 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 8.11 scrub starts 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 3.e deep-scrub starts 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 3.e deep-scrub ok 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 8.11 scrub ok 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:42:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:52 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:42:52.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:52.613 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:52.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:52.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:52.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:52.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:52.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:52.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:53.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:53.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:42:53.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:53.149 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:53.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:53.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:53.373 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: pgmap v467: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 6.1f deep-scrub starts 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 4.1f deep-scrub starts 2024-04-25T05:42:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 4.1f deep-scrub ok 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[27010]: 6.1f deep-scrub ok 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: pgmap v467: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 6.1f deep-scrub starts 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 4.1f deep-scrub starts 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 4.1f deep-scrub ok 2024-04-25T05:42:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:53 smithi043 ceph-mon[37492]: 6.1f deep-scrub ok 2024-04-25T05:42:53.407 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:53.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: pgmap v467: 168 pgs: 1 peering, 4 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 158 active+clean; 925 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 3663/45090 objects misplaced (8.124%); 27 MiB/s, 424 objects/s recovering 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 6.1f deep-scrub starts 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 4.1f deep-scrub starts 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 4.1f deep-scrub ok 2024-04-25T05:42:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:53 smithi060 ceph-mon[30898]: 6.1f deep-scrub ok 2024-04-25T05:42:53.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:53.547 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:53.649 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:53.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:53.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:53.814 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:53.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:53.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:53.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:54.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:54.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:54.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:54.223 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:54.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:54.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:54.365 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:42:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:42:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:42:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 6.1b scrub starts 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[27010]: 6.1b scrub ok 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 6.1b scrub starts 2024-04-25T05:42:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:42:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:42:54.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:54 smithi043 ceph-mon[37492]: 6.1b scrub ok 2024-04-25T05:42:54.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 6.1b scrub starts 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:42:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:54 smithi060 ceph-mon[30898]: 6.1b scrub ok 2024-04-25T05:42:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:54.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:54.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:54.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:54.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.762 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:54.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:54.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:54.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:54.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:54.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:55.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:55.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.155 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:55.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:55.289 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:55.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: pgmap v468: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:55.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:42:55.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:42:55.323 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[27010]: osdmap e293: 8 total, 8 up, 7 in 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: pgmap v468: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:42:55.324 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:55 smithi043 ceph-mon[37492]: osdmap e293: 8 total, 8 up, 7 in 2024-04-25T05:42:55.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.417 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: pgmap v468: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:42:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:55 smithi060 ceph-mon[30898]: osdmap e293: 8 total, 8 up, 7 in 2024-04-25T05:42:55.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:55.545 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:55.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.678 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:55.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:55.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:55.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:55.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:56.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.078 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:56.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:56.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:56.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:56.353 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:56.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:42:56.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:42:56.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:42:56.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 4.4 scrub starts 2024-04-25T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 4.4 scrub ok 2024-04-25T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2373629424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:42:56.384 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:42:56.384 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:42:56.384 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 293, 2024-04-25T05:42:56.384 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:42:56.384 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:42:54.865802+0000", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:40:54.892709+0000", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:42:56.385 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 8, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:42:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "174", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "171", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.1700000762939453, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.1700000762939453, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.392 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.393 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:42:56.394 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.395 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.396 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.398 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.400 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.401 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.402 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:42:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:42:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4900000095367432, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4900000095367432, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.190000057220459, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.190000057220459, 2024-04-25T05:42:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.7400000095367432, 2024-04-25T05:42:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.7400000095367432, 2024-04-25T05:42:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T05:42:56.414 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:42:56.414 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:42:56.414 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:42:56.414 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:42:56.414 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 4.4 scrub starts 2024-04-25T05:42:56.415 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 4.4 scrub ok 2024-04-25T05:42:56.415 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:42:56.415 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:42:56.415 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2373629424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:42:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T05:42:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 8, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_1", 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:39:18.082095+0000", 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:42:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 26, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 26, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 26, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 26, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 26, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:42:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "285", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "283", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:42:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:42:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.5299999713897705, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.8400000333786011, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.3500000238418579, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.6200000047683716, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:42:56.427 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 286, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:42:56.428 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.430 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.431 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 290, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.432 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.433 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:42:56.434 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:42:56.435 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 273, 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 290, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:42:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0.610748291015625, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 291, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:42:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:42:56.446 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 291, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.449 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:42:56.450 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 285, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:42:56.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.453 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:42:56.454 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 291, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:42:56.456 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:42:56.457 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.458 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:14.313293+0000", 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:45.578763+0000", 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:42:56.459 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:15.311930+0000", 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.460 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:10.701924+0000", 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:41:41.687034+0000", 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.461 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:07.036631+0000", 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:01.312404+0000", 2024-04-25T05:42:56.462 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:42:39.455963+0000", 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:42:56.463 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.b", 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.12", 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1e", 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 0, 2024-04-25T05:42:56.465 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 6 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 7, 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.466 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 3, 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "8.12", 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-25T05:42:56.467 INFO:teuthology.orchestra.run.smithi043.stdout: 5 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "8.15", 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "8.16", 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:42:56.468 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "8.18", 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:42:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:42:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:42:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:42:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:42:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:42:56.472 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:42:56.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:56.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:56.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 4.4 scrub starts 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 4.4 scrub ok 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:42:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2373629424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:42:56.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:56.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:56.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.764 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:56.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:56.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:56.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:57.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:57.050 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:57.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:57.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:57.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:57.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[27010]: pgmap v470: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[37492]: pgmap v470: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:42:57.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:57 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:42:57.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.454 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:57.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:57.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:57 smithi060 ceph-mon[30898]: pgmap v470: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:57 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:42:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:57 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:42:57.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:42:57.589 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:42:57.664 INFO:tasks.thrashosds.thrasher:j is {'epoch': 293, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:42:54.865802+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:40:54.892709+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '171', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-04-25T05:39:18.082095+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '285', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '283', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.840000033378601, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.350000023841858, 'raw_score_stable': 1.6200000047683716, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 290, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 273, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 290, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.610748291015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 285, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:14.313293+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:45.578763+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:42:15.311930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:10.701924+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:41:41.687034+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:07.036631+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:01.312404+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:42:39.455963+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.3', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '6.9', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.12', 'osds': [0, 4, 5]}, {'pgid': '8.15', 'osds': [7, 6, 4]}, {'pgid': '8.16', 'osds': [0, 6, 7]}, {'pgid': '8.18', 'osds': [3, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:42:57.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:42:57.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:57.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:57.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:57.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:57.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:57.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:58.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:58.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:58.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:58.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:58.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[37492]: 4.19 scrub starts 2024-04-25T05:42:58.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[37492]: 4.19 scrub ok 2024-04-25T05:42:58.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[37492]: 1.6 deep-scrub starts 2024-04-25T05:42:58.288 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[37492]: 1.6 deep-scrub ok 2024-04-25T05:42:58.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[27010]: 4.19 scrub starts 2024-04-25T05:42:58.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[27010]: 4.19 scrub ok 2024-04-25T05:42:58.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[27010]: 1.6 deep-scrub starts 2024-04-25T05:42:58.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:58 smithi043 ceph-mon[27010]: 1.6 deep-scrub ok 2024-04-25T05:42:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:58.305 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:58.380 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:42:58.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.415 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:58.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:58.419 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:42:58.454 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:42:58.494 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:42:58.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:58 smithi060 ceph-mon[30898]: 4.19 scrub starts 2024-04-25T05:42:58.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:58 smithi060 ceph-mon[30898]: 4.19 scrub ok 2024-04-25T05:42:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:58 smithi060 ceph-mon[30898]: 1.6 deep-scrub starts 2024-04-25T05:42:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:58 smithi060 ceph-mon[30898]: 1.6 deep-scrub ok 2024-04-25T05:42:58.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:58.533 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:42:58.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:58.575 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:42:58.617 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:42:58.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:58.661 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:42:58.692 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:58.700 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:42:58.707 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:42:58.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:42:58.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:42:58.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:42:58.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:58.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:42:59.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:59.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: pgmap v471: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:42:59.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/775418956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: pgmap v471: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:42:59.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/775418956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:59.148 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 124121 objects 2024-04-25T05:42:59.148 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :120.719 2024-04-25T05:42:59.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:42:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.233 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:59.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:42:59.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:59.354 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:59.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:42:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:59.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: pgmap v471: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 563 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 1759/27702 objects misplaced (6.350%) 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:42:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:42:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/775418956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:42:59.579 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:42:59.601 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:42:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:42:59.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:42:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:42:59.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:42:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:42:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:42:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:42:59.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:42:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:42:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:42:59.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:00.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:00.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:43:00.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:00.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:00.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[27010]: 6.c scrub starts 2024-04-25T05:43:00.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[27010]: 6.c scrub ok 2024-04-25T05:43:00.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:00.272 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[27010]: 3.10 deep-scrub starts 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[27010]: 3.10 deep-scrub ok 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[37492]: 6.c scrub starts 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[37492]: 6.c scrub ok 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[37492]: 3.10 deep-scrub starts 2024-04-25T05:43:00.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:00 smithi043 ceph-mon[37492]: 3.10 deep-scrub ok 2024-04-25T05:43:00.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:00.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:00.398 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:43:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:43:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:43:00.398 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:43:00.446 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:00.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:00 smithi060 ceph-mon[30898]: 6.c scrub starts 2024-04-25T05:43:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:00 smithi060 ceph-mon[30898]: 6.c scrub ok 2024-04-25T05:43:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:00 smithi060 ceph-mon[30898]: 3.10 deep-scrub starts 2024-04-25T05:43:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:00 smithi060 ceph-mon[30898]: 3.10 deep-scrub ok 2024-04-25T05:43:00.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:00.594 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:00.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:00.764 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-25T05:43:00.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-25T05:43:00.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.847 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:00.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:00.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:00.972 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:00.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:01.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:43:01.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:01.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:01.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.370 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:01.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: pgmap v472: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:43:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:43:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[27010]: osdmap e294: 8 total, 8 up, 7 in 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: pgmap v472: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:01 smithi043 ceph-mon[37492]: osdmap e294: 8 total, 8 up, 7 in 2024-04-25T05:43:01.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:01.494 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:43:01.497 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:43:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:01.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: pgmap v472: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3751736593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:01 smithi060 ceph-mon[30898]: osdmap e294: 8 total, 8 up, 7 in 2024-04-25T05:43:01.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:01.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:01.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:01.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:01.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:01.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:01.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:01.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:02.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:02.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:43:02.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:02.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:02.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:02.258 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:02.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:02.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:02 smithi043 ceph-mon[27010]: 8.d scrub starts 2024-04-25T05:43:02.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:02 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:43:02.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:02 smithi043 ceph-mon[37492]: 8.d scrub starts 2024-04-25T05:43:02.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:02 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:43:02.385 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:02.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:02 smithi060 ceph-mon[30898]: 8.d scrub starts 2024-04-25T05:43:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:02 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:43:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:02.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:02.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:02.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:02.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:02.741 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-04-25T05:43:00.112126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105912883,"num_objects":1838,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":348,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":1518,"num_read_kb":1238,"num_write":132283,"num_write_kb":3972494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35866,"num_bytes_recovered":2329485593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128092,"ondisk_log_size":128092,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":519,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3835944,"kb_used_data":1146216,"kb_used_omap":10,"kb_used_meta":2689333,"kb_avail":652380120,"statfs":{"total":671965249536,"available":668037242880,"internally_reserved":0,"allocated":1173725184,"data_stored":1169679385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2753877099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-864419840,"num_objects":-13192,"num_object_clones":0,"num_object_copies":-39576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3315,"num_objects_unfound":0,"num_objects_dirty":-13192,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":-30780,"num_write_kb":-1490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12609,"num_bytes_recovered":-826179584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.276660"},"pg_stats":[{"pgid":"4.c","version":"74'10","reported_seq":195,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.948244+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:42:54.948244+0000","last_peered":"2024-04-25T05:42:54.948244+0000","last_clean":"2024-04-25T05:42:54.948244+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:42:54.948244+0000","last_undegraded":"2024-04-25T05:42:54.948244+0000","last_fullsized":"2024-04-25T05:42:54.948244+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.683068+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946151+0000","last_change":"2024-04-25T05:42:17.311411+0000","last_active":"2024-04-25T05:42:54.946151+0000","last_peered":"2024-04-25T05:42:54.946151+0000","last_clean":"2024-04-25T05:42:54.946151+0000","last_became_active":"2024-04-25T05:40:43.432950+0000","last_became_peered":"2024-04-25T05:40:43.432950+0000","last_unstale":"2024-04-25T05:42:54.946151+0000","last_undegraded":"2024-04-25T05:42:54.946151+0000","last_fullsized":"2024-04-25T05:42:54.946151+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.857445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"243'10","reported_seq":250,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946771+0000","last_change":"2024-04-25T05:42:21.213565+0000","last_active":"2024-04-25T05:42:54.946771+0000","last_peered":"2024-04-25T05:42:54.946771+0000","last_clean":"2024-04-25T05:42:54.946771+0000","last_became_active":"2024-04-25T05:40:55.955721+0000","last_became_peered":"2024-04-25T05:40:55.955721+0000","last_unstale":"2024-04-25T05:42:54.946771+0000","last_undegraded":"2024-04-25T05:42:54.946771+0000","last_fullsized":"2024-04-25T05:42:54.946771+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.386980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":159,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937893+0000","last_change":"2024-04-25T05:40:55.964673+0000","last_active":"2024-04-25T05:42:54.937893+0000","last_peered":"2024-04-25T05:42:54.937893+0000","last_clean":"2024-04-25T05:42:54.937893+0000","last_became_active":"2024-04-25T05:40:55.964302+0000","last_became_peered":"2024-04-25T05:40:55.964302+0000","last_unstale":"2024-04-25T05:42:54.937893+0000","last_undegraded":"2024-04-25T05:42:54.937893+0000","last_fullsized":"2024-04-25T05:42:54.937893+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:02.813078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:40:02.813078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":798,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945203+0000","last_change":"2024-04-25T05:42:20.218748+0000","last_active":"2024-04-25T05:42:54.945203+0000","last_peered":"2024-04-25T05:42:54.945203+0000","last_clean":"2024-04-25T05:42:54.945203+0000","last_became_active":"2024-04-25T05:40:58.930089+0000","last_became_peered":"2024-04-25T05:40:58.930089+0000","last_unstale":"2024-04-25T05:42:54.945203+0000","last_undegraded":"2024-04-25T05:42:54.945203+0000","last_fullsized":"2024-04-25T05:42:54.945203+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.920803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.0","version":"293'14634","reported_seq":14859,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.380514+0000","last_change":"2024-04-25T05:42:41.497103+0000","last_active":"2024-04-25T05:42:58.380514+0000","last_peered":"2024-04-25T05:42:58.380514+0000","last_clean":"2024-04-25T05:42:58.380514+0000","last_became_active":"2024-04-25T05:42:40.980240+0000","last_became_peered":"2024-04-25T05:42:40.980240+0000","last_unstale":"2024-04-25T05:42:58.380514+0000","last_undegraded":"2024-04-25T05:42:58.380514+0000","last_fullsized":"2024-04-25T05:42:58.380514+0000","mapping_epoch":198,"log_start":"259'9751","ondisk_log_start":"259'9751","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14036","last_scrub_stamp":"2024-04-25T05:42:41.496774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:41.496774+0000","objects_scrubbed":649,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:46.466354+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":605,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"277'25","reported_seq":197,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.938918+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:42:54.938918+0000","last_peered":"2024-04-25T05:42:54.938918+0000","last_clean":"2024-04-25T05:42:54.938918+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:42:54.938918+0000","last_undegraded":"2024-04-25T05:42:54.938918+0000","last_fullsized":"2024-04-25T05:42:54.938918+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:42.840863+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":206,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942783+0000","last_change":"2024-04-25T05:42:19.197580+0000","last_active":"2024-04-25T05:42:54.942783+0000","last_peered":"2024-04-25T05:42:54.942783+0000","last_clean":"2024-04-25T05:42:54.942783+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:42:54.942783+0000","last_undegraded":"2024-04-25T05:42:54.942783+0000","last_fullsized":"2024-04-25T05:42:54.942783+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.197467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.197467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:47.403991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.938031+0000","last_change":"2024-04-25T05:40:44.537276+0000","last_active":"2024-04-25T05:42:54.938031+0000","last_peered":"2024-04-25T05:42:54.938031+0000","last_clean":"2024-04-25T05:42:54.938031+0000","last_became_active":"2024-04-25T05:40:43.449807+0000","last_became_peered":"2024-04-25T05:40:43.449807+0000","last_unstale":"2024-04-25T05:42:54.938031+0000","last_undegraded":"2024-04-25T05:42:54.938031+0000","last_fullsized":"2024-04-25T05:42:54.938031+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":117,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946328+0000","last_change":"2024-04-25T05:40:55.974556+0000","last_active":"2024-04-25T05:42:54.946328+0000","last_peered":"2024-04-25T05:42:54.946328+0000","last_clean":"2024-04-25T05:42:54.946328+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:42:54.946328+0000","last_undegraded":"2024-04-25T05:42:54.946328+0000","last_fullsized":"2024-04-25T05:42:54.946328+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.019675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.019675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":671,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940592+0000","last_change":"2024-04-25T05:42:51.975109+0000","last_active":"2024-04-25T05:42:54.940592+0000","last_peered":"2024-04-25T05:42:54.940592+0000","last_clean":"2024-04-25T05:42:54.940592+0000","last_became_active":"2024-04-25T05:40:58.926534+0000","last_became_peered":"2024-04-25T05:40:58.926534+0000","last_unstale":"2024-04-25T05:42:54.940592+0000","last_undegraded":"2024-04-25T05:42:54.940592+0000","last_fullsized":"2024-04-25T05:42:54.940592+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:51.974990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.974990+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:11.972058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"293'15148","reported_seq":25946,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.415693+0000","last_change":"2024-04-25T05:42:42.967942+0000","last_active":"2024-04-25T05:42:58.415693+0000","last_peered":"2024-04-25T05:42:58.415693+0000","last_clean":"2024-04-25T05:42:58.415693+0000","last_became_active":"2024-04-25T05:42:40.979407+0000","last_became_peered":"2024-04-25T05:42:40.979407+0000","last_unstale":"2024-04-25T05:42:58.415693+0000","last_undegraded":"2024-04-25T05:42:58.415693+0000","last_fullsized":"2024-04-25T05:42:58.415693+0000","mapping_epoch":271,"log_start":"259'10155","ondisk_log_start":"259'10155","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14511","last_scrub_stamp":"2024-04-25T05:42:42.967718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.967718+0000","objects_scrubbed":683,"log_size":4993,"log_dups_size":0,"ondisk_log_size":4993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:46.194669+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"74'17","reported_seq":507,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943497+0000","last_change":"2024-04-25T05:42:53.501053+0000","last_active":"2024-04-25T05:42:54.943497+0000","last_peered":"2024-04-25T05:42:54.943497+0000","last_clean":"2024-04-25T05:42:54.943497+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:42:54.943497+0000","last_undegraded":"2024-04-25T05:42:54.943497+0000","last_fullsized":"2024-04-25T05:42:54.943497+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:42:53.500776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.500776+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:22.545533+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.869441+0000","last_change":"2024-04-25T05:42:51.237692+0000","last_active":"2024-04-25T05:42:54.869441+0000","last_peered":"2024-04-25T05:42:54.869441+0000","last_clean":"2024-04-25T05:42:54.869441+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:42:54.869441+0000","last_undegraded":"2024-04-25T05:42:54.869441+0000","last_fullsized":"2024-04-25T05:42:54.869441+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.237570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.237570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:13.272604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941928+0000","last_change":"2024-04-25T05:41:27.501946+0000","last_active":"2024-04-25T05:42:54.941928+0000","last_peered":"2024-04-25T05:42:54.941928+0000","last_clean":"2024-04-25T05:42:54.941928+0000","last_became_active":"2024-04-25T05:39:29.792743+0000","last_became_peered":"2024-04-25T05:39:29.792743+0000","last_unstale":"2024-04-25T05:42:54.941928+0000","last_undegraded":"2024-04-25T05:42:54.941928+0000","last_fullsized":"2024-04-25T05:42:54.941928+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.501828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.501828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.948012+0000","last_change":"2024-04-25T05:39:39.208977+0000","last_active":"2024-04-25T05:42:54.948012+0000","last_peered":"2024-04-25T05:42:54.948012+0000","last_clean":"2024-04-25T05:42:54.948012+0000","last_became_active":"2024-04-25T05:39:29.797111+0000","last_became_peered":"2024-04-25T05:39:29.797111+0000","last_unstale":"2024-04-25T05:42:54.948012+0000","last_undegraded":"2024-04-25T05:42:54.948012+0000","last_fullsized":"2024-04-25T05:42:54.948012+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.208840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.208840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"293'14649","reported_seq":29238,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.820268+0000","last_change":"2024-04-25T05:42:41.379855+0000","last_active":"2024-04-25T05:42:57.820268+0000","last_peered":"2024-04-25T05:42:57.820268+0000","last_clean":"2024-04-25T05:42:57.820268+0000","last_became_active":"2024-04-25T05:42:40.954397+0000","last_became_peered":"2024-04-25T05:42:40.954397+0000","last_unstale":"2024-04-25T05:42:57.820268+0000","last_undegraded":"2024-04-25T05:42:57.820268+0000","last_fullsized":"2024-04-25T05:42:57.820268+0000","mapping_epoch":255,"log_start":"259'9718","ondisk_log_start":"259'9718","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14023","last_scrub_stamp":"2024-04-25T05:42:41.379563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:41.379563+0000","objects_scrubbed":723,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:46.521980+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"210'22","reported_seq":518,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941611+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:42:54.941611+0000","last_peered":"2024-04-25T05:42:54.941611+0000","last_clean":"2024-04-25T05:42:54.941611+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:42:54.941611+0000","last_undegraded":"2024-04-25T05:42:54.941611+0000","last_fullsized":"2024-04-25T05:42:54.941611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:17.696734+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944804+0000","last_change":"2024-04-25T05:40:51.592334+0000","last_active":"2024-04-25T05:42:54.944804+0000","last_peered":"2024-04-25T05:42:54.944804+0000","last_clean":"2024-04-25T05:42:54.944804+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:42:54.944804+0000","last_undegraded":"2024-04-25T05:42:54.944804+0000","last_fullsized":"2024-04-25T05:42:54.944804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.592111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.592111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"233'11","reported_seq":230,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942164+0000","last_change":"2024-04-25T05:41:38.552749+0000","last_active":"2024-04-25T05:42:54.942164+0000","last_peered":"2024-04-25T05:42:54.942164+0000","last_clean":"2024-04-25T05:42:54.942164+0000","last_became_active":"2024-04-25T05:40:43.453268+0000","last_became_peered":"2024-04-25T05:40:43.453268+0000","last_unstale":"2024-04-25T05:42:54.942164+0000","last_undegraded":"2024-04-25T05:42:54.942164+0000","last_fullsized":"2024-04-25T05:42:54.942164+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941497+0000","last_change":"2024-04-25T05:40:50.460093+0000","last_active":"2024-04-25T05:42:54.941497+0000","last_peered":"2024-04-25T05:42:54.941497+0000","last_clean":"2024-04-25T05:42:54.941497+0000","last_became_active":"2024-04-25T05:40:43.454117+0000","last_became_peered":"2024-04-25T05:40:43.454117+0000","last_unstale":"2024-04-25T05:42:54.941497+0000","last_undegraded":"2024-04-25T05:42:54.941497+0000","last_fullsized":"2024-04-25T05:42:54.941497+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":676,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941963+0000","last_change":"2024-04-25T05:42:54.012448+0000","last_active":"2024-04-25T05:42:54.941963+0000","last_peered":"2024-04-25T05:42:54.941963+0000","last_clean":"2024-04-25T05:42:54.941963+0000","last_became_active":"2024-04-25T05:40:58.923753+0000","last_became_peered":"2024-04-25T05:40:58.923753+0000","last_unstale":"2024-04-25T05:42:54.941963+0000","last_undegraded":"2024-04-25T05:42:54.941963+0000","last_fullsized":"2024-04-25T05:42:54.941963+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:54.012215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.012215+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:18.178766+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"293'14583","reported_seq":15591,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.386167+0000","last_change":"2024-04-25T05:42:42.031537+0000","last_active":"2024-04-25T05:42:58.386167+0000","last_peered":"2024-04-25T05:42:58.386167+0000","last_clean":"2024-04-25T05:42:58.386167+0000","last_became_active":"2024-04-25T05:42:40.970867+0000","last_became_peered":"2024-04-25T05:42:40.970867+0000","last_unstale":"2024-04-25T05:42:58.386167+0000","last_undegraded":"2024-04-25T05:42:58.386167+0000","last_fullsized":"2024-04-25T05:42:58.386167+0000","mapping_epoch":208,"log_start":"259'9600","ondisk_log_start":"259'9600","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14030","last_scrub_stamp":"2024-04-25T05:42:42.031306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.031306+0000","objects_scrubbed":592,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:09.868252+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":554,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.869256+0000","last_change":"2024-04-25T05:42:19.974041+0000","last_active":"2024-04-25T05:42:54.869256+0000","last_peered":"2024-04-25T05:42:54.869256+0000","last_clean":"2024-04-25T05:42:54.869256+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:42:54.869256+0000","last_undegraded":"2024-04-25T05:42:54.869256+0000","last_fullsized":"2024-04-25T05:42:54.869256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:19.973911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.973911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.078951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'30","reported_seq":591,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947588+0000","last_change":"2024-04-25T05:40:48.519454+0000","last_active":"2024-04-25T05:42:54.947588+0000","last_peered":"2024-04-25T05:42:54.947588+0000","last_clean":"2024-04-25T05:42:54.947588+0000","last_became_active":"2024-04-25T05:40:43.440689+0000","last_became_peered":"2024-04-25T05:40:43.440689+0000","last_unstale":"2024-04-25T05:42:54.947588+0000","last_undegraded":"2024-04-25T05:42:54.947588+0000","last_fullsized":"2024-04-25T05:42:54.947588+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:40:48.519288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.519288+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947232+0000","last_change":"2024-04-25T05:40:43.516969+0000","last_active":"2024-04-25T05:42:54.947232+0000","last_peered":"2024-04-25T05:42:54.947232+0000","last_clean":"2024-04-25T05:42:54.947232+0000","last_became_active":"2024-04-25T05:40:43.440253+0000","last_became_peered":"2024-04-25T05:40:43.440253+0000","last_unstale":"2024-04-25T05:42:54.947232+0000","last_undegraded":"2024-04-25T05:42:54.947232+0000","last_fullsized":"2024-04-25T05:42:54.947232+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:43.516897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.516897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":159,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945055+0000","last_change":"2024-04-25T05:40:55.975104+0000","last_active":"2024-04-25T05:42:54.945055+0000","last_peered":"2024-04-25T05:42:54.945055+0000","last_clean":"2024-04-25T05:42:54.945055+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:42:54.945055+0000","last_undegraded":"2024-04-25T05:42:54.945055+0000","last_fullsized":"2024-04-25T05:42:54.945055+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.719427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.719427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":684,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870042+0000","last_change":"2024-04-25T05:42:54.215106+0000","last_active":"2024-04-25T05:42:54.870042+0000","last_peered":"2024-04-25T05:42:54.870042+0000","last_clean":"2024-04-25T05:42:54.870042+0000","last_became_active":"2024-04-25T05:40:58.920400+0000","last_became_peered":"2024-04-25T05:40:58.920400+0000","last_unstale":"2024-04-25T05:42:54.870042+0000","last_undegraded":"2024-04-25T05:42:54.870042+0000","last_fullsized":"2024-04-25T05:42:54.870042+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:54.214835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.214835+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:14.582869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"293'15494","reported_seq":25726,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.347912+0000","last_change":"2024-04-25T05:42:06.908338+0000","last_active":"2024-04-25T05:42:58.347912+0000","last_peered":"2024-04-25T05:42:58.347912+0000","last_clean":"2024-04-25T05:42:58.347912+0000","last_became_active":"2024-04-25T05:42:06.870659+0000","last_became_peered":"2024-04-25T05:42:06.870659+0000","last_unstale":"2024-04-25T05:42:58.347912+0000","last_undegraded":"2024-04-25T05:42:58.347912+0000","last_fullsized":"2024-04-25T05:42:58.347912+0000","mapping_epoch":270,"log_start":"266'11105","ondisk_log_start":"266'11105","created":198,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15494,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5005,"num_bytes_recovered":328007680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":591,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947690+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:42:54.947690+0000","last_peered":"2024-04-25T05:42:54.947690+0000","last_clean":"2024-04-25T05:42:54.947690+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:42:54.947690+0000","last_undegraded":"2024-04-25T05:42:54.947690+0000","last_fullsized":"2024-04-25T05:42:54.947690+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:17.034593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'20","reported_seq":213,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941516+0000","last_change":"2024-04-25T05:40:48.391768+0000","last_active":"2024-04-25T05:42:54.941516+0000","last_peered":"2024-04-25T05:42:54.941516+0000","last_clean":"2024-04-25T05:42:54.941516+0000","last_became_active":"2024-04-25T05:40:09.733866+0000","last_became_peered":"2024-04-25T05:40:09.733866+0000","last_unstale":"2024-04-25T05:42:54.941516+0000","last_undegraded":"2024-04-25T05:42:54.941516+0000","last_fullsized":"2024-04-25T05:42:54.941516+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:40:48.391383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.391383+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":136,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944371+0000","last_change":"2024-04-25T05:40:43.463116+0000","last_active":"2024-04-25T05:42:54.944371+0000","last_peered":"2024-04-25T05:42:54.944371+0000","last_clean":"2024-04-25T05:42:54.944371+0000","last_became_active":"2024-04-25T05:40:43.443728+0000","last_became_peered":"2024-04-25T05:40:43.443728+0000","last_unstale":"2024-04-25T05:42:54.944371+0000","last_undegraded":"2024-04-25T05:42:54.944371+0000","last_fullsized":"2024-04-25T05:42:54.944371+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:39:34.740970+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:39:34.740970+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":115,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945936+0000","last_change":"2024-04-25T05:40:55.981504+0000","last_active":"2024-04-25T05:42:54.945936+0000","last_peered":"2024-04-25T05:42:54.945936+0000","last_clean":"2024-04-25T05:42:54.945936+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:42:54.945936+0000","last_undegraded":"2024-04-25T05:42:54.945936+0000","last_fullsized":"2024-04-25T05:42:54.945936+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.781719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.781719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":658,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947840+0000","last_change":"2024-04-25T05:42:22.285373+0000","last_active":"2024-04-25T05:42:54.947840+0000","last_peered":"2024-04-25T05:42:54.947840+0000","last_clean":"2024-04-25T05:42:54.947840+0000","last_became_active":"2024-04-25T05:38:34.761584+0000","last_became_peered":"2024-04-25T05:38:34.761584+0000","last_unstale":"2024-04-25T05:42:54.947840+0000","last_undegraded":"2024-04-25T05:42:54.947840+0000","last_fullsized":"2024-04-25T05:42:54.947840+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:29.371504+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"293'15453","reported_seq":29733,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.896277+0000","last_change":"2024-04-25T05:41:28.231423+0000","last_active":"2024-04-25T05:42:57.896277+0000","last_peered":"2024-04-25T05:42:57.896277+0000","last_clean":"2024-04-25T05:42:57.896277+0000","last_became_active":"2024-04-25T05:41:28.213582+0000","last_became_peered":"2024-04-25T05:41:28.213582+0000","last_unstale":"2024-04-25T05:42:57.896277+0000","last_undegraded":"2024-04-25T05:42:57.896277+0000","last_fullsized":"2024-04-25T05:42:57.896277+0000","mapping_epoch":255,"log_start":"269'11565","ondisk_log_start":"269'11565","created":198,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602200,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15453,"num_write_kb":498177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7030,"num_bytes_recovered":460652568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"233'12","reported_seq":163,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940522+0000","last_change":"2024-04-25T05:40:43.475139+0000","last_active":"2024-04-25T05:42:54.940522+0000","last_peered":"2024-04-25T05:42:54.940522+0000","last_clean":"2024-04-25T05:42:54.940522+0000","last_became_active":"2024-04-25T05:40:43.445833+0000","last_became_peered":"2024-04-25T05:40:43.445833+0000","last_unstale":"2024-04-25T05:42:54.940522+0000","last_undegraded":"2024-04-25T05:42:54.940522+0000","last_fullsized":"2024-04-25T05:42:54.940522+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:40:06.822122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.822122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"195'22","reported_seq":137,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941598+0000","last_change":"2024-04-25T05:42:18.092205+0000","last_active":"2024-04-25T05:42:54.941598+0000","last_peered":"2024-04-25T05:42:54.941598+0000","last_clean":"2024-04-25T05:42:54.941598+0000","last_became_active":"2024-04-25T05:40:55.964721+0000","last_became_peered":"2024-04-25T05:40:55.964721+0000","last_unstale":"2024-04-25T05:42:54.941598+0000","last_undegraded":"2024-04-25T05:42:54.941598+0000","last_fullsized":"2024-04-25T05:42:54.941598+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:30.000606+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":489,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942595+0000","last_change":"2024-04-25T05:40:55.975334+0000","last_active":"2024-04-25T05:42:54.942595+0000","last_peered":"2024-04-25T05:42:54.942595+0000","last_clean":"2024-04-25T05:42:54.942595+0000","last_became_active":"2024-04-25T05:40:55.974871+0000","last_became_peered":"2024-04-25T05:40:55.974871+0000","last_unstale":"2024-04-25T05:42:54.942595+0000","last_undegraded":"2024-04-25T05:42:54.942595+0000","last_fullsized":"2024-04-25T05:42:54.942595+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:38.369946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:38.369946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937690+0000","last_change":"2024-04-25T05:42:14.731871+0000","last_active":"2024-04-25T05:42:54.937690+0000","last_peered":"2024-04-25T05:42:54.937690+0000","last_clean":"2024-04-25T05:42:54.937690+0000","last_became_active":"2024-04-25T05:40:55.970110+0000","last_became_peered":"2024-04-25T05:40:55.970110+0000","last_unstale":"2024-04-25T05:42:54.937690+0000","last_undegraded":"2024-04-25T05:42:54.937690+0000","last_fullsized":"2024-04-25T05:42:54.937690+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.632461+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":661,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943690+0000","last_change":"2024-04-25T05:41:31.093059+0000","last_active":"2024-04-25T05:42:54.943690+0000","last_peered":"2024-04-25T05:42:54.943690+0000","last_clean":"2024-04-25T05:42:54.943690+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:42:54.943690+0000","last_undegraded":"2024-04-25T05:42:54.943690+0000","last_fullsized":"2024-04-25T05:42:54.943690+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:31.092923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.092923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.069935+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"293'15249","reported_seq":24392,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.329364+0000","last_change":"2024-04-25T05:42:12.209425+0000","last_active":"2024-04-25T05:42:58.329364+0000","last_peered":"2024-04-25T05:42:58.329364+0000","last_clean":"2024-04-25T05:42:58.329364+0000","last_became_active":"2024-04-25T05:42:12.177948+0000","last_became_peered":"2024-04-25T05:42:12.177948+0000","last_unstale":"2024-04-25T05:42:58.329364+0000","last_undegraded":"2024-04-25T05:42:58.329364+0000","last_fullsized":"2024-04-25T05:42:58.329364+0000","mapping_epoch":274,"log_start":"265'10812","ondisk_log_start":"265'10812","created":198,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":999,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15249,"num_write_kb":490240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4450,"num_bytes_recovered":291635200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":212,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942809+0000","last_change":"2024-04-25T05:42:14.225323+0000","last_active":"2024-04-25T05:42:54.942809+0000","last_peered":"2024-04-25T05:42:54.942809+0000","last_clean":"2024-04-25T05:42:54.942809+0000","last_became_active":"2024-04-25T05:40:43.429667+0000","last_became_peered":"2024-04-25T05:40:43.429667+0000","last_unstale":"2024-04-25T05:42:54.942809+0000","last_undegraded":"2024-04-25T05:42:54.942809+0000","last_fullsized":"2024-04-25T05:42:54.942809+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.225203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.225203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.751205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'25","reported_seq":160,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940635+0000","last_change":"2024-04-25T05:42:22.674343+0000","last_active":"2024-04-25T05:42:54.940635+0000","last_peered":"2024-04-25T05:42:54.940635+0000","last_clean":"2024-04-25T05:42:54.940635+0000","last_became_active":"2024-04-25T05:40:43.446526+0000","last_became_peered":"2024-04-25T05:40:43.446526+0000","last_unstale":"2024-04-25T05:42:54.940635+0000","last_undegraded":"2024-04-25T05:42:54.940635+0000","last_fullsized":"2024-04-25T05:42:54.940635+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:34.867005+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937484+0000","last_change":"2024-04-25T05:40:55.977907+0000","last_active":"2024-04-25T05:42:54.937484+0000","last_peered":"2024-04-25T05:42:54.937484+0000","last_clean":"2024-04-25T05:42:54.937484+0000","last_became_active":"2024-04-25T05:40:55.977574+0000","last_became_peered":"2024-04-25T05:40:55.977574+0000","last_unstale":"2024-04-25T05:42:54.937484+0000","last_undegraded":"2024-04-25T05:42:54.937484+0000","last_fullsized":"2024-04-25T05:42:54.937484+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:16.651620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:16.651620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940993+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:42:54.940993+0000","last_peered":"2024-04-25T05:42:54.940993+0000","last_clean":"2024-04-25T05:42:54.940993+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:42:54.940993+0000","last_undegraded":"2024-04-25T05:42:54.940993+0000","last_fullsized":"2024-04-25T05:42:54.940993+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.278820+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":678,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.500780+0000","last_change":"2024-04-25T05:42:57.500822+0000","last_active":"2024-04-25T05:42:57.500780+0000","last_peered":"2024-04-25T05:42:57.500780+0000","last_clean":"2024-04-25T05:42:57.500780+0000","last_became_active":"2024-04-25T05:40:45.845535+0000","last_became_peered":"2024-04-25T05:40:45.845535+0000","last_unstale":"2024-04-25T05:42:57.500780+0000","last_undegraded":"2024-04-25T05:42:57.500780+0000","last_fullsized":"2024-04-25T05:42:57.500780+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.500683+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:17.122506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"293'15748","reported_seq":28977,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.424566+0000","last_change":"2024-04-25T05:42:42.418953+0000","last_active":"2024-04-25T05:42:58.424566+0000","last_peered":"2024-04-25T05:42:58.424566+0000","last_clean":"2024-04-25T05:42:58.424566+0000","last_became_active":"2024-04-25T05:41:47.997545+0000","last_became_peered":"2024-04-25T05:41:47.997545+0000","last_unstale":"2024-04-25T05:42:58.424566+0000","last_undegraded":"2024-04-25T05:42:58.424566+0000","last_fullsized":"2024-04-25T05:42:58.424566+0000","mapping_epoch":262,"log_start":"272'11924","ondisk_log_start":"272'11924","created":198,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14524","last_scrub_stamp":"2024-04-25T05:42:42.418846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.418846+0000","objects_scrubbed":1306,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:54.242485+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15748,"num_write_kb":506560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6494,"num_bytes_recovered":425590784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942031+0000","last_change":"2024-04-25T05:40:55.418588+0000","last_active":"2024-04-25T05:42:54.942031+0000","last_peered":"2024-04-25T05:42:54.942031+0000","last_clean":"2024-04-25T05:42:54.942031+0000","last_became_active":"2024-04-25T05:40:43.454300+0000","last_became_peered":"2024-04-25T05:40:43.454300+0000","last_unstale":"2024-04-25T05:42:54.942031+0000","last_undegraded":"2024-04-25T05:42:54.942031+0000","last_fullsized":"2024-04-25T05:42:54.942031+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:55.418499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.418499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941562+0000","last_change":"2024-04-25T05:40:12.370708+0000","last_active":"2024-04-25T05:42:54.941562+0000","last_peered":"2024-04-25T05:42:54.941562+0000","last_clean":"2024-04-25T05:42:54.941562+0000","last_became_active":"2024-04-25T05:37:56.218253+0000","last_became_peered":"2024-04-25T05:37:56.218253+0000","last_unstale":"2024-04-25T05:42:54.941562+0000","last_undegraded":"2024-04-25T05:42:54.941562+0000","last_fullsized":"2024-04-25T05:42:54.941562+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1054,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:56.680334+0000","last_change":"2024-04-25T05:41:36.018017+0000","last_active":"2024-04-25T05:42:56.680334+0000","last_peered":"2024-04-25T05:42:56.680334+0000","last_clean":"2024-04-25T05:42:56.680334+0000","last_became_active":"2024-04-25T05:40:43.431415+0000","last_became_peered":"2024-04-25T05:40:43.431415+0000","last_unstale":"2024-04-25T05:42:56.680334+0000","last_undegraded":"2024-04-25T05:42:56.680334+0000","last_fullsized":"2024-04-25T05:42:56.680334+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:41:36.017857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.017857+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":526,"num_read_kb":521,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":132,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943114+0000","last_change":"2024-04-25T05:40:43.435255+0000","last_active":"2024-04-25T05:42:54.943114+0000","last_peered":"2024-04-25T05:42:54.943114+0000","last_clean":"2024-04-25T05:42:54.943114+0000","last_became_active":"2024-04-25T05:40:43.434912+0000","last_became_peered":"2024-04-25T05:40:43.434912+0000","last_unstale":"2024-04-25T05:42:54.943114+0000","last_undegraded":"2024-04-25T05:42:54.943114+0000","last_fullsized":"2024-04-25T05:42:54.943114+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.785355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.785355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937374+0000","last_change":"2024-04-25T05:40:47.469910+0000","last_active":"2024-04-25T05:42:54.937374+0000","last_peered":"2024-04-25T05:42:54.937374+0000","last_clean":"2024-04-25T05:42:54.937374+0000","last_became_active":"2024-04-25T05:34:04.726207+0000","last_became_peered":"2024-04-25T05:34:04.726207+0000","last_unstale":"2024-04-25T05:42:54.937374+0000","last_undegraded":"2024-04-25T05:42:54.937374+0000","last_fullsized":"2024-04-25T05:42:54.937374+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":673,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941383+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:42:54.941383+0000","last_peered":"2024-04-25T05:42:54.941383+0000","last_clean":"2024-04-25T05:42:54.941383+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:42:54.941383+0000","last_undegraded":"2024-04-25T05:42:54.941383+0000","last_fullsized":"2024-04-25T05:42:54.941383+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.357654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.a","version":"293'15582","reported_seq":26168,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.403461+0000","last_change":"2024-04-25T05:42:07.903446+0000","last_active":"2024-04-25T05:42:58.403461+0000","last_peered":"2024-04-25T05:42:58.403461+0000","last_clean":"2024-04-25T05:42:58.403461+0000","last_became_active":"2024-04-25T05:42:07.871807+0000","last_became_peered":"2024-04-25T05:42:07.871807+0000","last_unstale":"2024-04-25T05:42:58.403461+0000","last_undegraded":"2024-04-25T05:42:58.403461+0000","last_fullsized":"2024-04-25T05:42:58.403461+0000","mapping_epoch":271,"log_start":"267'11342","ondisk_log_start":"267'11342","created":198,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'7666","last_scrub_stamp":"2024-04-25T05:40:52.458924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.458924+0000","objects_scrubbed":7586,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15582,"num_write_kb":500928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5139,"num_bytes_recovered":336789504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":489,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946600+0000","last_change":"2024-04-25T05:41:20.125777+0000","last_active":"2024-04-25T05:42:54.946600+0000","last_peered":"2024-04-25T05:42:54.946600+0000","last_clean":"2024-04-25T05:42:54.946600+0000","last_became_active":"2024-04-25T05:40:55.953691+0000","last_became_peered":"2024-04-25T05:40:55.953691+0000","last_unstale":"2024-04-25T05:42:54.946600+0000","last_undegraded":"2024-04-25T05:42:54.946600+0000","last_fullsized":"2024-04-25T05:42:54.946600+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:20.125501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:41:20.125501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"208'19","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943437+0000","last_change":"2024-04-25T05:40:40.957968+0000","last_active":"2024-04-25T05:42:54.943437+0000","last_peered":"2024-04-25T05:42:54.943437+0000","last_clean":"2024-04-25T05:42:54.943437+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:42:54.943437+0000","last_undegraded":"2024-04-25T05:42:54.943437+0000","last_fullsized":"2024-04-25T05:42:54.943437+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:40:40.957665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.957665+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944792+0000","last_change":"2024-04-25T05:42:08.356121+0000","last_active":"2024-04-25T05:42:54.944792+0000","last_peered":"2024-04-25T05:42:54.944792+0000","last_clean":"2024-04-25T05:42:54.944792+0000","last_became_active":"2024-04-25T05:39:29.797080+0000","last_became_peered":"2024-04-25T05:39:29.797080+0000","last_unstale":"2024-04-25T05:42:54.944792+0000","last_undegraded":"2024-04-25T05:42:54.944792+0000","last_fullsized":"2024-04-25T05:42:54.944792+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:10.802129+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":799,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945242+0000","last_change":"2024-04-25T05:42:18.235935+0000","last_active":"2024-04-25T05:42:54.945242+0000","last_peered":"2024-04-25T05:42:54.945242+0000","last_clean":"2024-04-25T05:42:54.945242+0000","last_became_active":"2024-04-25T05:40:45.847131+0000","last_became_peered":"2024-04-25T05:40:45.847131+0000","last_unstale":"2024-04-25T05:42:54.945242+0000","last_undegraded":"2024-04-25T05:42:54.945242+0000","last_fullsized":"2024-04-25T05:42:54.945242+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.800850+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941622+0000","last_change":"2024-04-25T05:42:15.011589+0000","last_active":"2024-04-25T05:42:54.941622+0000","last_peered":"2024-04-25T05:42:54.941622+0000","last_clean":"2024-04-25T05:42:54.941622+0000","last_became_active":"2024-04-25T05:34:04.722436+0000","last_became_peered":"2024-04-25T05:34:04.722436+0000","last_unstale":"2024-04-25T05:42:54.941622+0000","last_undegraded":"2024-04-25T05:42:54.941622+0000","last_fullsized":"2024-04-25T05:42:54.941622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:16.656094+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"293'14653","reported_seq":29853,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.784306+0000","last_change":"2024-04-25T05:42:41.499673+0000","last_active":"2024-04-25T05:42:57.784306+0000","last_peered":"2024-04-25T05:42:57.784306+0000","last_clean":"2024-04-25T05:42:57.784306+0000","last_became_active":"2024-04-25T05:42:40.982917+0000","last_became_peered":"2024-04-25T05:42:40.982917+0000","last_unstale":"2024-04-25T05:42:57.784306+0000","last_undegraded":"2024-04-25T05:42:57.784306+0000","last_fullsized":"2024-04-25T05:42:57.784306+0000","mapping_epoch":258,"log_start":"256'9435","ondisk_log_start":"256'9435","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14083","last_scrub_stamp":"2024-04-25T05:42:41.499343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:41.499343+0000","objects_scrubbed":643,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:45.671837+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":116,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942267+0000","last_change":"2024-04-25T05:40:55.910414+0000","last_active":"2024-04-25T05:42:54.942267+0000","last_peered":"2024-04-25T05:42:54.942267+0000","last_clean":"2024-04-25T05:42:54.942267+0000","last_became_active":"2024-04-25T05:40:55.910074+0000","last_became_peered":"2024-04-25T05:40:55.910074+0000","last_unstale":"2024-04-25T05:42:54.942267+0000","last_undegraded":"2024-04-25T05:42:54.942267+0000","last_fullsized":"2024-04-25T05:42:54.942267+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:40.980299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.980299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"98'25","reported_seq":564,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941478+0000","last_change":"2024-04-25T05:40:43.477675+0000","last_active":"2024-04-25T05:42:54.941478+0000","last_peered":"2024-04-25T05:42:54.941478+0000","last_clean":"2024-04-25T05:42:54.941478+0000","last_became_active":"2024-04-25T05:40:43.450842+0000","last_became_peered":"2024-04-25T05:40:43.450842+0000","last_unstale":"2024-04-25T05:42:54.941478+0000","last_undegraded":"2024-04-25T05:42:54.941478+0000","last_fullsized":"2024-04-25T05:42:54.941478+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.592911+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:55.459122+0000","last_change":"2024-04-25T05:42:55.459163+0000","last_active":"2024-04-25T05:42:55.459122+0000","last_peered":"2024-04-25T05:42:55.459122+0000","last_clean":"2024-04-25T05:42:55.459122+0000","last_became_active":"2024-04-25T05:40:55.990929+0000","last_became_peered":"2024-04-25T05:40:55.990929+0000","last_unstale":"2024-04-25T05:42:55.459122+0000","last_undegraded":"2024-04-25T05:42:55.459122+0000","last_fullsized":"2024-04-25T05:42:55.459122+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:08.316392+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.938206+0000","last_change":"2024-04-25T05:40:55.966682+0000","last_active":"2024-04-25T05:42:54.938206+0000","last_peered":"2024-04-25T05:42:54.938206+0000","last_clean":"2024-04-25T05:42:54.938206+0000","last_became_active":"2024-04-25T05:40:55.966367+0000","last_became_peered":"2024-04-25T05:40:55.966367+0000","last_unstale":"2024-04-25T05:42:54.938206+0000","last_undegraded":"2024-04-25T05:42:54.938206+0000","last_fullsized":"2024-04-25T05:42:54.938206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:42.474808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:42.474808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":491,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946940+0000","last_change":"2024-04-25T05:40:52.973425+0000","last_active":"2024-04-25T05:42:54.946940+0000","last_peered":"2024-04-25T05:42:54.946940+0000","last_clean":"2024-04-25T05:42:54.946940+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:42:54.946940+0000","last_undegraded":"2024-04-25T05:42:54.946940+0000","last_fullsized":"2024-04-25T05:42:54.946940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:52.973302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:52.973302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":798,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946089+0000","last_change":"2024-04-25T05:41:44.999875+0000","last_active":"2024-04-25T05:42:54.946089+0000","last_peered":"2024-04-25T05:42:54.946089+0000","last_clean":"2024-04-25T05:42:54.946089+0000","last_became_active":"2024-04-25T05:40:45.848289+0000","last_became_peered":"2024-04-25T05:40:45.848289+0000","last_unstale":"2024-04-25T05:42:54.946089+0000","last_undegraded":"2024-04-25T05:42:54.946089+0000","last_fullsized":"2024-04-25T05:42:54.946089+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.306297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"293'15156","reported_seq":15380,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.385032+0000","last_change":"2024-04-25T05:42:43.415262+0000","last_active":"2024-04-25T05:42:58.385032+0000","last_peered":"2024-04-25T05:42:58.385032+0000","last_clean":"2024-04-25T05:42:58.385032+0000","last_became_active":"2024-04-25T05:39:19.228541+0000","last_became_peered":"2024-04-25T05:39:19.228541+0000","last_unstale":"2024-04-25T05:42:58.385032+0000","last_undegraded":"2024-04-25T05:42:58.385032+0000","last_fullsized":"2024-04-25T05:42:58.385032+0000","mapping_epoch":198,"log_start":"270'11319","ondisk_log_start":"270'11319","created":198,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14018","last_scrub_stamp":"2024-04-25T05:42:43.415159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:43.415159+0000","objects_scrubbed":1206,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:06.670640+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15156,"num_write_kb":487168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":132,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940724+0000","last_change":"2024-04-25T05:40:43.442114+0000","last_active":"2024-04-25T05:42:54.940724+0000","last_peered":"2024-04-25T05:42:54.940724+0000","last_clean":"2024-04-25T05:42:54.940724+0000","last_became_active":"2024-04-25T05:40:43.441816+0000","last_became_peered":"2024-04-25T05:40:43.441816+0000","last_unstale":"2024-04-25T05:42:54.940724+0000","last_undegraded":"2024-04-25T05:42:54.940724+0000","last_fullsized":"2024-04-25T05:42:54.940724+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"243'46","reported_seq":696,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.994006+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:42:54.994006+0000","last_peered":"2024-04-25T05:42:54.994006+0000","last_clean":"2024-04-25T05:42:54.994006+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:42:54.994006+0000","last_undegraded":"2024-04-25T05:42:54.994006+0000","last_fullsized":"2024-04-25T05:42:54.994006+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:20.547718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940845+0000","last_change":"2024-04-25T05:40:43.435305+0000","last_active":"2024-04-25T05:42:54.940845+0000","last_peered":"2024-04-25T05:42:54.940845+0000","last_clean":"2024-04-25T05:42:54.940845+0000","last_became_active":"2024-04-25T05:40:43.434803+0000","last_became_peered":"2024-04-25T05:40:43.434803+0000","last_unstale":"2024-04-25T05:42:54.940845+0000","last_undegraded":"2024-04-25T05:42:54.940845+0000","last_fullsized":"2024-04-25T05:42:54.940845+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:12.849716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.849716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":531,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942177+0000","last_change":"2024-04-25T05:42:44.047955+0000","last_active":"2024-04-25T05:42:54.942177+0000","last_peered":"2024-04-25T05:42:54.942177+0000","last_clean":"2024-04-25T05:42:54.942177+0000","last_became_active":"2024-04-25T05:39:29.794144+0000","last_became_peered":"2024-04-25T05:39:29.794144+0000","last_unstale":"2024-04-25T05:42:54.942177+0000","last_undegraded":"2024-04-25T05:42:54.942177+0000","last_fullsized":"2024-04-25T05:42:54.942177+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:54.982808+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":471,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946272+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:42:54.946272+0000","last_peered":"2024-04-25T05:42:54.946272+0000","last_clean":"2024-04-25T05:42:54.946272+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:42:54.946272+0000","last_undegraded":"2024-04-25T05:42:54.946272+0000","last_fullsized":"2024-04-25T05:42:54.946272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:43.757268+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":676,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941356+0000","last_change":"2024-04-25T05:40:58.922339+0000","last_active":"2024-04-25T05:42:54.941356+0000","last_peered":"2024-04-25T05:42:54.941356+0000","last_clean":"2024-04-25T05:42:54.941356+0000","last_became_active":"2024-04-25T05:40:58.921770+0000","last_became_peered":"2024-04-25T05:40:58.921770+0000","last_unstale":"2024-04-25T05:42:54.941356+0000","last_undegraded":"2024-04-25T05:42:54.941356+0000","last_fullsized":"2024-04-25T05:42:54.941356+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:40:07.411609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:40:07.411609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"293'14527","reported_seq":14748,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.870911+0000","last_change":"2024-04-25T05:42:41.301731+0000","last_active":"2024-04-25T05:42:57.870911+0000","last_peered":"2024-04-25T05:42:57.870911+0000","last_clean":"2024-04-25T05:42:57.870911+0000","last_became_active":"2024-04-25T05:42:40.981618+0000","last_became_peered":"2024-04-25T05:42:40.981618+0000","last_unstale":"2024-04-25T05:42:57.870911+0000","last_undegraded":"2024-04-25T05:42:57.870911+0000","last_fullsized":"2024-04-25T05:42:57.870911+0000","mapping_epoch":198,"log_start":"259'9594","ondisk_log_start":"259'9594","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'13947","last_scrub_stamp":"2024-04-25T05:42:41.301385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:41.301385+0000","objects_scrubbed":639,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.968650+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":588,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"211'20","reported_seq":162,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940979+0000","last_change":"2024-04-25T05:42:34.594637+0000","last_active":"2024-04-25T05:42:54.940979+0000","last_peered":"2024-04-25T05:42:54.940979+0000","last_clean":"2024-04-25T05:42:54.940979+0000","last_became_active":"2024-04-25T05:42:34.576621+0000","last_became_peered":"2024-04-25T05:42:34.576621+0000","last_unstale":"2024-04-25T05:42:54.940979+0000","last_undegraded":"2024-04-25T05:42:54.940979+0000","last_fullsized":"2024-04-25T05:42:54.940979+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.766685+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":502,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940928+0000","last_change":"2024-04-25T05:41:28.560703+0000","last_active":"2024-04-25T05:42:54.940928+0000","last_peered":"2024-04-25T05:42:54.940928+0000","last_clean":"2024-04-25T05:42:54.940928+0000","last_became_active":"2024-04-25T05:39:29.794429+0000","last_became_peered":"2024-04-25T05:39:29.794429+0000","last_unstale":"2024-04-25T05:42:54.940928+0000","last_undegraded":"2024-04-25T05:42:54.940928+0000","last_fullsized":"2024-04-25T05:42:54.940928+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":124,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870729+0000","last_change":"2024-04-25T05:42:05.954387+0000","last_active":"2024-04-25T05:42:54.870729+0000","last_peered":"2024-04-25T05:42:54.870729+0000","last_clean":"2024-04-25T05:42:54.870729+0000","last_became_active":"2024-04-25T05:40:55.981082+0000","last_became_peered":"2024-04-25T05:40:55.981082+0000","last_unstale":"2024-04-25T05:42:54.870729+0000","last_undegraded":"2024-04-25T05:42:54.870729+0000","last_fullsized":"2024-04-25T05:42:54.870729+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:05.954259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:05.954259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:33.753300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":152,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943023+0000","last_change":"2024-04-25T05:42:54.545923+0000","last_active":"2024-04-25T05:42:54.943023+0000","last_peered":"2024-04-25T05:42:54.943023+0000","last_clean":"2024-04-25T05:42:54.943023+0000","last_became_active":"2024-04-25T05:42:34.577940+0000","last_became_peered":"2024-04-25T05:42:34.577940+0000","last_unstale":"2024-04-25T05:42:54.943023+0000","last_undegraded":"2024-04-25T05:42:54.943023+0000","last_fullsized":"2024-04-25T05:42:54.943023+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:18.163256+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"293'14711","reported_seq":15904,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.412355+0000","last_change":"2024-04-25T05:42:44.039891+0000","last_active":"2024-04-25T05:42:58.412355+0000","last_peered":"2024-04-25T05:42:58.412355+0000","last_clean":"2024-04-25T05:42:58.412355+0000","last_became_active":"2024-04-25T05:42:40.970978+0000","last_became_peered":"2024-04-25T05:42:40.970978+0000","last_unstale":"2024-04-25T05:42:58.412355+0000","last_undegraded":"2024-04-25T05:42:58.412355+0000","last_fullsized":"2024-04-25T05:42:58.412355+0000","mapping_epoch":209,"log_start":"259'9767","ondisk_log_start":"259'9767","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14156","last_scrub_stamp":"2024-04-25T05:42:44.039488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.039488+0000","objects_scrubbed":590,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:02.990362+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"74'25","reported_seq":228,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942597+0000","last_change":"2024-04-25T05:42:15.186498+0000","last_active":"2024-04-25T05:42:54.942597+0000","last_peered":"2024-04-25T05:42:54.942597+0000","last_clean":"2024-04-25T05:42:54.942597+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:42:54.942597+0000","last_undegraded":"2024-04-25T05:42:54.942597+0000","last_fullsized":"2024-04-25T05:42:54.942597+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:15.186321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.186321+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:26.644738+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":517,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947019+0000","last_change":"2024-04-25T05:40:08.880333+0000","last_active":"2024-04-25T05:42:54.947019+0000","last_peered":"2024-04-25T05:42:54.947019+0000","last_clean":"2024-04-25T05:42:54.947019+0000","last_became_active":"2024-04-25T05:39:29.798539+0000","last_became_peered":"2024-04-25T05:39:29.798539+0000","last_unstale":"2024-04-25T05:42:54.947019+0000","last_undegraded":"2024-04-25T05:42:54.947019+0000","last_fullsized":"2024-04-25T05:42:54.947019+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:08.880207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:08.880207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940151+0000","last_change":"2024-04-25T05:40:55.953740+0000","last_active":"2024-04-25T05:42:54.940151+0000","last_peered":"2024-04-25T05:42:54.940151+0000","last_clean":"2024-04-25T05:42:54.940151+0000","last_became_active":"2024-04-25T05:40:55.953404+0000","last_became_peered":"2024-04-25T05:40:55.953404+0000","last_unstale":"2024-04-25T05:42:54.940151+0000","last_undegraded":"2024-04-25T05:42:54.940151+0000","last_fullsized":"2024-04-25T05:42:54.940151+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.796291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.796291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":485,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945041+0000","last_change":"2024-04-25T05:42:53.130326+0000","last_active":"2024-04-25T05:42:54.945041+0000","last_peered":"2024-04-25T05:42:54.945041+0000","last_clean":"2024-04-25T05:42:54.945041+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:42:54.945041+0000","last_undegraded":"2024-04-25T05:42:54.945041+0000","last_fullsized":"2024-04-25T05:42:54.945041+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.130201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.130201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:57.334358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"293'14831","reported_seq":29658,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.288677+0000","last_change":"2024-04-25T05:42:41.811553+0000","last_active":"2024-04-25T05:42:58.288677+0000","last_peered":"2024-04-25T05:42:58.288677+0000","last_clean":"2024-04-25T05:42:58.288677+0000","last_became_active":"2024-04-25T05:42:40.976550+0000","last_became_peered":"2024-04-25T05:42:40.976550+0000","last_unstale":"2024-04-25T05:42:58.288677+0000","last_undegraded":"2024-04-25T05:42:58.288677+0000","last_fullsized":"2024-04-25T05:42:58.288677+0000","mapping_epoch":260,"log_start":"253'9357","ondisk_log_start":"253'9357","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'14196","last_scrub_stamp":"2024-04-25T05:42:41.811284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:41.811284+0000","objects_scrubbed":682,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:57.252312+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":642,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":480,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937523+0000","last_change":"2024-04-25T05:40:55.988511+0000","last_active":"2024-04-25T05:42:54.937523+0000","last_peered":"2024-04-25T05:42:54.937523+0000","last_clean":"2024-04-25T05:42:54.937523+0000","last_became_active":"2024-04-25T05:40:55.988228+0000","last_became_peered":"2024-04-25T05:40:55.988228+0000","last_unstale":"2024-04-25T05:42:54.937523+0000","last_undegraded":"2024-04-25T05:42:54.937523+0000","last_fullsized":"2024-04-25T05:42:54.937523+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:39.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:39:39.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'24","reported_seq":131,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.938893+0000","last_change":"2024-04-25T05:40:56.145089+0000","last_active":"2024-04-25T05:42:54.938893+0000","last_peered":"2024-04-25T05:42:54.938893+0000","last_clean":"2024-04-25T05:42:54.938893+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:42:54.938893+0000","last_undegraded":"2024-04-25T05:42:54.938893+0000","last_fullsized":"2024-04-25T05:42:54.938893+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'24","last_scrub_stamp":"2024-04-25T05:40:37.943948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.943948+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946849+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:42:54.946849+0000","last_peered":"2024-04-25T05:42:54.946849+0000","last_clean":"2024-04-25T05:42:54.946849+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:42:54.946849+0000","last_undegraded":"2024-04-25T05:42:54.946849+0000","last_fullsized":"2024-04-25T05:42:54.946849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:08.869970+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946234+0000","last_change":"2024-04-25T05:40:55.989744+0000","last_active":"2024-04-25T05:42:54.946234+0000","last_peered":"2024-04-25T05:42:54.946234+0000","last_clean":"2024-04-25T05:42:54.946234+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:42:54.946234+0000","last_undegraded":"2024-04-25T05:42:54.946234+0000","last_fullsized":"2024-04-25T05:42:54.946234+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:11.579078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:11.579078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"293'14941","reported_seq":15422,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.883255+0000","last_change":"2024-04-25T05:42:42.398172+0000","last_active":"2024-04-25T05:42:57.883255+0000","last_peered":"2024-04-25T05:42:57.883255+0000","last_clean":"2024-04-25T05:42:57.883255+0000","last_became_active":"2024-04-25T05:42:40.997779+0000","last_became_peered":"2024-04-25T05:42:40.997779+0000","last_unstale":"2024-04-25T05:42:57.883255+0000","last_undegraded":"2024-04-25T05:42:57.883255+0000","last_fullsized":"2024-04-25T05:42:57.883255+0000","mapping_epoch":209,"log_start":"256'9614","ondisk_log_start":"256'9614","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14356","last_scrub_stamp":"2024-04-25T05:42:42.398035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.398035+0000","objects_scrubbed":665,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:45.214278+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":631,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945740+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:42:54.945740+0000","last_peered":"2024-04-25T05:42:54.945740+0000","last_clean":"2024-04-25T05:42:54.945740+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:42:54.945740+0000","last_undegraded":"2024-04-25T05:42:54.945740+0000","last_fullsized":"2024-04-25T05:42:54.945740+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:09.582353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'27","reported_seq":588,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.961231+0000","last_change":"2024-04-25T05:42:54.961292+0000","last_active":"2024-04-25T05:42:54.961231+0000","last_peered":"2024-04-25T05:42:54.961231+0000","last_clean":"2024-04-25T05:42:54.961231+0000","last_became_active":"2024-04-25T05:39:29.799149+0000","last_became_peered":"2024-04-25T05:39:29.799149+0000","last_unstale":"2024-04-25T05:42:54.961231+0000","last_undegraded":"2024-04-25T05:42:54.961231+0000","last_fullsized":"2024-04-25T05:42:54.961231+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:13.640904+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":526,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946488+0000","last_change":"2024-04-25T05:41:25.239411+0000","last_active":"2024-04-25T05:42:54.946488+0000","last_peered":"2024-04-25T05:42:54.946488+0000","last_clean":"2024-04-25T05:42:54.946488+0000","last_became_active":"2024-04-25T05:40:43.429608+0000","last_became_peered":"2024-04-25T05:40:43.429608+0000","last_unstale":"2024-04-25T05:42:54.946488+0000","last_undegraded":"2024-04-25T05:42:54.946488+0000","last_fullsized":"2024-04-25T05:42:54.946488+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:41:25.239292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:25.239292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":503,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942389+0000","last_change":"2024-04-25T05:40:51.460612+0000","last_active":"2024-04-25T05:42:54.942389+0000","last_peered":"2024-04-25T05:42:54.942389+0000","last_clean":"2024-04-25T05:42:54.942389+0000","last_became_active":"2024-04-25T05:40:43.453025+0000","last_became_peered":"2024-04-25T05:40:43.453025+0000","last_unstale":"2024-04-25T05:42:54.942389+0000","last_undegraded":"2024-04-25T05:42:54.942389+0000","last_fullsized":"2024-04-25T05:42:54.942389+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:51.460330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:51.460330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.4","version":"293'15030","reported_seq":29718,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.883542+0000","last_change":"2024-04-25T05:42:42.114601+0000","last_active":"2024-04-25T05:42:57.883542+0000","last_peered":"2024-04-25T05:42:57.883542+0000","last_clean":"2024-04-25T05:42:57.883542+0000","last_became_active":"2024-04-25T05:42:40.974819+0000","last_became_peered":"2024-04-25T05:42:40.974819+0000","last_unstale":"2024-04-25T05:42:57.883542+0000","last_undegraded":"2024-04-25T05:42:57.883542+0000","last_fullsized":"2024-04-25T05:42:57.883542+0000","mapping_epoch":255,"log_start":"259'10037","ondisk_log_start":"259'10037","created":198,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'14428","last_scrub_stamp":"2024-04-25T05:42:42.114462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.114462+0000","objects_scrubbed":660,"log_size":4993,"log_dups_size":0,"ondisk_log_size":4993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:07.724613+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"74'12","reported_seq":499,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947453+0000","last_change":"2024-04-25T05:40:41.507574+0000","last_active":"2024-04-25T05:42:54.947453+0000","last_peered":"2024-04-25T05:42:54.947453+0000","last_clean":"2024-04-25T05:42:54.947453+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:42:54.947453+0000","last_undegraded":"2024-04-25T05:42:54.947453+0000","last_fullsized":"2024-04-25T05:42:54.947453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:41.507400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.507400+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":204,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941741+0000","last_change":"2024-04-25T05:40:43.431845+0000","last_active":"2024-04-25T05:42:54.941741+0000","last_peered":"2024-04-25T05:42:54.941741+0000","last_clean":"2024-04-25T05:42:54.941741+0000","last_became_active":"2024-04-25T05:40:43.431502+0000","last_became_peered":"2024-04-25T05:40:43.431502+0000","last_unstale":"2024-04-25T05:42:54.941741+0000","last_undegraded":"2024-04-25T05:42:54.941741+0000","last_fullsized":"2024-04-25T05:42:54.941741+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.918723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.918723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941900+0000","last_change":"2024-04-25T05:40:46.378509+0000","last_active":"2024-04-25T05:42:54.941900+0000","last_peered":"2024-04-25T05:42:54.941900+0000","last_clean":"2024-04-25T05:42:54.941900+0000","last_became_active":"2024-04-25T05:34:02.699775+0000","last_became_peered":"2024-04-25T05:34:02.699775+0000","last_unstale":"2024-04-25T05:42:54.941900+0000","last_undegraded":"2024-04-25T05:42:54.941900+0000","last_fullsized":"2024-04-25T05:42:54.941900+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:46.378394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:40:46.378394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946482+0000","last_change":"2024-04-25T05:41:27.185161+0000","last_active":"2024-04-25T05:42:54.946482+0000","last_peered":"2024-04-25T05:42:54.946482+0000","last_clean":"2024-04-25T05:42:54.946482+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:42:54.946482+0000","last_undegraded":"2024-04-25T05:42:54.946482+0000","last_fullsized":"2024-04-25T05:42:54.946482+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:27.185044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:27.185044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'21","reported_seq":153,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944169+0000","last_change":"2024-04-25T05:42:21.659494+0000","last_active":"2024-04-25T05:42:54.944169+0000","last_peered":"2024-04-25T05:42:54.944169+0000","last_clean":"2024-04-25T05:42:54.944169+0000","last_became_active":"2024-04-25T05:40:43.441558+0000","last_became_peered":"2024-04-25T05:40:43.441558+0000","last_unstale":"2024-04-25T05:42:54.944169+0000","last_undegraded":"2024-04-25T05:42:54.944169+0000","last_fullsized":"2024-04-25T05:42:54.944169+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:36.744922+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945519+0000","last_change":"2024-04-25T05:40:50.906267+0000","last_active":"2024-04-25T05:42:54.945519+0000","last_peered":"2024-04-25T05:42:54.945519+0000","last_clean":"2024-04-25T05:42:54.945519+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:42:54.945519+0000","last_undegraded":"2024-04-25T05:42:54.945519+0000","last_fullsized":"2024-04-25T05:42:54.945519+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.905458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.905458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":466,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947352+0000","last_change":"2024-04-25T05:40:38.559014+0000","last_active":"2024-04-25T05:42:54.947352+0000","last_peered":"2024-04-25T05:42:54.947352+0000","last_clean":"2024-04-25T05:42:54.947352+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:42:54.947352+0000","last_undegraded":"2024-04-25T05:42:54.947352+0000","last_fullsized":"2024-04-25T05:42:54.947352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:38.558895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:38.558895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":125,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870273+0000","last_change":"2024-04-25T05:42:10.907844+0000","last_active":"2024-04-25T05:42:54.870273+0000","last_peered":"2024-04-25T05:42:54.870273+0000","last_clean":"2024-04-25T05:42:54.870273+0000","last_became_active":"2024-04-25T05:40:55.953924+0000","last_became_peered":"2024-04-25T05:40:55.953924+0000","last_unstale":"2024-04-25T05:42:54.870273+0000","last_undegraded":"2024-04-25T05:42:54.870273+0000","last_fullsized":"2024-04-25T05:42:54.870273+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:10.907707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.907707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:31.135983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"293'14668","reported_seq":30955,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.880029+0000","last_change":"2024-04-25T05:42:50.110441+0000","last_active":"2024-04-25T05:42:57.880029+0000","last_peered":"2024-04-25T05:42:57.880029+0000","last_clean":"2024-04-25T05:42:57.880029+0000","last_became_active":"2024-04-25T05:42:49.037441+0000","last_became_peered":"2024-04-25T05:42:49.037441+0000","last_unstale":"2024-04-25T05:42:57.880029+0000","last_undegraded":"2024-04-25T05:42:57.880029+0000","last_fullsized":"2024-04-25T05:42:57.880029+0000","mapping_epoch":290,"log_start":"256'9435","ondisk_log_start":"256'9435","created":283,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"292'14308","last_scrub_stamp":"2024-04-25T05:42:50.110080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.110080+0000","objects_scrubbed":423,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:12.665455+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":362,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"100'20","reported_seq":519,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942755+0000","last_change":"2024-04-25T05:42:08.829436+0000","last_active":"2024-04-25T05:42:54.942755+0000","last_peered":"2024-04-25T05:42:54.942755+0000","last_clean":"2024-04-25T05:42:54.942755+0000","last_became_active":"2024-04-25T05:34:01.001760+0000","last_became_peered":"2024-04-25T05:34:01.001760+0000","last_unstale":"2024-04-25T05:42:54.942755+0000","last_undegraded":"2024-04-25T05:42:54.942755+0000","last_fullsized":"2024-04-25T05:42:54.942755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:22.650865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":131,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941342+0000","last_change":"2024-04-25T05:40:43.434277+0000","last_active":"2024-04-25T05:42:54.941342+0000","last_peered":"2024-04-25T05:42:54.941342+0000","last_clean":"2024-04-25T05:42:54.941342+0000","last_became_active":"2024-04-25T05:40:43.433933+0000","last_became_peered":"2024-04-25T05:40:43.433933+0000","last_unstale":"2024-04-25T05:42:54.941342+0000","last_undegraded":"2024-04-25T05:42:54.941342+0000","last_fullsized":"2024-04-25T05:42:54.941342+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:44.112568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:39:44.112568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":592,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.948347+0000","last_change":"2024-04-25T05:40:56.012131+0000","last_active":"2024-04-25T05:42:54.948347+0000","last_peered":"2024-04-25T05:42:54.948347+0000","last_clean":"2024-04-25T05:42:54.948347+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:42:54.948347+0000","last_undegraded":"2024-04-25T05:42:54.948347+0000","last_fullsized":"2024-04-25T05:42:54.948347+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:40:37.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.529721+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":118,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941499+0000","last_change":"2024-04-25T05:40:55.954634+0000","last_active":"2024-04-25T05:42:54.941499+0000","last_peered":"2024-04-25T05:42:54.941499+0000","last_clean":"2024-04-25T05:42:54.941499+0000","last_became_active":"2024-04-25T05:40:55.954181+0000","last_became_peered":"2024-04-25T05:40:55.954181+0000","last_unstale":"2024-04-25T05:42:54.941499+0000","last_undegraded":"2024-04-25T05:42:54.941499+0000","last_fullsized":"2024-04-25T05:42:54.941499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:09.913324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:09.913324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.18","version":"293'14575","reported_seq":14793,"reported_epoch":293,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:42:58.379579+0000","last_change":"2024-04-25T05:42:44.100643+0000","last_active":"2024-04-25T05:42:58.379579+0000","last_peered":"2024-04-25T05:42:58.379579+0000","last_clean":"2024-04-25T05:42:41.839133+0000","last_became_active":"2024-04-25T05:42:44.064543+0000","last_became_peered":"2024-04-25T05:42:44.064543+0000","last_unstale":"2024-04-25T05:42:58.379579+0000","last_undegraded":"2024-04-25T05:42:58.379579+0000","last_fullsized":"2024-04-25T05:42:58.379579+0000","mapping_epoch":286,"log_start":"259'9594","ondisk_log_start":"259'9594","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7610,"num_write_kb":244224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":22}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'22","reported_seq":517,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.948061+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:42:54.948061+0000","last_peered":"2024-04-25T05:42:54.948061+0000","last_clean":"2024-04-25T05:42:54.948061+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:42:54.948061+0000","last_undegraded":"2024-04-25T05:42:54.948061+0000","last_fullsized":"2024-04-25T05:42:54.948061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:15.927150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946053+0000","last_change":"2024-04-25T05:41:37.012236+0000","last_active":"2024-04-25T05:42:54.946053+0000","last_peered":"2024-04-25T05:42:54.946053+0000","last_clean":"2024-04-25T05:42:54.946053+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:42:54.946053+0000","last_undegraded":"2024-04-25T05:42:54.946053+0000","last_fullsized":"2024-04-25T05:42:54.946053+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:37.012159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:37.012159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"233'10","reported_seq":619,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941892+0000","last_change":"2024-04-25T05:41:45.481735+0000","last_active":"2024-04-25T05:42:54.941892+0000","last_peered":"2024-04-25T05:42:54.941892+0000","last_clean":"2024-04-25T05:42:54.941892+0000","last_became_active":"2024-04-25T05:40:43.453246+0000","last_became_peered":"2024-04-25T05:40:43.453246+0000","last_unstale":"2024-04-25T05:42:54.941892+0000","last_undegraded":"2024-04-25T05:42:54.941892+0000","last_fullsized":"2024-04-25T05:42:54.941892+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":131,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944467+0000","last_change":"2024-04-25T05:40:43.443261+0000","last_active":"2024-04-25T05:42:54.944467+0000","last_peered":"2024-04-25T05:42:54.944467+0000","last_clean":"2024-04-25T05:42:54.944467+0000","last_became_active":"2024-04-25T05:40:43.439460+0000","last_became_peered":"2024-04-25T05:40:43.439460+0000","last_unstale":"2024-04-25T05:42:54.944467+0000","last_undegraded":"2024-04-25T05:42:54.944467+0000","last_fullsized":"2024-04-25T05:42:54.944467+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:43.132052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:39:43.132052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"291'16","reported_seq":589,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870961+0000","last_change":"2024-04-25T05:42:49.238116+0000","last_active":"2024-04-25T05:42:54.870961+0000","last_peered":"2024-04-25T05:42:54.870961+0000","last_clean":"2024-04-25T05:42:54.870961+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:42:54.870961+0000","last_undegraded":"2024-04-25T05:42:54.870961+0000","last_fullsized":"2024-04-25T05:42:54.870961+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'16","last_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.238022+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:10.314687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":170,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946414+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:42:54.946414+0000","last_peered":"2024-04-25T05:42:54.946414+0000","last_clean":"2024-04-25T05:42:54.946414+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:42:54.946414+0000","last_undegraded":"2024-04-25T05:42:54.946414+0000","last_fullsized":"2024-04-25T05:42:54.946414+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.152477+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":474,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941670+0000","last_change":"2024-04-25T05:40:06.644197+0000","last_active":"2024-04-25T05:42:54.941670+0000","last_peered":"2024-04-25T05:42:54.941670+0000","last_clean":"2024-04-25T05:42:54.941670+0000","last_became_active":"2024-04-25T05:34:02.698944+0000","last_became_peered":"2024-04-25T05:34:02.698944+0000","last_unstale":"2024-04-25T05:42:54.941670+0000","last_undegraded":"2024-04-25T05:42:54.941670+0000","last_fullsized":"2024-04-25T05:42:54.941670+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:06.644083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:06.644083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942956+0000","last_change":"2024-04-25T05:40:43.434425+0000","last_active":"2024-04-25T05:42:54.942956+0000","last_peered":"2024-04-25T05:42:54.942956+0000","last_clean":"2024-04-25T05:42:54.942956+0000","last_became_active":"2024-04-25T05:40:43.434051+0000","last_became_peered":"2024-04-25T05:40:43.434051+0000","last_unstale":"2024-04-25T05:42:54.942956+0000","last_undegraded":"2024-04-25T05:42:54.942956+0000","last_fullsized":"2024-04-25T05:42:54.942956+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:04.446776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:04.446776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"257'13","reported_seq":404,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941807+0000","last_change":"2024-04-25T05:41:30.532539+0000","last_active":"2024-04-25T05:42:54.941807+0000","last_peered":"2024-04-25T05:42:54.941807+0000","last_clean":"2024-04-25T05:42:54.941807+0000","last_became_active":"2024-04-25T05:40:55.954865+0000","last_became_peered":"2024-04-25T05:40:55.954865+0000","last_unstale":"2024-04-25T05:42:54.941807+0000","last_undegraded":"2024-04-25T05:42:54.941807+0000","last_fullsized":"2024-04-25T05:42:54.941807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:59.262752+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":498,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:56.099304+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:42:56.099304+0000","last_peered":"2024-04-25T05:42:56.099304+0000","last_clean":"2024-04-25T05:42:56.099304+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:42:56.099304+0000","last_undegraded":"2024-04-25T05:42:56.099304+0000","last_fullsized":"2024-04-25T05:42:56.099304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:01.666941+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":478,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946106+0000","last_change":"2024-04-25T05:40:43.433578+0000","last_active":"2024-04-25T05:42:54.946106+0000","last_peered":"2024-04-25T05:42:54.946106+0000","last_clean":"2024-04-25T05:42:54.946106+0000","last_became_active":"2024-04-25T05:40:43.433240+0000","last_became_peered":"2024-04-25T05:40:43.433240+0000","last_unstale":"2024-04-25T05:42:54.946106+0000","last_undegraded":"2024-04-25T05:42:54.946106+0000","last_fullsized":"2024-04-25T05:42:54.946106+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:14.526768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:14.526768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":139,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937431+0000","last_change":"2024-04-25T05:42:51.491420+0000","last_active":"2024-04-25T05:42:54.937431+0000","last_peered":"2024-04-25T05:42:54.937431+0000","last_clean":"2024-04-25T05:42:54.937431+0000","last_became_active":"2024-04-25T05:40:43.446278+0000","last_became_peered":"2024-04-25T05:40:43.446278+0000","last_unstale":"2024-04-25T05:42:54.937431+0000","last_undegraded":"2024-04-25T05:42:54.937431+0000","last_fullsized":"2024-04-25T05:42:54.937431+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:57.815632+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'17","reported_seq":549,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943234+0000","last_change":"2024-04-25T05:40:49.558496+0000","last_active":"2024-04-25T05:42:54.943234+0000","last_peered":"2024-04-25T05:42:54.943234+0000","last_clean":"2024-04-25T05:42:54.943234+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:42:54.943234+0000","last_undegraded":"2024-04-25T05:42:54.943234+0000","last_fullsized":"2024-04-25T05:42:54.943234+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:49.558354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.558354+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942028+0000","last_change":"2024-04-25T05:39:41.392240+0000","last_active":"2024-04-25T05:42:54.942028+0000","last_peered":"2024-04-25T05:42:54.942028+0000","last_clean":"2024-04-25T05:42:54.942028+0000","last_became_active":"2024-04-25T05:33:58.679361+0000","last_became_peered":"2024-04-25T05:33:58.679361+0000","last_unstale":"2024-04-25T05:42:54.942028+0000","last_undegraded":"2024-04-25T05:42:54.942028+0000","last_fullsized":"2024-04-25T05:42:54.942028+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:41.392167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:41.392167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945515+0000","last_change":"2024-04-25T05:40:53.928517+0000","last_active":"2024-04-25T05:42:54.945515+0000","last_peered":"2024-04-25T05:42:54.945515+0000","last_clean":"2024-04-25T05:42:54.945515+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:42:54.945515+0000","last_undegraded":"2024-04-25T05:42:54.945515+0000","last_fullsized":"2024-04-25T05:42:54.945515+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:53.928372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:53.928372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946340+0000","last_change":"2024-04-25T05:42:20.379921+0000","last_active":"2024-04-25T05:42:54.946340+0000","last_peered":"2024-04-25T05:42:54.946340+0000","last_clean":"2024-04-25T05:42:54.946340+0000","last_became_active":"2024-04-25T05:40:43.435420+0000","last_became_peered":"2024-04-25T05:40:43.435420+0000","last_unstale":"2024-04-25T05:42:54.946340+0000","last_undegraded":"2024-04-25T05:42:54.946340+0000","last_fullsized":"2024-04-25T05:42:54.946340+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:37.364344+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945695+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:42:54.945695+0000","last_peered":"2024-04-25T05:42:54.945695+0000","last_clean":"2024-04-25T05:42:54.945695+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:42:54.945695+0000","last_undegraded":"2024-04-25T05:42:54.945695+0000","last_fullsized":"2024-04-25T05:42:54.945695+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":131,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942677+0000","last_change":"2024-04-25T05:40:43.446312+0000","last_active":"2024-04-25T05:42:54.942677+0000","last_peered":"2024-04-25T05:42:54.942677+0000","last_clean":"2024-04-25T05:42:54.942677+0000","last_became_active":"2024-04-25T05:40:43.445367+0000","last_became_peered":"2024-04-25T05:40:43.445367+0000","last_unstale":"2024-04-25T05:42:54.942677+0000","last_undegraded":"2024-04-25T05:42:54.942677+0000","last_fullsized":"2024-04-25T05:42:54.942677+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:37.412344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:39:37.412344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":198,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942703+0000","last_change":"2024-04-25T05:41:40.152573+0000","last_active":"2024-04-25T05:42:54.942703+0000","last_peered":"2024-04-25T05:42:54.942703+0000","last_clean":"2024-04-25T05:42:54.942703+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:42:54.942703+0000","last_undegraded":"2024-04-25T05:42:54.942703+0000","last_fullsized":"2024-04-25T05:42:54.942703+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:40.152411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:40.152411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:02.701355+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":489,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946193+0000","last_change":"2024-04-25T05:41:39.031952+0000","last_active":"2024-04-25T05:42:54.946193+0000","last_peered":"2024-04-25T05:42:54.946193+0000","last_clean":"2024-04-25T05:42:54.946193+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:42:54.946193+0000","last_undegraded":"2024-04-25T05:42:54.946193+0000","last_fullsized":"2024-04-25T05:42:54.946193+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:41:39.031784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.031784+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:42:56.769610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"292'23","reported_seq":226,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943776+0000","last_change":"2024-04-25T05:42:52.491775+0000","last_active":"2024-04-25T05:42:54.943776+0000","last_peered":"2024-04-25T05:42:54.943776+0000","last_clean":"2024-04-25T05:42:54.943776+0000","last_became_active":"2024-04-25T05:39:29.802368+0000","last_became_peered":"2024-04-25T05:39:29.802368+0000","last_unstale":"2024-04-25T05:42:54.943776+0000","last_undegraded":"2024-04-25T05:42:54.943776+0000","last_fullsized":"2024-04-25T05:42:54.943776+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.491617+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:59.010372+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945602+0000","last_change":"2024-04-25T05:40:50.562106+0000","last_active":"2024-04-25T05:42:54.945602+0000","last_peered":"2024-04-25T05:42:54.945602+0000","last_clean":"2024-04-25T05:42:54.945602+0000","last_became_active":"2024-04-25T05:40:43.440363+0000","last_became_peered":"2024-04-25T05:40:43.440363+0000","last_unstale":"2024-04-25T05:42:54.945602+0000","last_undegraded":"2024-04-25T05:42:54.945602+0000","last_fullsized":"2024-04-25T05:42:54.945602+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.560317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.560317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943608+0000","last_change":"2024-04-25T05:42:48.418293+0000","last_active":"2024-04-25T05:42:54.943608+0000","last_peered":"2024-04-25T05:42:54.943608+0000","last_clean":"2024-04-25T05:42:54.943608+0000","last_became_active":"2024-04-25T05:42:34.575586+0000","last_became_peered":"2024-04-25T05:42:34.575586+0000","last_unstale":"2024-04-25T05:42:54.943608+0000","last_undegraded":"2024-04-25T05:42:54.943608+0000","last_fullsized":"2024-04-25T05:42:54.943608+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.418180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.418180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:12.177812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":470,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.938815+0000","last_change":"2024-04-25T05:40:15.703133+0000","last_active":"2024-04-25T05:42:54.938815+0000","last_peered":"2024-04-25T05:42:54.938815+0000","last_clean":"2024-04-25T05:42:54.938815+0000","last_became_active":"2024-04-25T05:34:04.721801+0000","last_became_peered":"2024-04-25T05:34:04.721801+0000","last_unstale":"2024-04-25T05:42:54.938815+0000","last_undegraded":"2024-04-25T05:42:54.938815+0000","last_fullsized":"2024-04-25T05:42:54.938815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.702999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.702999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"293'14624","reported_seq":16796,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:58.401908+0000","last_change":"2024-04-25T05:42:49.512303+0000","last_active":"2024-04-25T05:42:58.401908+0000","last_peered":"2024-04-25T05:42:58.401908+0000","last_clean":"2024-04-25T05:42:58.401908+0000","last_became_active":"2024-04-25T05:42:49.037390+0000","last_became_peered":"2024-04-25T05:42:49.037390+0000","last_unstale":"2024-04-25T05:42:58.401908+0000","last_undegraded":"2024-04-25T05:42:58.401908+0000","last_fullsized":"2024-04-25T05:42:58.401908+0000","mapping_epoch":290,"log_start":"259'9600","ondisk_log_start":"259'9600","created":283,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"291'14197","last_scrub_stamp":"2024-04-25T05:42:49.512042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.512042+0000","objects_scrubbed":477,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:02.473679+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":427,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"191'22","reported_seq":146,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944063+0000","last_change":"2024-04-25T05:40:43.482944+0000","last_active":"2024-04-25T05:42:54.944063+0000","last_peered":"2024-04-25T05:42:54.944063+0000","last_clean":"2024-04-25T05:42:54.944063+0000","last_became_active":"2024-04-25T05:40:43.442720+0000","last_became_peered":"2024-04-25T05:40:43.442720+0000","last_unstale":"2024-04-25T05:42:54.944063+0000","last_undegraded":"2024-04-25T05:42:54.944063+0000","last_fullsized":"2024-04-25T05:42:54.944063+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:39:02.184563+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":17,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":491,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940892+0000","last_change":"2024-04-25T05:40:40.500687+0000","last_active":"2024-04-25T05:42:54.940892+0000","last_peered":"2024-04-25T05:42:54.940892+0000","last_clean":"2024-04-25T05:42:54.940892+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:42:54.940892+0000","last_undegraded":"2024-04-25T05:42:54.940892+0000","last_fullsized":"2024-04-25T05:42:54.940892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:40.500174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.500174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":251,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.943389+0000","last_change":"2024-04-25T05:42:24.210053+0000","last_active":"2024-04-25T05:42:54.943389+0000","last_peered":"2024-04-25T05:42:54.943389+0000","last_clean":"2024-04-25T05:42:54.943389+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:42:54.943389+0000","last_undegraded":"2024-04-25T05:42:54.943389+0000","last_fullsized":"2024-04-25T05:42:54.943389+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:42:24.209923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.209923+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:51.982376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946597+0000","last_change":"2024-04-25T05:42:54.100194+0000","last_active":"2024-04-25T05:42:54.946597+0000","last_peered":"2024-04-25T05:42:54.946597+0000","last_clean":"2024-04-25T05:42:54.946597+0000","last_became_active":"2024-04-25T05:40:55.974471+0000","last_became_peered":"2024-04-25T05:40:55.974471+0000","last_unstale":"2024-04-25T05:42:54.946597+0000","last_undegraded":"2024-04-25T05:42:54.946597+0000","last_fullsized":"2024-04-25T05:42:54.946597+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:19.105304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"293'14644","reported_seq":29222,"reported_epoch":293,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:42:58.335631+0000","last_change":"2024-04-25T05:42:44.100514+0000","last_active":"2024-04-25T05:42:58.335631+0000","last_peered":"2024-04-25T05:42:58.335631+0000","last_clean":"2024-04-25T05:42:41.919409+0000","last_became_active":"2024-04-25T05:42:44.068767+0000","last_became_peered":"2024-04-25T05:42:44.068767+0000","last_unstale":"2024-04-25T05:42:58.335631+0000","last_undegraded":"2024-04-25T05:42:58.335631+0000","last_fullsized":"2024-04-25T05:42:58.335631+0000","mapping_epoch":286,"log_start":"259'9718","ondisk_log_start":"259'9718","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":5,"last_scrub":"231'6914","last_scrub_stamp":"2024-04-25T05:40:40.935916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.935916+0000","objects_scrubbed":6834,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521984,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":138,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":246752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3567,"num_bytes_recovered":233734144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[0,4,5],"avail_no_missing":["0","3","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":69}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"74'18","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946203+0000","last_change":"2024-04-25T05:40:15.536952+0000","last_active":"2024-04-25T05:42:54.946203+0000","last_peered":"2024-04-25T05:42:54.946203+0000","last_clean":"2024-04-25T05:42:54.946203+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:42:54.946203+0000","last_undegraded":"2024-04-25T05:42:54.946203+0000","last_fullsized":"2024-04-25T05:42:54.946203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:40:15.536802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.536802+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":495,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870561+0000","last_change":"2024-04-25T05:42:11.885667+0000","last_active":"2024-04-25T05:42:54.870561+0000","last_peered":"2024-04-25T05:42:54.870561+0000","last_clean":"2024-04-25T05:42:54.870561+0000","last_became_active":"2024-04-25T05:40:55.964565+0000","last_became_peered":"2024-04-25T05:40:55.964565+0000","last_unstale":"2024-04-25T05:42:54.870561+0000","last_undegraded":"2024-04-25T05:42:54.870561+0000","last_fullsized":"2024-04-25T05:42:54.870561+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:11.885545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:11.885545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:20.830666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":168,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.948173+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:42:54.948173+0000","last_peered":"2024-04-25T05:42:54.948173+0000","last_clean":"2024-04-25T05:42:54.948173+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:42:54.948173+0000","last_undegraded":"2024-04-25T05:42:54.948173+0000","last_fullsized":"2024-04-25T05:42:54.948173+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:08.416189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947357+0000","last_change":"2024-04-25T05:42:52.519578+0000","last_active":"2024-04-25T05:42:54.947357+0000","last_peered":"2024-04-25T05:42:54.947357+0000","last_clean":"2024-04-25T05:42:54.947357+0000","last_became_active":"2024-04-25T05:39:29.797893+0000","last_became_peered":"2024-04-25T05:39:29.797893+0000","last_unstale":"2024-04-25T05:42:54.947357+0000","last_undegraded":"2024-04-25T05:42:54.947357+0000","last_fullsized":"2024-04-25T05:42:54.947357+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:54.534315+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"293'15085","reported_seq":27524,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.825713+0000","last_change":"2024-04-25T05:42:51.110705+0000","last_active":"2024-04-25T05:42:57.825713+0000","last_peered":"2024-04-25T05:42:57.825713+0000","last_clean":"2024-04-25T05:42:57.825713+0000","last_became_active":"2024-04-25T05:42:50.044513+0000","last_became_peered":"2024-04-25T05:42:50.044513+0000","last_unstale":"2024-04-25T05:42:57.825713+0000","last_undegraded":"2024-04-25T05:42:57.825713+0000","last_fullsized":"2024-04-25T05:42:57.825713+0000","mapping_epoch":291,"log_start":"259'10155","ondisk_log_start":"259'10155","created":283,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"292'14765","last_scrub_stamp":"2024-04-25T05:42:51.107563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.107563+0000","objects_scrubbed":386,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:00.626489+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":323,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"150'22","reported_seq":349,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937420+0000","last_change":"2024-04-25T05:42:52.924801+0000","last_active":"2024-04-25T05:42:54.937420+0000","last_peered":"2024-04-25T05:42:54.937420+0000","last_clean":"2024-04-25T05:42:54.937420+0000","last_became_active":"2024-04-25T05:40:55.991291+0000","last_became_peered":"2024-04-25T05:40:55.991291+0000","last_unstale":"2024-04-25T05:42:54.937420+0000","last_undegraded":"2024-04-25T05:42:54.937420+0000","last_fullsized":"2024-04-25T05:42:54.937420+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:05.903480+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":131,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940752+0000","last_change":"2024-04-25T05:40:43.435522+0000","last_active":"2024-04-25T05:42:54.940752+0000","last_peered":"2024-04-25T05:42:54.940752+0000","last_clean":"2024-04-25T05:42:54.940752+0000","last_became_active":"2024-04-25T05:40:43.434830+0000","last_became_peered":"2024-04-25T05:40:43.434830+0000","last_unstale":"2024-04-25T05:42:54.940752+0000","last_undegraded":"2024-04-25T05:42:54.940752+0000","last_fullsized":"2024-04-25T05:42:54.940752+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941445+0000","last_change":"2024-04-25T05:40:43.451998+0000","last_active":"2024-04-25T05:42:54.941445+0000","last_peered":"2024-04-25T05:42:54.941445+0000","last_clean":"2024-04-25T05:42:54.941445+0000","last_became_active":"2024-04-25T05:40:43.449712+0000","last_became_peered":"2024-04-25T05:40:43.449712+0000","last_unstale":"2024-04-25T05:42:54.941445+0000","last_undegraded":"2024-04-25T05:42:54.941445+0000","last_fullsized":"2024-04-25T05:42:54.941445+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:41.506653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:41.506653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.870442+0000","last_change":"2024-04-25T05:42:24.062360+0000","last_active":"2024-04-25T05:42:54.870442+0000","last_peered":"2024-04-25T05:42:54.870442+0000","last_clean":"2024-04-25T05:42:54.870442+0000","last_became_active":"2024-04-25T05:39:29.788222+0000","last_became_peered":"2024-04-25T05:39:29.788222+0000","last_unstale":"2024-04-25T05:42:54.870442+0000","last_undegraded":"2024-04-25T05:42:54.870442+0000","last_fullsized":"2024-04-25T05:42:54.870442+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.062132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:49.623846+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.10","version":"293'14595","reported_seq":16133,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.856335+0000","last_change":"2024-04-25T05:42:50.230209+0000","last_active":"2024-04-25T05:42:57.856335+0000","last_peered":"2024-04-25T05:42:57.856335+0000","last_clean":"2024-04-25T05:42:57.856335+0000","last_became_active":"2024-04-25T05:42:50.039367+0000","last_became_peered":"2024-04-25T05:42:50.039367+0000","last_unstale":"2024-04-25T05:42:57.856335+0000","last_undegraded":"2024-04-25T05:42:57.856335+0000","last_fullsized":"2024-04-25T05:42:57.856335+0000","mapping_epoch":291,"log_start":"259'9751","ondisk_log_start":"259'9751","created":283,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"292'14249","last_scrub_stamp":"2024-04-25T05:42:50.230055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.230055+0000","objects_scrubbed":421,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:14.755872+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'11","reported_seq":216,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945987+0000","last_change":"2024-04-25T05:42:13.374107+0000","last_active":"2024-04-25T05:42:54.945987+0000","last_peered":"2024-04-25T05:42:54.945987+0000","last_clean":"2024-04-25T05:42:54.945987+0000","last_became_active":"2024-04-25T05:40:55.984807+0000","last_became_peered":"2024-04-25T05:40:55.984807+0000","last_unstale":"2024-04-25T05:42:54.945987+0000","last_undegraded":"2024-04-25T05:42:54.945987+0000","last_fullsized":"2024-04-25T05:42:54.945987+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:35.061132+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":490,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940837+0000","last_change":"2024-04-25T05:40:49.444611+0000","last_active":"2024-04-25T05:42:54.940837+0000","last_peered":"2024-04-25T05:42:54.940837+0000","last_clean":"2024-04-25T05:42:54.940837+0000","last_became_active":"2024-04-25T05:33:58.688133+0000","last_became_peered":"2024-04-25T05:33:58.688133+0000","last_unstale":"2024-04-25T05:42:54.940837+0000","last_undegraded":"2024-04-25T05:42:54.940837+0000","last_fullsized":"2024-04-25T05:42:54.940837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:49.444492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.444492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.945986+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:42:54.945986+0000","last_peered":"2024-04-25T05:42:54.945986+0000","last_clean":"2024-04-25T05:42:54.945986+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:42:54.945986+0000","last_undegraded":"2024-04-25T05:42:54.945986+0000","last_fullsized":"2024-04-25T05:42:54.945986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:04.487676+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":478,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941624+0000","last_change":"2024-04-25T05:41:31.571478+0000","last_active":"2024-04-25T05:42:54.941624+0000","last_peered":"2024-04-25T05:42:54.941624+0000","last_clean":"2024-04-25T05:42:54.941624+0000","last_became_active":"2024-04-25T05:34:04.722309+0000","last_became_peered":"2024-04-25T05:34:04.722309+0000","last_unstale":"2024-04-25T05:42:54.941624+0000","last_undegraded":"2024-04-25T05:42:54.941624+0000","last_fullsized":"2024-04-25T05:42:54.941624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.17","version":"293'14713","reported_seq":17494,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.825637+0000","last_change":"2024-04-25T05:42:49.196867+0000","last_active":"2024-04-25T05:42:57.825637+0000","last_peered":"2024-04-25T05:42:57.825637+0000","last_clean":"2024-04-25T05:42:57.825637+0000","last_became_active":"2024-04-25T05:42:49.037097+0000","last_became_peered":"2024-04-25T05:42:49.037097+0000","last_unstale":"2024-04-25T05:42:57.825637+0000","last_undegraded":"2024-04-25T05:42:57.825637+0000","last_fullsized":"2024-04-25T05:42:57.825637+0000","mapping_epoch":290,"log_start":"259'9767","ondisk_log_start":"259'9767","created":283,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"291'14327","last_scrub_stamp":"2024-04-25T05:42:49.196762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.196762+0000","objects_scrubbed":441,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:51.548848+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'22","reported_seq":241,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.944899+0000","last_change":"2024-04-25T05:42:19.252275+0000","last_active":"2024-04-25T05:42:54.944899+0000","last_peered":"2024-04-25T05:42:54.944899+0000","last_clean":"2024-04-25T05:42:54.944899+0000","last_became_active":"2024-04-25T05:40:43.433465+0000","last_became_peered":"2024-04-25T05:40:43.433465+0000","last_unstale":"2024-04-25T05:42:54.944899+0000","last_undegraded":"2024-04-25T05:42:54.944899+0000","last_fullsized":"2024-04-25T05:42:54.944899+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:24.890181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":502,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941761+0000","last_change":"2024-04-25T05:39:40.373777+0000","last_active":"2024-04-25T05:42:54.941761+0000","last_peered":"2024-04-25T05:42:54.941761+0000","last_clean":"2024-04-25T05:42:54.941761+0000","last_became_active":"2024-04-25T05:39:29.797406+0000","last_became_peered":"2024-04-25T05:39:29.797406+0000","last_unstale":"2024-04-25T05:42:54.941761+0000","last_undegraded":"2024-04-25T05:42:54.941761+0000","last_fullsized":"2024-04-25T05:42:54.941761+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:40.373603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:39:40.373603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":486,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.940393+0000","last_change":"2024-04-25T05:40:43.453942+0000","last_active":"2024-04-25T05:42:54.940393+0000","last_peered":"2024-04-25T05:42:54.940393+0000","last_clean":"2024-04-25T05:42:54.940393+0000","last_became_active":"2024-04-25T05:40:43.453366+0000","last_became_peered":"2024-04-25T05:40:43.453366+0000","last_unstale":"2024-04-25T05:42:54.940393+0000","last_undegraded":"2024-04-25T05:42:54.940393+0000","last_fullsized":"2024-04-25T05:42:54.940393+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":139,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.937600+0000","last_change":"2024-04-25T05:42:49.433877+0000","last_active":"2024-04-25T05:42:54.937600+0000","last_peered":"2024-04-25T05:42:54.937600+0000","last_clean":"2024-04-25T05:42:54.937600+0000","last_became_active":"2024-04-25T05:40:43.441308+0000","last_became_peered":"2024-04-25T05:40:43.441308+0000","last_unstale":"2024-04-25T05:42:54.937600+0000","last_undegraded":"2024-04-25T05:42:54.937600+0000","last_fullsized":"2024-04-25T05:42:54.937600+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:14.115420+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"293'14790","reported_seq":29612,"reported_epoch":293,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:42:58.289193+0000","last_change":"2024-04-25T05:42:44.126974+0000","last_active":"2024-04-25T05:42:58.289193+0000","last_peered":"2024-04-25T05:42:58.289193+0000","last_clean":"2024-04-25T05:42:41.954792+0000","last_became_active":"2024-04-25T05:42:44.065302+0000","last_became_peered":"2024-04-25T05:42:44.065302+0000","last_unstale":"2024-04-25T05:42:58.289193+0000","last_undegraded":"2024-04-25T05:42:58.289193+0000","last_fullsized":"2024-04-25T05:42:58.289193+0000","mapping_epoch":286,"log_start":"253'9357","ondisk_log_start":"253'9357","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:39:18.209488+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":248864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3646,"num_bytes_recovered":238845952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'39","reported_seq":641,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.487302+0000","last_change":"2024-04-25T05:42:57.487370+0000","last_active":"2024-04-25T05:42:57.487302+0000","last_peered":"2024-04-25T05:42:57.487302+0000","last_clean":"2024-04-25T05:42:57.487302+0000","last_became_active":"2024-04-25T05:40:55.974452+0000","last_became_peered":"2024-04-25T05:40:55.974452+0000","last_unstale":"2024-04-25T05:42:57.487302+0000","last_undegraded":"2024-04-25T05:42:57.487302+0000","last_fullsized":"2024-04-25T05:42:57.487302+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:18.131337+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":501,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.947122+0000","last_change":"2024-04-25T05:40:55.982555+0000","last_active":"2024-04-25T05:42:54.947122+0000","last_peered":"2024-04-25T05:42:54.947122+0000","last_clean":"2024-04-25T05:42:54.947122+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:42:54.947122+0000","last_undegraded":"2024-04-25T05:42:54.947122+0000","last_fullsized":"2024-04-25T05:42:54.947122+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.485053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":450,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941994+0000","last_change":"2024-04-25T05:40:13.865954+0000","last_active":"2024-04-25T05:42:54.941994+0000","last_peered":"2024-04-25T05:42:54.941994+0000","last_clean":"2024-04-25T05:42:54.941994+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:42:54.941994+0000","last_undegraded":"2024-04-25T05:42:54.941994+0000","last_fullsized":"2024-04-25T05:42:54.941994+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:13.865834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:13.865834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.946316+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:42:54.946316+0000","last_peered":"2024-04-25T05:42:54.946316+0000","last_clean":"2024-04-25T05:42:54.946316+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:42:54.946316+0000","last_undegraded":"2024-04-25T05:42:54.946316+0000","last_fullsized":"2024-04-25T05:42:54.946316+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:56.109656+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.15","version":"293'14938","reported_seq":15406,"reported_epoch":293,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:42:57.857049+0000","last_change":"2024-04-25T05:42:44.086512+0000","last_active":"2024-04-25T05:42:57.857049+0000","last_peered":"2024-04-25T05:42:57.857049+0000","last_clean":"2024-04-25T05:42:41.882035+0000","last_became_active":"2024-04-25T05:42:44.053070+0000","last_became_peered":"2024-04-25T05:42:44.053070+0000","last_unstale":"2024-04-25T05:42:57.857049+0000","last_undegraded":"2024-04-25T05:42:57.857049+0000","last_fullsized":"2024-04-25T05:42:57.857049+0000","mapping_epoch":286,"log_start":"256'9614","ondisk_log_start":"256'9614","created":283,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":5,"last_scrub":"259'9942","last_scrub_stamp":"2024-04-25T05:41:36.031226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:41:36.031226+0000","objects_scrubbed":5781,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":68,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7787,"num_write_kb":251360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":13959168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":68}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"211'20","reported_seq":554,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.869130+0000","last_change":"2024-04-25T05:42:06.942759+0000","last_active":"2024-04-25T05:42:54.869130+0000","last_peered":"2024-04-25T05:42:54.869130+0000","last_clean":"2024-04-25T05:42:54.869130+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:42:54.869130+0000","last_undegraded":"2024-04-25T05:42:54.869130+0000","last_fullsized":"2024-04-25T05:42:54.869130+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:42:06.942600+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:42:06.942600+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:25.919821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":146,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.544142+0000","last_change":"2024-04-25T05:42:57.544176+0000","last_active":"2024-04-25T05:42:57.544142+0000","last_peered":"2024-04-25T05:42:57.544142+0000","last_clean":"2024-04-25T05:42:57.544142+0000","last_became_active":"2024-04-25T05:40:55.953726+0000","last_became_peered":"2024-04-25T05:40:55.953726+0000","last_unstale":"2024-04-25T05:42:57.544142+0000","last_undegraded":"2024-04-25T05:42:57.544142+0000","last_fullsized":"2024-04-25T05:42:57.544142+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:57.544063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.544063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:10.798448+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.941099+0000","last_change":"2024-04-25T05:40:37.473300+0000","last_active":"2024-04-25T05:42:54.941099+0000","last_peered":"2024-04-25T05:42:54.941099+0000","last_clean":"2024-04-25T05:42:54.941099+0000","last_became_active":"2024-04-25T05:34:02.700452+0000","last_became_peered":"2024-04-25T05:34:02.700452+0000","last_unstale":"2024-04-25T05:42:54.941099+0000","last_undegraded":"2024-04-25T05:42:54.941099+0000","last_fullsized":"2024-04-25T05:42:54.941099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:37.473018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:37.473018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":161,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:54.942023+0000","last_change":"2024-04-25T05:41:35.583008+0000","last_active":"2024-04-25T05:42:54.942023+0000","last_peered":"2024-04-25T05:42:54.942023+0000","last_clean":"2024-04-25T05:42:54.942023+0000","last_became_active":"2024-04-25T05:40:09.734217+0000","last_became_peered":"2024-04-25T05:40:09.734217+0000","last_unstale":"2024-04-25T05:42:54.942023+0000","last_undegraded":"2024-04-25T05:42:54.942023+0000","last_fullsized":"2024-04-25T05:42:54.942023+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:43:01.911520+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"293'15018","reported_seq":31351,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-04-25T05:42:57.751309+0000","last_change":"2024-04-25T05:42:50.412721+0000","last_active":"2024-04-25T05:42:57.751309+0000","last_peered":"2024-04-25T05:42:57.751309+0000","last_clean":"2024-04-25T05:42:57.751309+0000","last_became_active":"2024-04-25T05:42:50.036083+0000","last_became_peered":"2024-04-25T05:42:50.036083+0000","last_unstale":"2024-04-25T05:42:57.751309+0000","last_undegraded":"2024-04-25T05:42:57.751309+0000","last_fullsized":"2024-04-25T05:42:57.751309+0000","mapping_epoch":291,"log_start":"259'10037","ondisk_log_start":"259'10037","created":283,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"292'14626","last_scrub_stamp":"2024-04-25T05:42:50.412410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:18.209488+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.412410+0000","objects_scrubbed":448,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:16.186460+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":395,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":105316376,"num_objects":1608,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":348,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131655,"num_write_kb":3972449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35546,"num_bytes_recovered":2329215000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418080256,"data_stored":1418068040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125916,"ondisk_log_size":125916,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":526,"num_read_kb":521,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":974,"num_read_kb":699,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":5506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":12344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":671,"ondisk_log_size":671,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070577,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449940,"kb_used_data":107724,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93295212,"statfs":{"total":95995035648,"available":95534297088,"internally_reserved":0,"allocated":110309376,"data_stored":109735060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":350353866},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364917,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416344,"kb_used_data":64396,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93328808,"statfs":{"total":95995035648,"available":95568699392,"internally_reserved":0,"allocated":65941504,"data_stored":65367102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574990,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464552,"kb_used_data":45856,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93280600,"statfs":{"total":95995035648,"available":95519334400,"internally_reserved":0,"allocated":46956544,"data_stored":46360762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487696,"kb_used_data":81672,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93257456,"statfs":{"total":95995035648,"available":95495634944,"internally_reserved":0,"allocated":83632128,"data_stored":83060371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927904,"kb_used_data":449688,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92817248,"statfs":{"total":95995035648,"available":95044861952,"internally_reserved":0,"allocated":460480512,"data_stored":459893208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":489683399},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986302,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":420628,"kb_used_data":92364,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93324524,"statfs":{"total":95995035648,"available":95564312576,"internally_reserved":0,"allocated":94580736,"data_stored":94005480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691962,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668880,"kb_used_data":304516,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93076272,"statfs":{"total":95995035648,"available":95310102528,"internally_reserved":0,"allocated":311824384,"data_stored":311257402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44695552,"data_stored":44695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28770304,"data_stored":28770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":463273984,"data_stored":463273984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":458293248,"data_stored":458293248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61210624,"data_stored":61210624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":251269120,"data_stored":251265048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":47779840,"data_stored":47775768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":62787584,"data_stored":62783512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:43:02.749 INFO:tasks.thrashosds.thrasher:Setting 8.16 to [4, 1, 7] 2024-04-25T05:43:02.749 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '8.16', '4', '1', '7'] 2024-04-25T05:43:02.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 8.16 4 1 7 2024-04-25T05:43:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:02.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:02.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:02.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:02.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:02.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:03.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:03.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:03.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:03.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:03.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:03.274 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:03.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:03.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[27010]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:43:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[27010]: pgmap v474: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 924 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:03.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[37492]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:43:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[37492]: pgmap v474: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 924 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:03.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:03.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:03.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:03 smithi060 ceph-mon[30898]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:43:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:03 smithi060 ceph-mon[30898]: pgmap v474: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 101 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 924 op/s; 348/5514 objects misplaced (6.311%) 2024-04-25T05:43:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:03.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:03.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:03.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:03.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:03.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:03.847 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:03.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:03.859 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:43:03.859 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:43:03.859 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:43:03.859 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:43:03.859 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:43:03.879 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:03.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:03.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:43:03.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:04.038 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:04.131 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_1' removed 2024-04-25T05:43:04.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:04.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:04.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:04.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:04.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:04.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:43:04.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:04.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:04.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:04.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:43:04.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:04.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:04.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:04.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:43:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:04.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:04.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:04.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.693 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:04.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:04.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:04.816 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:04.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:04.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:04.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:05.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:05.061 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-25T05:43:05.062 INFO:tasks.radosbench:Beginning radosbench... 2024-04-25T05:43:05.062 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-25T05:43:05.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-04-25T05:43:05.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:05.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.190 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:05.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:05.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:05.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:05.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[27010]: pgmap v475: 168 pgs: 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 162 active+clean; 4.4 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 999 op/s; 109/876 objects misplaced (12.443%) 2024-04-25T05:43:05.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:43:05.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[27010]: osdmap e295: 8 total, 8 up, 7 in 2024-04-25T05:43:05.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[37492]: pgmap v475: 168 pgs: 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 162 active+clean; 4.4 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 999 op/s; 109/876 objects misplaced (12.443%) 2024-04-25T05:43:05.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:43:05.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:05 smithi043 ceph-mon[37492]: osdmap e295: 8 total, 8 up, 7 in 2024-04-25T05:43:05.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:05 smithi060 ceph-mon[30898]: pgmap v475: 168 pgs: 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 162 active+clean; 4.4 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 999 op/s; 109/876 objects misplaced (12.443%) 2024-04-25T05:43:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2931554105' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:43:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:05 smithi060 ceph-mon[30898]: osdmap e295: 8 total, 8 up, 7 in 2024-04-25T05:43:05.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:05.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:05.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:05.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:05.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:05.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:05.865 INFO:teuthology.orchestra.run.smithi043.stderr:Error ENOENT: pgid '8.16' does not exist 2024-04-25T05:43:05.868 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:05.891 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:05.911 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:05.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:05.932 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:05.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:05.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:05.952 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:05.974 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:05.994 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:06.014 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:06.015 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:06.016 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:06.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:06.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:06.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:06.184 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/404413795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1980580206' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/404413795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1980580206' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.251 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:06.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:06.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:06.433 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/404413795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1980580206' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.16", "id": [4, 1, 7]}]: dispatch 2024-04-25T05:43:06.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:06.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:06.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:43:06.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:06.709 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:06.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:06.833 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:06.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:06.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:06.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:07.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-25T05:43:07.080 INFO:tasks.thrashosds.thrasher:Failed to rm-pg-upmap, ignoring 2024-04-25T05:43:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.082 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:07.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:07.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:07.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:07.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:07.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:07.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: pgmap v477: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:07.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: pgmap v477: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:43:07.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:07.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:07.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: pgmap v477: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:43:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:07.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:07.575 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:07.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:07.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:07.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:07.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:07.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:07.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:07.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:08.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:08.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:08.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:08.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:08.318 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:08.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:08.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:08.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:08.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:08.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:08.764 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:08.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:08.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:08.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:08.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:08.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:09.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:09.071 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:09.138 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[27010]: pgmap v478: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:43:09.139 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_2' created 2024-04-25T05:43:09.139 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:43:09.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:09.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:09.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:09.321 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:09.321 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:09.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/290121427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-25T05:43:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[37492]: pgmap v478: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:43:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/290121427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-25T05:43:09.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:09.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:09 smithi060 ceph-mon[30898]: pgmap v478: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 583 KiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-25T05:43:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/290121427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-25T05:43:09.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.575 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:09.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:09.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:09.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:09.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:43:09.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:09.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:09.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:43:09.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:09.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:10.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-25T05:43:10.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:10.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:10.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:10.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:10.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:10.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:10.382 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[37492]: osdmap e296: 8 total, 8 up, 7 in 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[37492]: osdmap e297: 8 total, 8 up, 7 in 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[27010]: osdmap e296: 8 total, 8 up, 7 in 2024-04-25T05:43:10.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:10 smithi043 ceph-mon[27010]: osdmap e297: 8 total, 8 up, 7 in 2024-04-25T05:43:10.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:10.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:10 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3290434552' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-25T05:43:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:10 smithi060 ceph-mon[30898]: osdmap e296: 8 total, 8 up, 7 in 2024-04-25T05:43:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:10 smithi060 ceph-mon[30898]: osdmap e297: 8 total, 8 up, 7 in 2024-04-25T05:43:10.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:10.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:10.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:10.732 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:10.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.859 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:10.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:10.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:10.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:10.982 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:10.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:11.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:11.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:11.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:11.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:11.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:11.359 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[37492]: pgmap v480: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:43:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[37492]: osdmap e298: 8 total, 8 up, 7 in 2024-04-25T05:43:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[27010]: pgmap v480: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:43:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:11 smithi043 ceph-mon[27010]: osdmap e298: 8 total, 8 up, 7 in 2024-04-25T05:43:11.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:11.487 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:11 smithi060 ceph-mon[30898]: pgmap v480: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:11 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:43:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:11 smithi060 ceph-mon[30898]: osdmap e298: 8 total, 8 up, 7 in 2024-04-25T05:43:11.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:11.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:11.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:43:11.660 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:11.661 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:11.716 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:11.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:11.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:11.915 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:11.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:11.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:11.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:12.081 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 7, 6, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:43:12.081 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:43:12.082 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 3, 4, 5, 7, 6, 0] 2024-04-25T05:43:12.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd out 5 2024-04-25T05:43:12.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:12.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:12.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:12.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:12.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:12.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:12 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:12.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:12 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:12.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:12 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:12.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:12 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:12.351 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:12 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:12.351 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:12 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:12.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:12.406 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:12.406 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:12.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:12.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:12.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:12.601 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:12.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:12.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:12.735 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:12.755 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:12.775 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:12.797 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:12.801 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:12.819 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:12.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:12.841 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:12.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:12.861 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:12.881 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:12.882 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:12.882 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:12.882 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:12.883 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:12.883 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:12.883 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:12.883 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:12.883 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:12.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:12.971 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:12.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:13.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:13.094 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:13.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:43:13.166 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-25T05:43:13.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:13.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:43:13.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:13.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:13.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:13.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:13.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:13.478 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:13.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-mon[30898]: pgmap v483: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:43:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-mon[30898]: osdmap e299: 8 total, 8 up, 7 in 2024-04-25T05:43:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1082052936' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1234348352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:13.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:13.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:13.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[27010]: pgmap v483: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:43:13.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[27010]: osdmap e299: 8 total, 8 up, 7 in 2024-04-25T05:43:13.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1082052936' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1234348352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:13.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[37492]: pgmap v483: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:43:13.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[37492]: osdmap e299: 8 total, 8 up, 7 in 2024-04-25T05:43:13.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1082052936' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:43:13.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1234348352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:13.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:13.714 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:13.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:13.837 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:13.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:13.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:13.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:13.981 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:13.981 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:43:13.981 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:43:13.981 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:43:13.981 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:14.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:14.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:14.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:14.214 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:14.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[27010]: osdmap e300: 8 total, 8 up, 7 in 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[37492]: osdmap e300: 8 total, 8 up, 7 in 2024-04-25T05:43:14.335 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:14.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:14.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.462 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:14.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:43:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:14 smithi060 ceph-mon[30898]: osdmap e300: 8 total, 8 up, 7 in 2024-04-25T05:43:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:14.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.585 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:14.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:14.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:43:14.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:14.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:14.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:14.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:14.888 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:14 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:43:14.593+0000 7f173ec68700 -1 mon.a@0(leader).osd e300 definitely_dead 0 2024-04-25T05:43:14.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:14.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:15.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:15.076 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:15.106 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-25T05:43:15.120 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-25T05:43:15.120 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-25T05:43:15.144 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:15.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:15.184 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.5. 2024-04-25T05:43:15.197 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-04-25T05:43:15.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-25T05:43:15.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_129820 2024-04-25T05:43:15.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:43:15.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-04-25T05:43:15.211 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:15.284 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:15 smithi060 ceph-mon[30898]: pgmap v486: 158 pgs: 158 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:43:15.284 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:15 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:43:15.284 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-25T05:43:15.284 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:15 smithi060 ceph-mon[30898]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-25T05:43:15.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:15.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[37492]: pgmap v486: 158 pgs: 158 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:43:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:43:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-25T05:43:15.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[37492]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-25T05:43:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[27010]: pgmap v486: 158 pgs: 158 active+clean; 583 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:43:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:43:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-25T05:43:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:15 smithi043 ceph-mon[27010]: Client client.admin marked osd.5 out, while it was still marked up 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:129506 7.1 7:85e13d17:::benchmark_data_smithi043_74356_object12637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:43:15.416 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 372.731945768, 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.149986696, 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129506 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682202+0000", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.017e-06 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682208+0000", 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8930000000000002e-06 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682218+0000", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0373e-05 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682226+0000", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2579999999999995e-06 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682278+0000", 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2231999999999997e-05 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682392+0000", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011332000000000001 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682493+0000", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010118 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.684382+0000", 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018894020000000001 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.684645+0000", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026231699999999998 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.741914+0000", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.057269792 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832066+0000", 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.090151063000000004 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832106+0000", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0522000000000003e-05 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832187+0000", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1327000000000002e-05 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131176 7.1 7:892c0929:::benchmark_data_smithi043_74356_object14307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.057114+0000", 2024-04-25T05:43:15.422 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 370.35703250400002, 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115324623, 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131176 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057114+0000", 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057114+0000", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057116+0000", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.604e-06 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057121+0000", 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8850000000000004e-06 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057135+0000", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3472000000000001e-05 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057143+0000", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3240000000000003e-06 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057204+0000", 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0443999999999999e-05 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057340+0000", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013671199999999999 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057479+0000", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013823899999999999 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.062170+0000", 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046911230000000002 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.062194+0000", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3765e-05 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.171048+0000", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.108854272 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.172292+0000", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001244415 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.172357+0000", 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4354000000000001e-05 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.172439+0000", 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2014000000000005e-05 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.428 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:174866 7.1 7:88ee4521:::benchmark_data_smithi043_74356_object57997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 312.34785266199998, 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12223832499999999, 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.429 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 174866 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:43:15.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066295+0000", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3e-06 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066302+0000", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2300000000000002e-06 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066314+0000", 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1701e-05 2024-04-25T05:43:15.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066323+0000", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.524e-06 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066368+0000", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5392999999999997e-05 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066449+0000", 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0623000000000004e-05 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066576+0000", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012777700000000001 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.068628+0000", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020514819999999999 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.069247+0000", 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000619153 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188455+0000", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.119208436 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188498+0000", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2179999999999999e-05 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188532+0000", 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4526000000000001e-05 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196432 7.4 7:201f4639:::benchmark_data_smithi043_74356_object79563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 283.51260414900003, 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.18528801, 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196432 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901544+0000", 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3030000000000001e-06 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901549+0000", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901564+0000", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5019e-05 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901572+0000", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2009999999999999e-06 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.437 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901623+0000", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1400000000000003e-05 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901753+0000", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012954700000000001 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901869+0000", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011555199999999999 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904486+0000", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002616967 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904523+0000", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7156000000000001e-05 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086719+0000", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.182196424 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086782+0000", 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2833e-05 2024-04-25T05:43:15.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086830+0000", 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8378000000000003e-05 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196436 7.1 7:86601c62:::benchmark_data_smithi043_74356_object79567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 283.51210586100001, 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.177590574, 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.440 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196436 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:43:15.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8000000000000005e-07 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902048+0000", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9419999999999998e-06 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902058+0000", 2024-04-25T05:43:15.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.008e-05 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902066+0000", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2790000000000001e-06 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902249+0000", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018350800000000001 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902354+0000", 2024-04-25T05:43:15.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105227 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902460+0000", 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105164 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904687+0000", 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022273990000000001 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907211+0000", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002523941 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079433+0000", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17222240799999999 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079565+0000", 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013155999999999999 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079631+0000", 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6285999999999995e-05 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124112 8.2 8:4cdac3fe:::benchmark_data_smithi043_104001_object124111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 138.30224721799999, 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16378072499999999, 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.446 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124112 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111905+0000", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1059999999999996e-06 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111939+0000", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3980999999999999e-05 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111951+0000", 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1635e-05 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111959+0000", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1689999999999992e-06 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112075+0000", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011596500000000001 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112203+0000", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012803900000000001 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112363+0000", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000160039 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116878+0000", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045148760000000001 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116924+0000", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5593999999999997e-05 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.119845+0000", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002921589 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275525+0000", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15567968900000001 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275567+0000", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1882999999999999e-05 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275680+0000", 2024-04-25T05:43:15.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011315999999999999 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125664 8.2 8:4acd28bf:::benchmark_data_smithi043_104001_object1540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 135.127890302, 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11907625199999999, 2024-04-25T05:43:15.452 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125664 2024-04-25T05:43:15.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286258+0000", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.415e-06 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286264+0000", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286279+0000", 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5085e-05 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286287+0000", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9650000000000008e-06 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287683+0000", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001395269 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287830+0000", 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147777 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287984+0000", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015359799999999999 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291040+0000", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030556089999999999 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291225+0000", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018544300000000001 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291399+0000", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017389099999999999 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405138+0000", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11373876199999999 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405204+0000", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6484999999999998e-05 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405332+0000", 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012813700000000001 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:126959 8.2 8:4e62c023:::benchmark_data_smithi043_104001_object2835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 133.56245076499999, 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14632261699999999, 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.459 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 126959 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851697+0000", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.237e-06 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851705+0000", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8320000000000006e-06 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851714+0000", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.064e-06 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851720+0000", 2024-04-25T05:43:15.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0859999999999998e-06 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851962+0000", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024216799999999999 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852126+0000", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016425399999999999 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852306+0000", 2024-04-25T05:43:15.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000180082 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.854938+0000", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026320139999999998 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855570+0000", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00063133700000000004 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855614+0000", 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4480000000000001e-05 2024-04-25T05:43:15.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997822+0000", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.142207745 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997883+0000", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1085000000000002e-05 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.998018+0000", 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013523300000000001 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.464 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129538 8.2 8:4c65616f:::benchmark_data_smithi043_104001_object5414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.40312303900001, 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12767905500000001, 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129538 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.465 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011025+0000", 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1650000000000001e-06 2024-04-25T05:43:15.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011031+0000", 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2639999999999997e-06 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011042+0000", 2024-04-25T05:43:15.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1385e-05 2024-04-25T05:43:15.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:15.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011051+0000", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3149999999999992e-06 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.012950+0000", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018998400000000001 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013084+0000", 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013384000000000001 2024-04-25T05:43:15.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013219+0000", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013510700000000001 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018871+0000", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056513270000000003 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018951+0000", 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0836999999999999e-05 2024-04-25T05:43:15.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022366+0000", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034148400000000001 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023969+0000", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001602526 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.024068+0000", 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9500999999999995e-05 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138702+0000", 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.114634108 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129549 8.2 8:40741811:::benchmark_data_smithi043_104001_object5425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.39571432599999, 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13464008199999999, 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.472 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129549 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018434+0000", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.528e-06 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018439+0000", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3040000000000004e-06 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018450+0000", 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0788000000000001e-05 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018458+0000", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.405e-06 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018994+0000", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053616199999999997 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019116+0000", 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121228 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019243+0000", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012719599999999999 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022328+0000", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030850640000000002 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138716+0000", 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11638823700000001 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138755+0000", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8992999999999997e-05 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152783+0000", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014027773 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152919+0000", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013651599999999999 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153072+0000", 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152888 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129543 8.e e246/240 8:7455de0e:::benchmark_data_smithi043_104001_object5419:head v 246'7976, mlcod=245'7972)", 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.39410261099999, 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12636956799999999, 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.82e-07 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020049+0000", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2490000000000003e-06 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020060+0000", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1070000000000001e-05 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020065+0000", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2310000000000004e-06 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020859+0000", 2024-04-25T05:43:15.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000793862 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020882+0000", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3170999999999999e-05 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146330+0000", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12544777300000001 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146413+0000", 2024-04-25T05:43:15.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3629999999999997e-05 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:129549 8.2 e246/233 ondisk, result = 0)", 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.39100916800001, 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1156257, 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023138+0000", 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8999999999999996e-07 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023142+0000", 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2039999999999999e-06 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023149+0000", 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5019999999999998e-06 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023155+0000", 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0229999999999998e-06 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138742+0000", 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115587254 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138750+0000", 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8089999999999993e-06 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138763+0000", 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3318e-05 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129546 8.e e246/240 8:758a41b0:::benchmark_data_smithi043_104001_object5422:head v 246'7977, mlcod=245'7972)", 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.39029367500001, 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12274840099999999, 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.486 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9200000000000001e-07 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023859+0000", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023867+0000", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0909999999999993e-06 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023874+0000", 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8240000000000002e-06 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139217+0000", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115343705 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139245+0000", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7317999999999998e-05 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146417+0000", 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0071721129999999999 2024-04-25T05:43:15.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146601+0000", 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018442799999999999 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.490 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129556 8.2 8:484428a6:::benchmark_data_smithi043_104001_object5432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 130.386085087, 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.125238935, 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129556 2024-04-25T05:43:15.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028062+0000", 2024-04-25T05:43:15.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.091e-06 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028067+0000", 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8489999999999997e-06 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137388+0000", 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.109320815 2024-04-25T05:43:15.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137405+0000", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6677999999999999e-05 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138822+0000", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001417614 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138949+0000", 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012701800000000001 2024-04-25T05:43:15.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139092+0000", 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014223099999999999 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.141864+0000", 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027722189999999998 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143235+0000", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001370778 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143353+0000", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118484 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153140+0000", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0097873479999999995 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153195+0000", 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5000999999999997e-05 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153300+0000", 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010480900000000001 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.2 8.3a1a8b12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:43:15.497 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.543508066000001, 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12946094299999999, 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870640+0000", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7379999999999999e-06 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870647+0000", 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0330000000000003e-06 2024-04-25T05:43:15.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870659+0000", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1748e-05 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870667+0000", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3100000000000001e-06 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000085+0000", 2024-04-25T05:43:15.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12941729699999999 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000099+0000", 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4817000000000001e-05 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.2 8.516c972 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.535660436000001, 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122103355, 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.501 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878488+0000", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0209999999999998e-06 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878500+0000", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1629e-05 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878519+0000", 2024-04-25T05:43:15.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9522999999999999e-05 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878530+0000", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0815000000000001e-05 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000576+0000", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122046021 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000589+0000", 2024-04-25T05:43:15.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3346000000000001e-05 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.12 8:48d1585c:::benchmark_data_smithi043_104001_object102215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.520629325000002, 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1294363890000001, 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.505 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893519+0000", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1519999999999999e-06 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893531+0000", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1849e-05 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893551+0000", 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9638000000000001e-05 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893564+0000", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.323e-05 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000742+0000", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10717832200000001 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000752+0000", 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9930000000000007e-06 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006085+0000", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0053329209999999 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006212+0000", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126873 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006455+0000", 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024327100000000001 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011490+0000", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050344350000000003 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.012956+0000", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001465872 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022845+0000", 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0098890200000000001 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022904+0000", 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9239e-05 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022954+0000", 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9573999999999997e-05 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.12 8:4e9368a0:::benchmark_data_smithi043_104001_object102219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:15.511 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 35.520554683999997, 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1317489059999999, 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4000000000000002e-07 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893595+0000", 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8600000000000001e-06 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893603+0000", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9000000000000006e-06 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893609+0000", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2500000000000003e-06 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000774+0000", 2024-04-25T05:43:15.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10716527300000001 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000780+0000", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7119999999999997e-06 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009444+0000", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086639690000001 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009577+0000", 2024-04-25T05:43:15.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013284399999999999 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009825+0000", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024752200000000002 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.013047+0000", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003222042 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020080+0000", 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070331810000000003 2024-04-25T05:43:15.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025198+0000", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051180369999999998 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025295+0000", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6797000000000002e-05 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025341+0000", 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6078999999999997e-05 2024-04-25T05:43:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228033 8.14 8:2d5c3048:::benchmark_data_smithi043_104001_object103909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.425992905000001, 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0985097700000002, 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.518 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228033 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988156+0000", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9879999999999999e-06 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988162+0000", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7870000000000004e-06 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988180+0000", 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7886000000000001e-05 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988190+0000", 2024-04-25T05:43:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9729999999999992e-06 2024-04-25T05:43:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988272+0000", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1977999999999993e-05 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988285+0000", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3176000000000001e-05 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980591+0000", 2024-04-25T05:43:15.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99230595399999999 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980607+0000", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5548000000000001e-05 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081723+0000", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1011164630000001 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081910+0000", 2024-04-25T05:43:15.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018638699999999999 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.082205+0000", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00029534400000000001 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085239+0000", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030338330000000001 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085283+0000", 2024-04-25T05:43:15.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3695000000000001e-05 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085372+0000", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.954e-05 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086212+0000", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084000600000000002 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086435+0000", 2024-04-25T05:43:15.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022244300000000001 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086501+0000", 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6000000000000005e-05 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086663+0000", 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162769 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228057 8.14 8:29afc790:::benchmark_data_smithi043_104001_object103933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 33.404180701000001, 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0771824429999999, 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228057 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:15.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009967+0000", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.418e-06 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:15.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009982+0000", 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5211000000000001e-05 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009997+0000", 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4542000000000001e-05 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010006+0000", 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3619999999999995e-06 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010067+0000", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0965000000000003e-05 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010076+0000", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1249999999999999e-06 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980702+0000", 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97062569499999996 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980712+0000", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0081999999999999e-05 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083607+0000", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1028946399999999 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083747+0000", 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013999600000000001 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.084002+0000", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025568299999999998 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085297+0000", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001294033 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085711+0000", 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041416199999999998 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086266+0000", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055577100000000004 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086728+0000", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000461335 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086920+0000", 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019192499999999999 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086999+0000", 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9742999999999999e-05 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.087148+0000", 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014875500000000001 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:15.534 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:15.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:15.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:15.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:15.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:15.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:15.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:15.917 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:15.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:16.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:16.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:16.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.164 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:16.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:16.197 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:43:16.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 36 20 1.25001 1.25 0.00146388 0.00881213 2024-04-25T05:43:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-25T05:43:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:16 smithi060 ceph-mon[30898]: osdmap e301: 8 total, 8 up, 6 in 2024-04-25T05:43:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/925980108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:16 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:16.296 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:16.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:16.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:16.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:43:16.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:16.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[27010]: osdmap e301: 8 total, 8 up, 6 in 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/925980108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4170265843' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[37492]: osdmap e301: 8 total, 8 up, 6 in 2024-04-25T05:43:16.602 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/925980108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.603 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:16 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:16.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:16.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:16.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:16.846 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:16.874 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:16.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:16.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:17.005 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:17.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:17.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:43:17.140 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:17.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 3026 3010 94.058 186.875 0.00267765 0.0105448 2024-04-25T05:43:17.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: pgmap v488: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:17 smithi060 ceph-mon[30898]: osdmap e302: 8 total, 8 up, 6 in 2024-04-25T05:43:17.280 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:17.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: pgmap v488: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[27010]: osdmap e302: 8 total, 8 up, 6 in 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: pgmap v488: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:17.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:17 smithi043 ceph-mon[37492]: osdmap e302: 8 total, 8 up, 6 in 2024-04-25T05:43:17.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:17.472 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:17.472 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:43:17.472 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:43:17.472 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:17.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:17.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:17.584 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:17.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:17.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:17.832 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:17.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:17.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:17.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:18.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:18.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:18.198 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 5871 5855 121.972 177.812 0.00659417 0.0081654 2024-04-25T05:43:18.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:18.239 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: osdmap e303: 8 total, 8 up, 6 in 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:43:18.258 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:18.259 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:18.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:43:18.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:18.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:18.510 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: osdmap e303: 8 total, 8 up, 6 in 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:18.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: osdmap e303: 8 total, 8 up, 6 in 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:18.512 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:18.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:18.548 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:18.644 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:18.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:18.688 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:18.690 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:18.735 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:18.782 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:18.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:18.823 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:18.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:18.862 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:18.904 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:18.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:18.947 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:18.951 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:18.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:18.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:19.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:19.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:19.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:19.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 8499 8483 132.538 164.25 0.00546119 0.00753095 2024-04-25T05:43:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:19.306 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:19.306 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:19.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:19.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:19.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: pgmap v491: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: osdmap e304: 8 total, 8 up, 6 in 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 4.1e scrub starts 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 4.1e scrub ok 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3837542541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:43:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:19 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:43:19.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:43:19.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:19.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: pgmap v491: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: osdmap e304: 8 total, 8 up, 6 in 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 4.1e scrub starts 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 4.1e scrub ok 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3837542541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: pgmap v491: 158 pgs: 158 active+clean; 583 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: osdmap e304: 8 total, 8 up, 6 in 2024-04-25T05:43:19.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 4.1e scrub starts 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 4.1e scrub ok 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3837542541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:43:19.549 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:43:19.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:19.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:19.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:19.784 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:19.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:43:19.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:19.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:19.985 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:20.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:20.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:20.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:43:20.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:20.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 11105 11089 138.602 162.875 0.00531477 0.00720522 2024-04-25T05:43:20.215 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:20.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:20.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[27010]: osdmap e305: 8 total, 8 up, 6 in 2024-04-25T05:43:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:43:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:43:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[37492]: osdmap e305: 8 total, 8 up, 6 in 2024-04-25T05:43:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:43:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:20 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:43:20.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:20.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:20.497 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:20 smithi060 ceph-mon[30898]: osdmap e305: 8 total, 8 up, 6 in 2024-04-25T05:43:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:20 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:43:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:20 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:43:20.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:20.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:20.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:20.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:20.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:20.906 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:20.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:21.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:21.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:21.106 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:21.106 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:21.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:21.176 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:21.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 13478 13462 140.218 148.312 0.00819714 0.00712278 2024-04-25T05:43:21.238 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:43:21.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.307 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:21.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:21.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:21.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: pgmap v494: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 8.0 KiB/s rd, 94 MiB/s wr, 1.52k op/s; 120 KiB/s, 17 objects/s recovering 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2486554896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:43:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:21 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:43:21.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: pgmap v494: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 8.0 KiB/s rd, 94 MiB/s wr, 1.52k op/s; 120 KiB/s, 17 objects/s recovering 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2486554896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: pgmap v494: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 8.0 KiB/s rd, 94 MiB/s wr, 1.52k op/s; 120 KiB/s, 17 objects/s recovering 2024-04-25T05:43:21.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:43:21.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:43:21.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2486554896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:21.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:43:21.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:21 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:43:21.659 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:43:21.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:43:21.659 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-25T05:43:21.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-04-25T05:43:21.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:21.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:43:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:21.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:21.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:43:21.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:21.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:21.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:22.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:22.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:22.119 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:22.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:22.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:22.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 15667 15651 139.72 136.812 0.0035836 0.00714986 2024-04-25T05:43:22.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:22.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:22.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:22.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: osdmap e306: 8 total, 8 up, 6 in 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:43:22.495 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:43:22.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:22.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: osdmap e306: 8 total, 8 up, 6 in 2024-04-25T05:43:22.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:43:22.496 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:22 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:43:22.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:43:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:43:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: osdmap e306: 8 total, 8 up, 6 in 2024-04-25T05:43:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:43:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:22 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:43:22.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.574 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:22.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:22.670 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:22.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:22.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:22.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:22.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:22.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:22.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:22.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:23.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:23.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:23.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 15 17965 17950 140.215 143.688 0.00385743 0.00712449 2024-04-25T05:43:23.217 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:23.217 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:23.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:23.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:23.310 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[27010]: pgmap v496: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 6.9 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 103 KiB/s, 14 objects/s recovering 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[27010]: 4.7 scrub starts 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[27010]: 4.7 scrub ok 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[37492]: pgmap v496: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 6.9 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 103 KiB/s, 14 objects/s recovering 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[37492]: 4.7 scrub starts 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[37492]: 4.7 scrub ok 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:43:23.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:23 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:43:23.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:23.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-mon[30898]: pgmap v496: 158 pgs: 3 peering, 3 active+undersized+remapped, 152 active+clean; 377 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 6.9 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 103 KiB/s, 14 objects/s recovering 2024-04-25T05:43:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-mon[30898]: 4.7 scrub starts 2024-04-25T05:43:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-mon[30898]: 4.7 scrub ok 2024-04-25T05:43:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:43:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:23 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:43:23.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:23.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:23.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:23.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:23.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:23.794 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:23.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:23.935 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:23.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:24.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:43:24.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:24.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:24.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 19857 19841 137.766 118.188 0.00894888 0.0072516 2024-04-25T05:43:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:24.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:24.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:24.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:24.393 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:24.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:24.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:24 smithi060 ceph-mon[30898]: 3.6 scrub starts 2024-04-25T05:43:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:24 smithi060 ceph-mon[30898]: 3.6 scrub ok 2024-04-25T05:43:24.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:24.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[27010]: 3.6 scrub starts 2024-04-25T05:43:24.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[27010]: 3.6 scrub ok 2024-04-25T05:43:24.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:24.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[37492]: 3.6 scrub starts 2024-04-25T05:43:24.538 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:24 smithi043 ceph-mon[37492]: 3.6 scrub ok 2024-04-25T05:43:24.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:43:24.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:24.614 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:24.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:24.759 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:24.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:24.886 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:24.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:24.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:25.008 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:25.008 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:25.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:25.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:25.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:25.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 21786 21770 136.045 120.562 0.00545511 0.00734403 2024-04-25T05:43:25.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:25.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:25.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:25.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.581 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:25.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: pgmap v497: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 188 MiB/s wr, 3.02k op/s; 11 KiB/s, 11 objects/s recovering 2024-04-25T05:43:25.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:43:25.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: pgmap v497: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 188 MiB/s wr, 3.02k op/s; 11 KiB/s, 11 objects/s recovering 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:43:25.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:43:25.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:43:25.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:43:25.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:25 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:43:25.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: pgmap v497: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 188 MiB/s wr, 3.02k op/s; 11 KiB/s, 11 objects/s recovering 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:43:25.698 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:43:25.699 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:25 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:43:25.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:25.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:43:25.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.850 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:25.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:25.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:25.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:25.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:26.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:26.108 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:26.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:26.199 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 23738 23722 134.767 122 0.00874905 0.00741229 2024-04-25T05:43:26.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.258 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:26.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:26.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:26.384 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:26.420 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:26.468 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:26.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:26.510 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:26.511 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:26.552 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:26.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:26.611 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:26.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:26.642 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:26.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:26.681 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:26.686 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:26.686 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:26.687 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:26.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:26.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:26.876 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:43:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6421 8.5 8.f1be6a5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.59720083299999, 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.497201589, 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.877 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6421 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:43:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277267+0000", 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1319999999999999e-06 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277387+0000", 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011962 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277399+0000", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2403e-05 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277406+0000", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7109999999999998e-06 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277467+0000", 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1427000000000007e-05 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277479+0000", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1579999999999999e-05 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.774455+0000", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.496976258 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.774468+0000", 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2458e-05 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6434 8.9 8.bb656db9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.59414308500001, 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.49508851199999998, 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6434 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280325+0000", 2024-04-25T05:43:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0580000000000001e-06 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280387+0000", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2152999999999999e-05 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280404+0000", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.658e-05 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280412+0000", 2024-04-25T05:43:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3950000000000001e-06 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280479+0000", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6492e-05 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280487+0000", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7269999999999996e-06 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775403+0000", 2024-04-25T05:43:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.494915463 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775412+0000", 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6439999999999995e-06 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6438 8.9 8.e7f58319 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:43:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.58786479099999, 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48884452099999998, 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6438 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286603+0000", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.139e-06 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286692+0000", 2024-04-25T05:43:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8694000000000002e-05 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286705+0000", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3135999999999999e-05 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286714+0000", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7439999999999996e-06 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286763+0000", 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8854999999999999e-05 2024-04-25T05:43:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286768+0000", 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3709999999999999e-06 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775438+0000", 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.488669992 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775447+0000", 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5900000000000008e-06 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6446 8.6 8.62fadf16 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:43:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.585146536, 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48884683800000001, 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6446 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289321+0000", 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5300000000000004e-07 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289390+0000", 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8848000000000002e-05 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289400+0000", 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0473e-05 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289406+0000", 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8510000000000004e-06 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289473+0000", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6346000000000005e-05 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289491+0000", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8439000000000001e-05 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778157+0000", 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48866554099999998 2024-04-25T05:43:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778167+0000", 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0686999999999999e-05 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6450 8.9 8.8eb80129 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:43:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.582619467, 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48362897700000002, 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6450 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4499999999999996e-07 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291910+0000", 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1204999999999994e-05 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291920+0000", 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0862e-05 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291927+0000", 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5629999999999997e-06 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291992+0000", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4504000000000004e-05 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291999+0000", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8159999999999994e-06 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775469+0000", 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48346932799999998 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775477+0000", 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8539999999999997e-06 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6453 8.6 8.63dba756 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.581408694, 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48515036299999997, 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6453 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293059+0000", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5899999999999995e-07 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293122+0000", 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2519000000000002e-05 2024-04-25T05:43:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293132+0000", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0234000000000001e-05 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293138+0000", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293189+0000", 2024-04-25T05:43:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1654999999999999e-05 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293198+0000", 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5429999999999994e-06 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778200+0000", 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48500220399999999 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778209+0000", 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5900000000000008e-06 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6455 8.9 8.8d6e43a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 237.580224001, 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48138075499999999, 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6455 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294244+0000", 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.99e-07 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294314+0000", 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0374000000000001e-05 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294326+0000", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2265e-05 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294336+0000", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4110000000000002e-06 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294382+0000", 2024-04-25T05:43:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5765e-05 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294386+0000", 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6709999999999998e-06 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775604+0000", 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48121744700000002 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775624+0000", 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0123e-05 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124108 8.9 8:9546d480:::benchmark_data_smithi043_104001_object124107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 149.76337485400001, 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16467942699999999, 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124108 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:43:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111094+0000", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4950000000000001e-06 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111129+0000", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5827000000000001e-05 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111141+0000", 2024-04-25T05:43:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1758e-05 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111150+0000", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0480000000000004e-06 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111533+0000", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038226200000000002 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111676+0000", 2024-04-25T05:43:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014314800000000001 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111863+0000", 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018720400000000001 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116324+0000", 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044614750000000003 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116341+0000", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6549999999999999e-05 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.118495+0000", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021544089999999999 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275636+0000", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15714107499999999 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275702+0000", 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5479000000000001e-05 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275772+0000", 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9696999999999994e-05 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125653 8.9 8:998fdd37:::benchmark_data_smithi043_104001_object1529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:43:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 146.59564004399999, 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11526359899999999, 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125653 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:43:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278828+0000", 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2589999999999999e-06 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278834+0000", 2024-04-25T05:43:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1149999999999999e-06 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278847+0000", 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2585000000000001e-05 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278855+0000", 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0150000000000003e-06 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280624+0000", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001768635 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280741+0000", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117799 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280865+0000", 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000123301 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287412+0000", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006547766 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287422+0000", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4779999999999997e-06 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294464+0000", 2024-04-25T05:43:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070417780000000003 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.393906+0000", 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099442738000000003 2024-04-25T05:43:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.393962+0000", 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5572000000000003e-05 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.394091+0000", 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128558 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125666 8.9 8:9242a807:::benchmark_data_smithi043_104001_object1542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 146.586996333, 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.112053468, 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125666 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:43:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287474+0000", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3299999999999999e-06 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287480+0000", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2430000000000002e-06 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287492+0000", 2024-04-25T05:43:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2150999999999999e-05 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287510+0000", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7833999999999999e-05 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287630+0000", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119732 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287867+0000", 2024-04-25T05:43:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000237235 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288022+0000", 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015471199999999999 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291108+0000", 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003086082 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294515+0000", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034066919999999998 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296045+0000", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015305239999999999 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399316+0000", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.103270836 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399392+0000", 2024-04-25T05:43:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5893e-05 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399524+0000", 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132204 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125667 8.9 8:928428f8:::benchmark_data_smithi043_104001_object1543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 146.58582438299999, 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.111196555, 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125667 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288644+0000", 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6899999999999996e-07 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288660+0000", 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6252e-05 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288677+0000", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7002000000000001e-05 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288686+0000", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8459999999999997e-06 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291154+0000", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024682900000000002 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291315+0000", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016116399999999999 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291477+0000", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016224300000000001 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294566+0000", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030882739999999998 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296052+0000", 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001486076 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296138+0000", 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6753000000000001e-05 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399659+0000", 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10352016999999999 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399721+0000", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2592e-05 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399839+0000", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117924 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125909 8.9 8:94a2b73d:::benchmark_data_smithi043_104001_object1785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:43:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 146.26931009500001, 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115848672, 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125909 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1399999999999997e-07 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605162+0000", 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7859999999999996e-06 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605168+0000", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1430000000000003e-06 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605174+0000", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606391+0000", 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001217079 2024-04-25T05:43:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606512+0000", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012084899999999999 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606639+0000", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012696499999999999 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.611089+0000", 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044506930000000004 2024-04-25T05:43:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.611123+0000", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3827e-05 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.612135+0000", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001011458 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.720612+0000", 2024-04-25T05:43:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.108477643 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.720676+0000", 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3501000000000001e-05 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.721006+0000", 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00032973799999999999 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129544 8.9 8:983c06aa:::benchmark_data_smithi043_104001_object5420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 141.86099478, 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13394251400000001, 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129544 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013473+0000", 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.249e-06 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013477+0000", 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7290000000000002e-06 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013485+0000", 2024-04-25T05:43:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8800000000000008e-06 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013491+0000", 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0599999999999996e-06 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013565+0000", 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4268000000000001e-05 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013690+0000", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012419699999999999 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013817+0000", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127875 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.016786+0000", 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029689080000000001 2024-04-25T05:43:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.017604+0000", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00081734600000000004 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018660+0000", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010562309999999999 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147216+0000", 2024-04-25T05:43:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.128556437 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147282+0000", 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5395000000000007e-05 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147415+0000", 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132939 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129548 8.9 8:90c26391:::benchmark_data_smithi043_104001_object5424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 141.85576604100001, 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12964294300000001, 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129548 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018702+0000", 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2160000000000001e-06 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018711+0000", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4090000000000003e-06 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018722+0000", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1080000000000001e-05 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018729+0000", 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4460000000000004e-06 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.021688+0000", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029587459999999999 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.021861+0000", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000173306 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022027+0000", 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016550899999999999 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025055+0000", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030286250000000001 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025839+0000", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00078327500000000005 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025862+0000", 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3841000000000001e-05 2024-04-25T05:43:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148136+0000", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122273643 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148203+0000", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7297000000000003e-05 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148344+0000", 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014055 2024-04-25T05:43:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPushReply(8.9 246/233 [PushReplyOp(8:97a75abd:::benchmark_data_smithi043_104001_object42380:head)])", 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 141.84800943400001, 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11376488899999999, 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:43:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.48e-07 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026461+0000", 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3280000000000002e-06 2024-04-25T05:43:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026468+0000", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2470000000000002e-06 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026476+0000", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8369999999999997e-06 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026525+0000", 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8912000000000002e-05 2024-04-25T05:43:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.140222+0000", 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11369731700000001 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:149154 8.9 8:9744f1fd:::benchmark_data_smithi043_104001_object25030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 121.79317080600001, 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11038581, 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 149154 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081298+0000", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8810000000000001e-06 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081305+0000", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9020000000000002e-06 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081327+0000", 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1769000000000001e-05 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081334+0000", 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6939999999999995e-06 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081396+0000", 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1356e-05 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081519+0000", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012302799999999999 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081645+0000", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012661399999999999 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.084389+0000", 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002743161 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.087947+0000", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003558748 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.089766+0000", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001818587 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191451+0000", 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.101684582 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191552+0000", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101734 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191682+0000", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012975399999999999 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226342 8.5 8:a7a71f38:::benchmark_data_smithi043_104001_object102218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:43:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 47.000836858, 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.208876007, 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226342 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:43:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873632+0000", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.784e-06 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873639+0000", 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1029999999999996e-06 2024-04-25T05:43:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873654+0000", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5387999999999999e-05 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873663+0000", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9679999999999995e-06 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873759+0000", 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6012000000000002e-05 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873897+0000", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000137479 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.874034+0000", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013683900000000001 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881894+0000", 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0078598769999999995 2024-04-25T05:43:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881921+0000", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6939000000000001e-05 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082491+0000", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.20057040500000001 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082506+0000", 2024-04-25T05:43:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5213e-05 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226342 8.5 8:a7a71f38:::benchmark_data_smithi043_104001_object102218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.980900327999997, 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1555693469999999, 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226342 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893569+0000", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2000000000000001e-06 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893577+0000", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5590000000000006e-06 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893597+0000", 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9826e-05 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893608+0000", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1221e-05 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893676+0000", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7723000000000002e-05 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893681+0000", 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6539999999999998e-06 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082813+0000", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.189132576 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082823+0000", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6500000000000008e-06 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015505+0000", 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.93268209199999996 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015634+0000", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012852800000000001 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048918+0000", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033284728999999999 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049136+0000", 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021758899999999999 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:226352 8.7 e282/209)", 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.967424233000003, 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.207367144, 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:43:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.34e-07 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907069+0000", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5814999999999999e-05 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907079+0000", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0471e-05 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907099+0000", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9745000000000002e-05 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.114390+0000", 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.207291169 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.114410+0000", 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.961e-05 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226356 8.5 8:a631803d:::benchmark_data_smithi043_104001_object102232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 46.744766276999997, 2024-04-25T05:43:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92008493099999999, 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226356 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129703+0000", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6369999999999999e-06 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129718+0000", 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4413000000000001e-05 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129742+0000", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.472e-05 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129757+0000", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4751e-05 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130011+0000", 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025401500000000001 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130048+0000", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6837e-05 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016167+0000", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88611901299999996 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016334+0000", 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016642 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016642+0000", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00030824900000000002 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048561+0000", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.031919212000000002 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049494+0000", 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093273599999999998 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049612+0000", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118149 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049689+0000", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7195999999999997e-05 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049786+0000", 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6582999999999993e-05 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:26.991 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:26.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-04-25T05:43:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:26.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:26.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:27.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:27.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:27.200 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 25340 25324 131.87 100.125 0.00833853 0.00757433 2024-04-25T05:43:27.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:27.263 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: pgmap v498: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 535 B/s rd, 138 MiB/s wr, 2.21k op/s; 8.4 KiB/s, 5 objects/s recovering 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/826635561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: pgmap v498: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 535 B/s rd, 138 MiB/s wr, 2.21k op/s; 8.4 KiB/s, 5 objects/s recovering 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:43:27.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/826635561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:27.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:43:27.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:43:27.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:43:27.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:27 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:43:27.268 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:27.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:27.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:27.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:27.542 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:27.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:27.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: pgmap v498: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 535 B/s rd, 138 MiB/s wr, 2.21k op/s; 8.4 KiB/s, 5 objects/s recovering 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/826635561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:43:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:27 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:43:27.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:27.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:27.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:27.945 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:27.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:28.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:28.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:28.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:28.202 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 26991 26975 129.663 103.188 0.0120953 0.00770619 2024-04-25T05:43:28.225 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:28.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:28.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:28.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:28.496 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:28.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:28.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:43:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:43:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[27010]: 6.b scrub starts 2024-04-25T05:43:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[27010]: 6.b scrub ok 2024-04-25T05:43:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:43:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:43:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[37492]: 6.b scrub starts 2024-04-25T05:43:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:28 smithi043 ceph-mon[37492]: 6.b scrub ok 2024-04-25T05:43:28.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:28 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:43:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:28 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:43:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:28 smithi060 ceph-mon[30898]: 6.b scrub starts 2024-04-25T05:43:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:28 smithi060 ceph-mon[30898]: 6.b scrub ok 2024-04-25T05:43:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:28.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:28.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:28.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:29.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:29.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:29.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:29.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 28699 28683 128.026 106.75 0.0054465 0.00780407 2024-04-25T05:43:29.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:29.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:29.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:29.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:29.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:29.620 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:29 smithi060 ceph-mon[30898]: pgmap v499: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-25T05:43:29.620 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:29 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:29.620 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:29 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:29.620 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:29 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:43:29.620 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:29 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:43:29.635 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[27010]: pgmap v499: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-25T05:43:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:43:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:43:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[37492]: pgmap v499: 158 pgs: 158 active+clean; 1.1 GiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-04-25T05:43:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:43:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:43:29.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:43:29.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:29.727 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:29.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:29.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:29.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:29.995 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:29.995 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:29.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:30.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:30.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:43:30.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 30383 30367 126.507 105.25 0.0041343 0.00789768 2024-04-25T05:43:30.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:30.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:30.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:30.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:30.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: osdmap e307: 8 total, 8 up, 6 in 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 6.5 deep-scrub starts 2024-04-25T05:43:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:30 smithi060 ceph-mon[30898]: 6.5 deep-scrub ok 2024-04-25T05:43:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:30.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:30.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: osdmap e307: 8 total, 8 up, 6 in 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:43:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 6.5 deep-scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[27010]: 6.5 deep-scrub ok 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: osdmap e307: 8 total, 8 up, 6 in 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 6.5 deep-scrub starts 2024-04-25T05:43:30.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:30 smithi043 ceph-mon[37492]: 6.5 deep-scrub ok 2024-04-25T05:43:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:30.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:30.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:30.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:30.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:30.921 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:30.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:31.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:31.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:31895 9.d 9:b4a24096:::benchmark_data_smithi043_129820_object31894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:31.127594+0000", 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0038359470000000001, 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038611510000000002, 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:43:31.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 31895 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127594+0000", 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127594+0000", 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127595+0000", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1710000000000001e-06 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127643+0000", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8313999999999999e-05 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127654+0000", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0712e-05 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:31.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127660+0000", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7529999999999998e-06 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127708+0000", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8949000000000002e-05 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127814+0000", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105808 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:43:31.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.127912+0000", 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8136999999999999e-05 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.129296+0000", 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001383392 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:31.129583+0000", 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028709200000000002 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:31.136 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:43:31.137 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 1 2024-04-25T05:43:31.137 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:31.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.187 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:31.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:31.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 31974 31958 124.815 99.4375 0.0068898 0.00798798 2024-04-25T05:43:31.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:31.296 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:43:31.315 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:31.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:31.466 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:31.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:31.600 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: pgmap v501: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-25T05:43:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: osdmap e308: 8 total, 8 up, 6 in 2024-04-25T05:43:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: pgmap v501: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: osdmap e308: 8 total, 8 up, 6 in 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:43:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:31 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:43:31.694 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: pgmap v501: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-04-25T05:43:31.694 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: osdmap e308: 8 total, 8 up, 6 in 2024-04-25T05:43:31.694 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:43:31.695 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:43:31.695 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:43:31.695 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:31 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:43:31.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:31.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:31.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:31.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:31.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:31.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:32.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:32.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:43:32.202 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 33523 33507 123.167 96.8125 0.00234062 0.00810258 2024-04-25T05:43:32.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:32.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:32.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:32 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:43:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:32 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:43:32.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:32.532 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:32.554 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:32.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:32 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:43:32.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:32 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:43:32.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:32 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:43:32.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:32 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:43:32.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:32.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:32.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:32.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:32.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:32.960 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:32.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:33.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:33.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:33.103 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:33.103 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:43:33.103 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:43:33.103 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:43:33.103 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:33.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:33.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 34936 34920 121.23 88.3125 0.00329399 0.0082421 2024-04-25T05:43:33.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:33.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:33.356 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:33.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-mon[30898]: pgmap v503: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-25T05:43:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-mon[30898]: 4.1d scrub starts 2024-04-25T05:43:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-mon[30898]: 4.1d scrub ok 2024-04-25T05:43:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:33.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:33.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:33.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[27010]: pgmap v503: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-25T05:43:33.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[27010]: 4.1d scrub starts 2024-04-25T05:43:33.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[27010]: 4.1d scrub ok 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[37492]: pgmap v503: 158 pgs: 158 active+clean; 1.7 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[37492]: 4.1d scrub starts 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[37492]: 4.1d scrub ok 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:33.633 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:33.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:33.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:33.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:33.788 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:33.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:33.930 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:33.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:33.961 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:34.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:34.078 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:34.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:43:34.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:34.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 36534 36518 120.106 99.875 0.00594287 0.00831965 2024-04-25T05:43:34.210 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:34.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:34.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:34.357 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:43:34.442 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:43:34.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.485 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:34.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:34.574 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:34.575 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-04-25T05:43:34.575 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3740758441' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.575 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.575 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:43:34.575 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:34 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:43:34.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-04-25T05:43:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3740758441' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3740758441' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:43:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:34 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:43:34.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:34.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:34.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:34.873 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:34.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:34.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:34.962 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:34.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:35.006 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:35.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:35.049 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:35.058 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:43:35.058 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:130500 7.5 7:a4f80177:::benchmark_data_smithi043_74356_object13631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 390.92833303999998, 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10185490899999999, 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.059 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 130500 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127203+0000", 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127204+0000", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4050000000000001e-06 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127211+0000", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6930000000000001e-06 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127225+0000", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3815000000000001e-05 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127234+0000", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1719999999999996e-06 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127294+0000", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9904000000000001e-05 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127421+0000", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012758499999999999 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:43:35.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.127543+0000", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012183199999999999 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.129963+0000", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024198269999999998 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.130082+0000", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.228908+0000", 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098825536000000005 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229009+0000", 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101307 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:04.229058+0000", 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8513000000000001e-05 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131177 7.5 7:a3c24791:::benchmark_data_smithi043_74356_object14308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 389.99810878, 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1201996, 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.065 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131177 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057427+0000", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057428+0000", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5799999999999998e-07 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057435+0000", 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3420000000000002e-06 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057451+0000", 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6067999999999999e-05 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.057460+0000", 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8990000000000007e-06 2024-04-25T05:43:35.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058761+0000", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013012029999999999 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.058890+0000", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012905600000000001 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.059004+0000", 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011395699999999999 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.061474+0000", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024696000000000002 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.063725+0000", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022509140000000001 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177555+0000", 2024-04-25T05:43:35.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11383001700000001 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177596+0000", 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0970000000000002e-05 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.177627+0000", 2024-04-25T05:43:35.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0815999999999998e-05 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131521 7.5 7:a454f61c:::benchmark_data_smithi043_74356_object14652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:43:35.072 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 389.48247455199999, 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14413125099999999, 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.073 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131521 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573061+0000", 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573063+0000", 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.449e-06 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573069+0000", 2024-04-25T05:43:35.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1099999999999999e-06 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573082+0000", 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3181e-05 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573089+0000", 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2300000000000002e-06 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573149+0000", 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9386999999999999e-05 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573243+0000", 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4758000000000003e-05 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.573328+0000", 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4320999999999994e-05 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.575369+0000", 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00204163 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.577122+0000", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017531059999999999 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717083+0000", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13996059199999999 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717144+0000", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1328000000000006e-05 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717193+0000", 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8158999999999997e-05 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:131530 7.5 7:a6cf53ae:::benchmark_data_smithi043_74356_object14661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 389.44086551800001, 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10267322399999999, 2024-04-25T05:43:35.082 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 131530 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614670+0000", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614672+0000", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2419999999999999e-06 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614677+0000", 2024-04-25T05:43:35.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5010000000000004e-06 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614688+0000", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1085e-05 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614697+0000", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4749999999999993e-06 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614749+0000", 2024-04-25T05:43:35.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2003000000000003e-05 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.614882+0000", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013371999999999999 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,2", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.615037+0000", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015419300000000001 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.617721+0000", 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026844970000000001 2024-04-25T05:43:35.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.621195+0000", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034735780000000002 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717252+0000", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096057038999999997 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717303+0000", 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1363000000000003e-05 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:05.717344+0000", 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0528000000000002e-05 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:188543 7.5 7:ac5985ff:::benchmark_data_smithi043_74356_object71674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 313.73720425499999, 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10455339800000001, 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 188543 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318332+0000", 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318333+0000", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3009999999999999e-06 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318338+0000", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9389999999999996e-06 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318349+0000", 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1216999999999999e-05 2024-04-25T05:43:35.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318357+0000", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6359999999999994e-06 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318398+0000", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1177999999999998e-05 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318596+0000", 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019781900000000001 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.318760+0000", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000164198 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.321782+0000", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003021714 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422025+0000", 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10024319900000001 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422069+0000", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3764999999999999e-05 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422680+0000", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061150799999999997 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422770+0000", 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0245000000000001e-05 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:21.422885+0000", 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114679 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196430 7.6 7:67a0b441:::benchmark_data_smithi043_74356_object79561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 303.15704353699999, 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17769133400000001, 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.094 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196430 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898492+0000", 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898496+0000", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5209999999999998e-06 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898500+0000", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9099999999999998e-06 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898509+0000", 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0329999999999997e-06 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898514+0000", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5659999999999997e-06 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898569+0000", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4317999999999997e-05 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898692+0000", 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012370400000000001 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.898805+0000", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011234300000000001 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900717+0000", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001912569 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.900846+0000", 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128825 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076080+0000", 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17523397700000001 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076138+0000", 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7815000000000001e-05 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.076184+0000", 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5753000000000001e-05 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196431 7.6 7:64208f17:::benchmark_data_smithi043_74356_object79562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 303.15402868500001, 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17835189000000001, 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196431 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901507+0000", 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901509+0000", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7439999999999999e-06 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901517+0000", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8520000000000004e-06 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901526+0000", 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4679999999999998e-06 2024-04-25T05:43:35.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901534+0000", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8720000000000002e-06 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901588+0000", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3638999999999999e-05 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901700+0000", 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112355 2024-04-25T05:43:35.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901816+0000", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116359 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.903841+0000", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020240940000000002 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907001+0000", 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031600640000000002 2024-04-25T05:43:35.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079789+0000", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17278797700000001 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079821+0000", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2480000000000001e-05 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079859+0000", 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7985999999999998e-05 2024-04-25T05:43:35.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196437 7.5 7:a3615d17:::benchmark_data_smithi043_74356_object79568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 303.15098933000002, 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16688376599999999, 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196437 2024-04-25T05:43:35.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904546+0000", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904548+0000", 2024-04-25T05:43:35.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904553+0000", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2429999999999996e-06 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904563+0000", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0046e-05 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904571+0000", 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0169999999999996e-06 2024-04-25T05:43:35.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904710+0000", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013878799999999999 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904838+0000", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128014 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,2", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904935+0000", 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7627999999999994e-05 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.906988+0000", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020523189999999999 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908135+0000", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00114685 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.908341+0000", 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020669199999999999 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071248+0000", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.162907146 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071322+0000", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3274000000000004e-05 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.071430+0000", 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010847400000000001 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196438 7.6 7:6894f35c:::benchmark_data_smithi043_74356_object79569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 303.15088460200002, 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17980945200000001, 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196438 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904651+0000", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904652+0000", 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2099999999999996e-07 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904655+0000", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4249999999999998e-06 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904666+0000", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1185999999999999e-05 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904673+0000", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8009999999999998e-06 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904726+0000", 2024-04-25T05:43:35.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2479999999999999e-05 2024-04-25T05:43:35.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.116 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:35.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904836+0000", 2024-04-25T05:43:35.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110016 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904947+0000", 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111721 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907188+0000", 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022400810000000001 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907463+0000", 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027584899999999999 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084360+0000", 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17689687000000001 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084421+0000", 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1032000000000001e-05 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.084461+0000", 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9369999999999997e-05 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198457 7.5 7:aeb56927:::benchmark_data_smithi043_74356_object81588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 299.25134033, 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97568060099999998, 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.119 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198457 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804195+0000", 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804197+0000", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.544e-06 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804204+0000", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7569999999999998e-06 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804216+0000", 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1841000000000001e-05 2024-04-25T05:43:35.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804224+0000", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7099999999999996e-06 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804278+0000", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3275000000000001e-05 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804284+0000", 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6030000000000001e-06 2024-04-25T05:43:35.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773320+0000", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96903589899999998 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773471+0000", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015128900000000001 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.773614+0000", 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014288699999999999 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777300+0000", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036858519999999999 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778657+0000", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013569280000000001 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779814+0000", 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001156481 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779846+0000", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2005999999999998e-05 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779876+0000", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0528999999999999e-05 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198459 7.5 7:a46444d1:::benchmark_data_smithi043_74356_object81590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:43:35.125 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 299.25099723400001, 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.977629847, 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198459 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804539+0000", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3499999999999996e-07 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804544+0000", 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3390000000000003e-06 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804553+0000", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.969e-06 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804559+0000", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.55e-06 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804600+0000", 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1149999999999997e-05 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.804606+0000", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4179999999999996e-06 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775688+0000", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97108227300000005 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775807+0000", 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011906899999999999 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.775929+0000", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012165199999999999 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779949+0000", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040201059999999999 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779988+0000", 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8943000000000003e-05 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782076+0000", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020885729999999998 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782128+0000", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2080999999999998e-05 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782168+0000", 2024-04-25T05:43:35.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0089000000000002e-05 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198466 7.5 7:a9a716c6:::benchmark_data_smithi043_74356_object81597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 299.24607780600002, 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97804308600000001, 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198466 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809458+0000", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809459+0000", 2024-04-25T05:43:35.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4529999999999999e-06 2024-04-25T05:43:35.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809465+0000", 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8669999999999999e-06 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809479+0000", 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.324e-05 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809487+0000", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2020000000000004e-06 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809543+0000", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6376999999999999e-05 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.809555+0000", 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1435e-05 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777347+0000", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96779198399999999 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777465+0000", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011815999999999999 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.777591+0000", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012601299999999999 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779953+0000", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023620949999999998 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782689+0000", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027361059999999999 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787431+0000", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047419230000000003 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787467+0000", 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6081000000000002e-05 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787501+0000", 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4150000000000003e-05 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:198479 7.5 7:a8eb3f00:::benchmark_data_smithi043_74356_object81610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 299.23238285799999, 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96450478500000003, 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.143 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 198479 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823153+0000", 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823155+0000", 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.717e-06 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823172+0000", 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7611000000000001e-05 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823191+0000", 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8842e-05 2024-04-25T05:43:35.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823202+0000", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0407e-05 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823258+0000", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6428999999999998e-05 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:35.823265+0000", 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4699999999999996e-06 2024-04-25T05:43:35.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778700+0000", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95543422700000002 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778769+0000", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9831000000000003e-05 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.778858+0000", 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8744000000000003e-05 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.779956+0000", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010981400000000001 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.782730+0000", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002773546 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787552+0000", 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048221590000000003 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787612+0000", 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9880999999999998e-05 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:36.787658+0000", 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5781000000000001e-05 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228018 8.17 8.878e0737 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.079613924999997, 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0050960120000001, 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228018 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:43:35.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975924+0000", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.711e-06 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975930+0000", 2024-04-25T05:43:35.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1469999999999998e-06 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975947+0000", 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7601999999999999e-05 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975956+0000", 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9400000000000008e-06 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976889+0000", 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093282800000000002 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976898+0000", 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8769999999999999e-06 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980999+0000", 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.00410116 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981018+0000", 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8746999999999999e-05 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228040 8.17 8.3c35da57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.062159635, 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98798208499999995, 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.157 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228040 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993377+0000", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.012e-06 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993383+0000", 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9769999999999997e-06 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993394+0000", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0370999999999999e-05 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993402+0000", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1729999999999995e-06 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993458+0000", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6025e-05 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993465+0000", 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2339999999999997e-06 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981341+0000", 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98787640899999996 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981358+0000", 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6884e-05 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228041 8.17 8.7bebe17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 53.062004326999997, 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98792126800000002, 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228041 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993532+0000", 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8400000000000005e-07 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993535+0000", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6070000000000002e-06 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993542+0000", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9619999999999996e-06 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993548+0000", 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8190000000000003e-06 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993585+0000", 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6328000000000002e-05 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993588+0000", 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4319999999999999e-06 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981443+0000", 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98785475499999997 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981453+0000", 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8810000000000008e-06 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.166 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:6 9.7 9:e9b4c7c6:::benchmark_data_smithi043_129820_object5:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.854986449999998, 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.006128809, 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:43:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200550+0000", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3300000000000002e-07 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200583+0000", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2659e-05 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200590+0000", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9890000000000003e-06 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200595+0000", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7899999999999996e-06 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200646+0000", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0822000000000002e-05 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200656+0000", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5109999999999992e-06 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192142+0000", 2024-04-25T05:43:35.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99148672699999996 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192281+0000", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139018 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192427+0000", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014579999999999999 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.194168+0000", 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017404429999999999 2024-04-25T05:43:35.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205031+0000", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010862966 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206577+0000", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015465780000000001 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206635+0000", 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7998000000000003e-05 2024-04-25T05:43:35.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206678+0000", 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2975000000000001e-05 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:12 9.d 9:b543e130:::benchmark_data_smithi043_129820_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.854832699999999, 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0180756520000001, 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.174 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:43:35.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200704+0000", 2024-04-25T05:43:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2099999999999997e-07 2024-04-25T05:43:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212291+0000", 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011587573 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212312+0000", 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0627000000000001e-05 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212324+0000", 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.185e-05 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212430+0000", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010610199999999999 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212444+0000", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4188999999999999e-05 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205046+0000", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99260219400000005 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205178+0000", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013207299999999999 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205330+0000", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000151438 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.209820+0000", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044903390000000003 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210175+0000", 2024-04-25T05:43:35.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00035483800000000001 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218678+0000", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0085029509999999999 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218737+0000", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9036000000000002e-05 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218779+0000", 2024-04-25T05:43:35.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1921000000000001e-05 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:20 9.d 9:b555abe5:::benchmark_data_smithi043_129820_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.840795065999998, 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0041661829999999, 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:43:35.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 20 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2600000000000002e-07 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214807+0000", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5287e-05 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214816+0000", 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7630000000000006e-06 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214821+0000", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214856+0000", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5147000000000001e-05 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214861+0000", 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:43:35.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205852+0000", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99099027500000003 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206034+0000", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000182628 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206177+0000", 2024-04-25T05:43:35.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014306599999999999 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210096+0000", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039190930000000002 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210182+0000", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6057000000000006e-05 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218819+0000", 2024-04-25T05:43:35.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086365039999999997 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218868+0000", 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9302000000000003e-05 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218907+0000", 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8674999999999998e-05 2024-04-25T05:43:35.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:34 9.7 9:e321d650:::benchmark_data_smithi043_129820_object33:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.836695066000001, 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.988632811, 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:43:35.189 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 34 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:43:35.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4099999999999998e-07 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218902+0000", 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0786999999999999e-05 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:43:35.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218912+0000", 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4930000000000004e-06 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218920+0000", 2024-04-25T05:43:35.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1890000000000007e-06 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218967+0000", 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6733000000000001e-05 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218984+0000", 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7153999999999999e-05 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192758+0000", 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97377451299999995 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192893+0000", 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013458900000000001 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.193067+0000", 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017409899999999999 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.195263+0000", 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021959700000000002 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206451+0000", 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011187704 2024-04-25T05:43:35.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207381+0000", 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093041 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207433+0000", 2024-04-25T05:43:35.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2145000000000003e-05 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207474+0000", 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0284000000000003e-05 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:43:35.199 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:35.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:35.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:43:35.202564+0000 min lat: 0.00146388 max lat: 1.02736 avg lat: 0.00838346 2024-04-25T05:43:35.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:43:35.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 38166 38150 119.2 102 0.0112835 0.00838346 2024-04-25T05:43:35.227 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:35.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:35.357 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:35.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:35.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: pgmap v504: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: osdmap e309: 8 total, 8 up, 6 in 2024-04-25T05:43:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1671191446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:35.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:35.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: pgmap v504: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-04-25T05:43:35.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:35.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: osdmap e309: 8 total, 8 up, 6 in 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1671191446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: pgmap v504: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.13k op/s 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: osdmap e309: 8 total, 8 up, 6 in 2024-04-25T05:43:35.632 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1671191446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:35.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:35.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:35.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:35.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:35.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:36.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:36.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:36.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:36.201 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 39792 39776 118.363 101.625 0.0066511 0.0084426 2024-04-25T05:43:36.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:36.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:36.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:36.433 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:36.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:36.563 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:36.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[37492]: osdmap e310: 8 total, 8 up, 6 in 2024-04-25T05:43:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[27010]: osdmap e310: 8 total, 8 up, 6 in 2024-04-25T05:43:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:36 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:36.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.702 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:36.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:36 smithi060 ceph-mon[30898]: osdmap e310: 8 total, 8 up, 6 in 2024-04-25T05:43:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:36 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:36 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:36.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:36.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:36.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:36.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:36.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:37.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-25T05:43:37.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:37.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:43:37.203 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 41457 41441 117.711 104.062 0.0164445 0.008489 2024-04-25T05:43:37.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.243 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:37.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:37.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:37.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:37.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:37.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:37.562 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:37.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:37 smithi043 ceph-mon[37492]: pgmap v507: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:37 smithi043 ceph-mon[37492]: osdmap e311: 8 total, 8 up, 6 in 2024-04-25T05:43:37.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:37 smithi043 ceph-mon[27010]: pgmap v507: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:37.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:37 smithi043 ceph-mon[27010]: osdmap e311: 8 total, 8 up, 6 in 2024-04-25T05:43:37.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:37.685 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:43:37.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:37 smithi060 ceph-mon[30898]: pgmap v507: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:37 smithi060 ceph-mon[30898]: osdmap e311: 8 total, 8 up, 6 in 2024-04-25T05:43:37.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:37.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:37.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:38.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:38.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:38.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:38.196 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:38.202 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 42874 42858 116.443 88.5625 0.00636243 0.0085805 2024-04-25T05:43:38.208 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:38.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.349 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:38.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:38.386 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:38.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:38.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:43:38.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:38.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:38.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:38.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:38.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:38.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:38.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:38.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:39.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:39.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:39.158 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:39.202 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 44317 44301 115.348 90.1875 0.0109076 0.0086636 2024-04-25T05:43:39.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:39.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:39.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:39.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:39.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:39 smithi043 ceph-mon[27010]: pgmap v509: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:39.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:39 smithi043 ceph-mon[37492]: pgmap v509: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:39.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:39.724 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:39.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:39 smithi060 ceph-mon[30898]: pgmap v509: 158 pgs: 158 active+clean; 2.1 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:43:39.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:39.854 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:39.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:39.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:39.982 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:43:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.007 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:40.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:40.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-25T05:43:40.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:40.203 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 45845 45829 114.552 95.5 0.0142002 0.00872328 2024-04-25T05:43:40.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:40.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:40.383 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:40.409 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:40.417 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:43:40.417 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:43:40.434 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:40.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:40.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:40.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:40.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:40.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:40.846 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:40.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:40.977 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:40.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:41.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:41.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:40 smithi043 ceph-mon[27010]: pgmap v510: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:43:41.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:40 smithi043 ceph-mon[27010]: osdmap e312: 8 total, 8 up, 6 in 2024-04-25T05:43:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:40 smithi043 ceph-mon[37492]: pgmap v510: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:43:41.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:40 smithi043 ceph-mon[37492]: osdmap e312: 8 total, 8 up, 6 in 2024-04-25T05:43:41.203 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 47318 47302 113.687 92.0625 0.00725681 0.00879068 2024-04-25T05:43:41.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:41.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:40 smithi060 ceph-mon[30898]: pgmap v510: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s; 18 B/s, 0 objects/s recovering 2024-04-25T05:43:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:40 smithi060 ceph-mon[30898]: osdmap e312: 8 total, 8 up, 6 in 2024-04-25T05:43:41.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:41.378 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:41.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:41.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:41.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.633 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:41.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:41.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:41.770 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:41.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:43:41.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:41.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:42.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:42.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:42.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:42.203 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 48667 48651 112.599 84.3125 0.0237012 0.00887476 2024-04-25T05:43:42.223 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:42.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:42.351 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:42.454 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:42.493 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:42.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:42.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:42.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:42.784 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:42.855 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:42.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:43:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:42.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:42.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:42.918 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:42.977 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:42.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:42 smithi043 ceph-mon[27010]: pgmap v512: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 784 B/s rd, 76 MiB/s wr, 1.21k op/s; 16 B/s, 0 objects/s recovering 2024-04-25T05:43:42.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:42.977 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:42 smithi043 ceph-mon[37492]: pgmap v512: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 784 B/s rd, 76 MiB/s wr, 1.21k op/s; 16 B/s, 0 objects/s recovering 2024-04-25T05:43:42.978 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:43.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:43.022 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:43.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:43.066 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:43.112 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:43.153 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:43.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:43.194 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:43.203 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 49903 49887 111.333 77.25 0.0112559 0.00897544 2024-04-25T05:43:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:43.219 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:42 smithi060 ceph-mon[30898]: pgmap v512: 158 pgs: 158 active+clean; 2.6 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 784 B/s rd, 76 MiB/s wr, 1.21k op/s; 16 B/s, 0 objects/s recovering 2024-04-25T05:43:43.219 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:43:43.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:43.274 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:43.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:43.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:43.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:43.474 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:43.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:43.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:43.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:43.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:43.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:43.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:43.877 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:43.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:43.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:44.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[27010]: osdmap e313: 8 total, 8 up, 6 in 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2514668322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[37492]: osdmap e313: 8 total, 8 up, 6 in 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2514668322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:44.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:44.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 51406 51390 110.733 93.9375 0.00942695 0.00902537 2024-04-25T05:43:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2295773954' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:43:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-mon[30898]: osdmap e313: 8 total, 8 up, 6 in 2024-04-25T05:43:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2514668322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:44.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:44.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:44.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:44.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:44.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:44.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:44.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:44.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:44.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:44.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:44.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:45.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:45.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:45.184 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:43:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:43:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:43:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:43:45.185 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:43:45.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:44 smithi043 ceph-mon[27010]: pgmap v514: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 125 MiB/s wr, 1.99k op/s; 14 B/s, 0 objects/s recovering 2024-04-25T05:43:45.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:44 smithi043 ceph-mon[27010]: osdmap e314: 8 total, 8 up, 6 in 2024-04-25T05:43:45.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:44 smithi043 ceph-mon[37492]: pgmap v514: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 125 MiB/s wr, 1.99k op/s; 14 B/s, 0 objects/s recovering 2024-04-25T05:43:45.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:44 smithi043 ceph-mon[37492]: osdmap e314: 8 total, 8 up, 6 in 2024-04-25T05:43:45.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 52802 52786 109.95 87.25 0.00546191 0.0090891 2024-04-25T05:43:45.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:45.248 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:44 smithi060 ceph-mon[30898]: pgmap v514: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 125 MiB/s wr, 1.99k op/s; 14 B/s, 0 objects/s recovering 2024-04-25T05:43:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:44 smithi060 ceph-mon[30898]: osdmap e314: 8 total, 8 up, 6 in 2024-04-25T05:43:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:45.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:45.374 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:45.476 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:45.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:45.616 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:45.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:45.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:45.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:45.793 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:45.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:45.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:45.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:46.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:46.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:46.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:46.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:46.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 54235 54219 109.292 89.5625 0.00989506 0.00914384 2024-04-25T05:43:46.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:43:46.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:46.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:43:46.330 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:46.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:46.456 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:46.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:46.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:46.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.757 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:46.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:46.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:46.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:46.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:46.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:43:47.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:47.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:47.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:47.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 55444 55428 108.238 75.5625 0.0176766 0.00923332 2024-04-25T05:43:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:46 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:46 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:46 smithi060 ceph-mon[30898]: pgmap v516: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:46 smithi060 ceph-mon[30898]: osdmap e315: 8 total, 8 up, 6 in 2024-04-25T05:43:47.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:47.296 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[27010]: pgmap v516: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[27010]: osdmap e315: 8 total, 8 up, 6 in 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[37492]: pgmap v516: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:47.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:46 smithi043 ceph-mon[37492]: osdmap e315: 8 total, 8 up, 6 in 2024-04-25T05:43:47.324 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:47.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:47.455 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:47.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.579 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:47.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:47.680 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:47.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:47.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:47.852 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:47.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:47.981 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:47.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:48.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:48.114 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:48.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 56732 56716 107.398 80.5 0.0124447 0.00930566 2024-04-25T05:43:48.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:48.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:48.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.370 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:48.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:48.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:48.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:48.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:48.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:48.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:48.774 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:48.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:48.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:48.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:48.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:48 smithi060 ceph-mon[30898]: pgmap v518: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:49.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:49.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:49.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:49.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:49.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[27010]: pgmap v518: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:49.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:43:49.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:43:49.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:48 smithi043 ceph-mon[37492]: pgmap v518: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-25T05:43:49.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:49.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:49.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 57942 57926 106.463 75.625 0.0141697 0.00938795 2024-04-25T05:43:49.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:49.330 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:49.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:49.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:49.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:49.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:49.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:49.721 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:49.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:49.848 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:49.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:49.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:49.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:50.089 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:50.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:50.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:43:50.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 59274 59258 105.8 83.25 0.00422382 0.00944667 2024-04-25T05:43:50.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:50.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:50.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:50.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:50.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.512 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:50.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:50.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:50.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:50.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.777 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:50.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:50.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:50.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:50.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:51.017 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:51.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:50 smithi043 ceph-mon[27010]: pgmap v519: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-04-25T05:43:51.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:50 smithi043 ceph-mon[27010]: osdmap e316: 8 total, 8 up, 6 in 2024-04-25T05:43:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:50 smithi043 ceph-mon[37492]: pgmap v519: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-04-25T05:43:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:50 smithi043 ceph-mon[37492]: osdmap e316: 8 total, 8 up, 6 in 2024-04-25T05:43:51.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:51.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:51.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 60525 60509 105.033 78.1875 0.0083605 0.00951526 2024-04-25T05:43:51.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:50 smithi060 ceph-mon[30898]: pgmap v519: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-04-25T05:43:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:50 smithi060 ceph-mon[30898]: osdmap e316: 8 total, 8 up, 6 in 2024-04-25T05:43:51.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:51.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:51.399 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:51.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-04-25T05:43:51.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:51.481 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:51.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:51.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:51.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:51.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:51.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:51.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:51.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:52.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:52.052 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:52.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:52.198 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:52.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 61728 61712 104.225 75.1875 0.0109913 0.0095897 2024-04-25T05:43:52.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:52.395 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146292 7.4 7:27c079d3:::benchmark_data_smithi043_74356_object29423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.31280424699997, 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.830877023, 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:43:52.396 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146292 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078845+0000", 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.22e-06 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078850+0000", 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8199999999999996e-06 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078865+0000", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5109000000000001e-05 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078872+0000", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0040000000000002e-06 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079233+0000", 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036102900000000002 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079358+0000", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012519399999999999 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079469+0000", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110436 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087939+0000", 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0084700910000000008 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087953+0000", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4079e-05 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734514+0000", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.646560994 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734529+0000", 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4897e-05 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869085+0000", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1345559490000001 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869249+0000", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016453999999999999 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909625+0000", 2024-04-25T05:43:52.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.040376196000000003 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909721+0000", 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5464999999999995e-05 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146325 7.4 7:287eff09:::benchmark_data_smithi043_74356_object29456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.26926211300002, 2024-04-25T05:43:52.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.78676667, 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146325 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.404 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122388+0000", 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8509999999999999e-06 2024-04-25T05:43:52.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122394+0000", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4629999999999998e-06 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122407+0000", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2965e-05 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122416+0000", 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1439999999999992e-06 2024-04-25T05:43:52.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122487+0000", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0815000000000006e-05 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122500+0000", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2641e-05 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734558+0000", 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.61205790800000004 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734567+0000", 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869583+0000", 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.135015624 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869717+0000", 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013412499999999999 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869897+0000", 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018037399999999999 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876526+0000", 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066293460000000004 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882816+0000", 2024-04-25T05:43:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062898030000000001 2024-04-25T05:43:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889558+0000", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067413789999999996 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.908921+0000", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.01936336 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909004+0000", 2024-04-25T05:43:52.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3045000000000002e-05 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909153+0000", 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014846799999999999 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146329 7.4 7:2d613d15:::benchmark_data_smithi043_74356_object29460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.26668821099997, 2024-04-25T05:43:52.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.784987538, 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146329 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124962+0000", 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.835e-06 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124968+0000", 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5849999999999996e-06 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124986+0000", 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.823e-05 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124996+0000", 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3719999999999994e-06 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125091+0000", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5061000000000003e-05 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125102+0000", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.094e-05 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734591+0000", 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.60948939099999999 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734600+0000", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4509999999999992e-06 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872375+0000", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1377753349999999 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872499+0000", 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124499 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872641+0000", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141438 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876579+0000", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039384210000000001 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882926+0000", 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0063468170000000003 2024-04-25T05:43:52.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889565+0000", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066388469999999998 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909760+0000", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020195310000000001 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909819+0000", 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8798000000000002e-05 2024-04-25T05:43:52.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909947+0000", 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012820799999999999 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146354 7.4 7:2311a945:::benchmark_data_smithi043_74356_object29485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.226466878, 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.745096427, 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.420 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146354 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165182+0000", 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.32e-07 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165184+0000", 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5189999999999999e-06 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165189+0000", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3580000000000002e-06 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165195+0000", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.356e-06 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165284+0000", 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8756000000000002e-05 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165295+0000", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1338000000000001e-05 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734624+0000", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.56932913799999996 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734632+0000", 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4710000000000007e-06 2024-04-25T05:43:52.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876617+0000", 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.141984085 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876755+0000", 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138923 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876917+0000", 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016150299999999999 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882874+0000", 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059565750000000004 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883014+0000", 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013997799999999999 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889572+0000", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065581820000000001 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910091+0000", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.02051905 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910155+0000", 2024-04-25T05:43:52.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4060000000000007e-05 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910278+0000", 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012270299999999999 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146371 7.4 7:20dc29ab:::benchmark_data_smithi043_74356_object29502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:43:52.428 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.20841481100001, 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7272659080000001, 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146371 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183234+0000", 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.46e-07 2024-04-25T05:43:52.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183236+0000", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4040000000000002e-06 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183240+0000", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4000000000000002e-06 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183245+0000", 2024-04-25T05:43:52.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6750000000000001e-06 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183308+0000", 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2946000000000003e-05 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183320+0000", 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1895e-05 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734658+0000", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.55133849499999998 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734668+0000", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0145e-05 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879759+0000", 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.145090291 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879914+0000", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154835 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.880070+0000", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015658800000000001 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882969+0000", 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028987420000000002 2024-04-25T05:43:52.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889540+0000", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065711279999999999 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889579+0000", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8476999999999998e-05 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910326+0000", 2024-04-25T05:43:52.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020747422000000001 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910386+0000", 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0270999999999999e-05 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910499+0000", 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112848 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146375 7.4 7:20e96b3d:::benchmark_data_smithi043_74356_object29506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.20362987700003, 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7227024800000001, 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146375 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.437 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:43:52.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188019+0000", 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.333e-06 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188025+0000", 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7180000000000001e-06 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188038+0000", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3264999999999999e-05 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188047+0000", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0450000000000006e-06 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188122+0000", 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4140000000000005e-05 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188132+0000", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0737000000000001e-05 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734692+0000", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.54655990700000001 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734701+0000", 2024-04-25T05:43:52.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2749999999999995e-06 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883051+0000", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1483502480000001 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883195+0000", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014417400000000001 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883347+0000", 2024-04-25T05:43:52.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015209500000000001 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889584+0000", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062372699999999996 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889631+0000", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7012999999999999e-05 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889684+0000", 2024-04-25T05:43:52.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3078999999999998e-05 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910550+0000", 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020865748999999999 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910610+0000", 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9883999999999998e-05 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910721+0000", 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110548 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146378 7.4 7:2089bf57:::benchmark_data_smithi043_74356_object29509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.19495419700002, 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.714255568, 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146378 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196695+0000", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.491e-06 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196703+0000", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4789999999999999e-06 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196715+0000", 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1938e-05 2024-04-25T05:43:52.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196724+0000", 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8410000000000006e-06 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196789+0000", 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5946999999999997e-05 2024-04-25T05:43:52.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196801+0000", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1279e-05 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734725+0000", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53792390300000004 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734733+0000", 2024-04-25T05:43:52.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3599999999999996e-06 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886492+0000", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1517590280000001 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886663+0000", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000171094 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886838+0000", 2024-04-25T05:43:52.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017525099999999999 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889701+0000", 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028622180000000001 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889754+0000", 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3440000000000003e-05 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899237+0000", 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0094831459999999996 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910763+0000", 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011526290999999999 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910824+0000", 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0829999999999999e-05 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910949+0000", 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012503199999999999 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146380 7.4 7:2f11d6b2:::benchmark_data_smithi043_74356_object29511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.19422942599999, 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7323310589999998, 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146380 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6000000000000003e-07 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197424+0000", 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.369e-06 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197441+0000", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7419999999999999e-05 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197449+0000", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4910000000000005e-06 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197538+0000", 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8030999999999998e-05 2024-04-25T05:43:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197548+0000", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0334e-05 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734761+0000", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721291999999998 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734769+0000", 2024-04-25T05:43:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4290000000000001e-06 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889796+0000", 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1550264050000001 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890050+0000", 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025425699999999997 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890229+0000", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017900300000000001 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899288+0000", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090589869999999992 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899330+0000", 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2453999999999997e-05 2024-04-25T05:43:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911000+0000", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011669538 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929593+0000", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018593014000000001 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929639+0000", 2024-04-25T05:43:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6319999999999997e-05 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929750+0000", 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011042700000000001 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146381 7.4 7:2ef47468:::benchmark_data_smithi043_74356_object29512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.19417801600002, 2024-04-25T05:43:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.732786876, 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146381 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:43:52.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6800000000000001e-07 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197474+0000", 2024-04-25T05:43:52.464 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3900000000000002e-06 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197481+0000", 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7730000000000002e-06 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197487+0000", 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.421e-06 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.465 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197574+0000", 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7058000000000006e-05 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197582+0000", 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3489999999999992e-06 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734793+0000", 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721096300000004 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.466 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734802+0000", 2024-04-25T05:43:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5159999999999994e-06 2024-04-25T05:43:52.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895295+0000", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1604930549999999 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895455+0000", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015972800000000001 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895617+0000", 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016215500000000001 2024-04-25T05:43:52.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899375+0000", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037581239999999998 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909210+0000", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098352049999999996 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911053+0000", 2024-04-25T05:43:52.469 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018428959999999999 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930101+0000", 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019048197999999999 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930150+0000", 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8424000000000003e-05 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.470 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930257+0000", 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107153 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146407 7.4 7:23e67bf6:::benchmark_data_smithi043_74356_object29538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.15457042499997, 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.693338502, 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.471 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146407 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237079+0000", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.516e-06 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237089+0000", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6940000000000007e-06 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237100+0000", 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1114000000000001e-05 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237108+0000", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1040000000000007e-06 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237170+0000", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1851000000000001e-05 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237180+0000", 2024-04-25T05:43:52.474 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.063e-05 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734826+0000", 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.497645477 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734834+0000", 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1210000000000007e-06 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.475 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899407+0000", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.164572851 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899548+0000", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141142 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899702+0000", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015383899999999999 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909262+0000", 2024-04-25T05:43:52.476 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0095605629999999994 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909994+0000", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00073198100000000004 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926123+0000", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016128397999999999 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930265+0000", 2024-04-25T05:43:52.477 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041419300000000003 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930315+0000", 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0584000000000002e-05 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930416+0000", 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100707 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146411 7.4 7:2d6ecc61:::benchmark_data_smithi043_74356_object29542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.152319879, 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6912330820000001, 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146411 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.479 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239329+0000", 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2399999999999997e-07 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239333+0000", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5249999999999999e-06 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239343+0000", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7049999999999994e-06 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239351+0000", 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1499999999999999e-06 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239474+0000", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012281099999999999 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239484+0000", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9709999999999999e-06 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735030+0000", 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.49554585800000001 2024-04-25T05:43:52.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735043+0000", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3399e-05 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905732+0000", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.170688524 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905903+0000", 2024-04-25T05:43:52.483 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017154299999999999 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.906084+0000", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000180502 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910045+0000", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039608480000000003 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916884+0000", 2024-04-25T05:43:52.484 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0068390710000000004 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926215+0000", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.009331331 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930423+0000", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042081159999999996 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930465+0000", 2024-04-25T05:43:52.485 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1239000000000003e-05 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930561+0000", 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6664999999999997e-05 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146416 7.4 7:20f93e30:::benchmark_data_smithi043_74356_object29547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 388.14455194800001, 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6920498500000001, 2024-04-25T05:43:52.486 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146416 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.487 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247097+0000", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.268e-06 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247103+0000", 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6799999999999998e-06 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.488 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247113+0000", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5869999999999999e-06 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247120+0000", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8630000000000008e-06 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247187+0000", 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6701999999999999e-05 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247197+0000", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0013000000000001e-05 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736134+0000", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.48893706999999997 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736148+0000", 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3709999999999999e-05 2024-04-25T05:43:52.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911091+0000", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1749430030000001 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911228+0000", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013674 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911385+0000", 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015695600000000001 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916945+0000", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055600700000000003 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930568+0000", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013623273999999999 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.931318+0000", 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075026300000000004 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.935451+0000", 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041326820000000004 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939020+0000", 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035686450000000001 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939146+0000", 2024-04-25T05:43:52.494 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012632400000000001 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184730 8.c 8:3895024e:::benchmark_data_smithi043_104001_object60606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 106.489238227, 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0106298600000001, 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.495 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184730 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.496 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902411+0000", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.483e-06 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902418+0000", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3269999999999997e-06 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902437+0000", 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9673999999999998e-05 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.497 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902446+0000", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902550+0000", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104485 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902561+0000", 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0351e-05 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906055+0000", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.003494664 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906151+0000", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5391999999999997e-05 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906246+0000", 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5552999999999998e-05 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.499 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912838+0000", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006591675 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912925+0000", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7159999999999999e-05 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912941+0000", 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5968999999999999e-05 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.500 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912992+0000", 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0841000000000003e-05 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913040+0000", 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7842e-05 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184731 8.c 8:3f348624:::benchmark_data_smithi043_104001_object60607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:43:52.501 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 106.489184142, 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.010906069, 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184731 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.502 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3099999999999998e-07 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902468+0000", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.337e-06 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902473+0000", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5310000000000001e-06 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902479+0000", 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4700000000000001e-06 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.504 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902585+0000", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106054 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902591+0000", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3790000000000003e-06 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908540+0000", 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.00594837 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908663+0000", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123437 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908776+0000", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112724 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913122+0000", 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043464560000000003 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913170+0000", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7431e-05 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913290+0000", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120698 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913330+0000", 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0070000000000001e-05 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.507 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913370+0000", 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9580999999999999e-05 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184780 8.c 8:3fdf1b59:::benchmark_data_smithi043_104001_object60656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 106.434313374, 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95616667600000005, 2024-04-25T05:43:52.508 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184780 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957336+0000", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5799999999999999e-06 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957344+0000", 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.807e-06 2024-04-25T05:43:52.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957356+0000", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2404e-05 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957365+0000", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5040000000000002e-06 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957437+0000", 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350999999999999e-05 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957448+0000", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0669999999999999e-05 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910672+0000", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95322447200000004 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910797+0000", 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124442 2024-04-25T05:43:52.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910926+0000", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128849 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913223+0000", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022977219999999999 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913420+0000", 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019708099999999999 2024-04-25T05:43:52.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913431+0000", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0312000000000001e-05 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913459+0000", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8577000000000001e-05 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913501+0000", 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1904999999999999e-05 2024-04-25T05:43:52.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184787 8.c 8:3320ee50:::benchmark_data_smithi043_104001_object60663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 106.426982147, 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95582188800000001, 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.515 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184787 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964667+0000", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964673+0000", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964690+0000", 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7629999999999999e-05 2024-04-25T05:43:52.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964699+0000", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1390000000000001e-06 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964760+0000", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0217999999999998e-05 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964769+0000", 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7860000000000008e-06 2024-04-25T05:43:52.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913564+0000", 2024-04-25T05:43:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94879425100000003 2024-04-25T05:43:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913643+0000", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9556999999999997e-05 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913717+0000", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4209999999999996e-05 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920254+0000", 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065365060000000001 2024-04-25T05:43:52.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920304+0000", 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9512000000000003e-05 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920385+0000", 2024-04-25T05:43:52.523 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1979999999999998e-05 2024-04-25T05:43:52.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.524 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920439+0000", 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3928000000000003e-05 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920488+0000", 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8301999999999999e-05 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226338 8.6 8:6585b86d:::benchmark_data_smithi043_104001_object102214:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:43:52.525 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 72.494811636999998, 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.103761714, 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226338 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896840+0000", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2289999999999999e-06 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896861+0000", 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1603999999999998e-05 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896888+0000", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7047999999999999e-05 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896903+0000", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4888999999999999e-05 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019552+0000", 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12264891999999999 2024-04-25T05:43:52.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019561+0000", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0690000000000008e-06 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000512+0000", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98095072000000005 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000598+0000", 2024-04-25T05:43:52.529 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6235000000000004e-05 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226349 8.6 8:6373a4df:::benchmark_data_smithi043_104001_object102225:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 72.494718888999998, 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.139697025, 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.530 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226349 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896930+0000", 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4199999999999996e-07 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896933+0000", 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.551e-06 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.532 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896939+0000", 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1380000000000004e-06 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896945+0000", 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1959999999999999e-06 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019583+0000", 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122638941 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.533 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019591+0000", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.002927+0000", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98333629700000003 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003056+0000", 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129037 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003403+0000", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000346733 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.005083+0000", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016799779999999999 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.008504+0000", 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034209710000000001 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036533+0000", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028029505999999999 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036586+0000", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2278000000000003e-05 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036626+0000", 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0438000000000001e-05 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228017 8.16 8:6e8160c2:::benchmark_data_smithi043_104001_object103893:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 69.413747547, 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.134225407, 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228017 2024-04-25T05:43:52.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:52.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977902+0000", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5659999999999999e-06 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977908+0000", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.592e-06 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977923+0000", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5132e-05 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977934+0000", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0869999999999999e-05 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978035+0000", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101284 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978042+0000", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5229999999999998e-06 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982071+0000", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004028472 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982081+0000", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0589e-05 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104437+0000", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.122355209 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:43:52.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104572+0000", 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013536399999999999 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112038+0000", 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074657969999999997 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112126+0000", 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8009000000000004e-05 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228023 8.16 8:6ad7e352:::benchmark_data_smithi043_104001_object103899:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 69.413691846999996, 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1340453349999999, 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228023 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.544 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977957+0000", 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9500000000000003e-07 2024-04-25T05:43:52.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977960+0000", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3230000000000002e-06 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977967+0000", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977974+0000", 2024-04-25T05:43:52.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9659999999999999e-06 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978175+0000", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020133299999999999 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting_for_map not empty", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978182+0000", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1530000000000002e-06 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982727+0000", 2024-04-25T05:43:52.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045456259999999997 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982737+0000", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3440000000000007e-06 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104931+0000", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.12219398 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105055+0000", 2024-04-25T05:43:52.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012434499999999999 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5,6,7", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105352+0000", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297082 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.108987+0000", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00363503 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111435+0000", 2024-04-25T05:43:52.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024477959999999999 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111738+0000", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030323699999999998 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111789+0000", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0565999999999997e-05 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111869+0000", 2024-04-25T05:43:52.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0865999999999995e-05 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111908+0000", 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8052000000000001e-05 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112001+0000", 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3929000000000001e-05 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:43:52.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:43:52.552 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:43:52.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:52.655 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:43:52.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:52.701 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:43:52.741 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:43:52.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:52.783 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:43:52.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:52.823 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:43:52.863 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:43:52.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:52.903 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:43:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:52.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:52.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:52.948 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:43:52.953 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:43:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:52 smithi043 ceph-mon[27010]: pgmap v521: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 734 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:43:52.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3401277849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:52.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:52 smithi043 ceph-mon[37492]: pgmap v521: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 734 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:43:52.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3401277849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:53.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:53.080 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:53.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:53.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 15 62927 62912 103.456 75 0.0132385 0.00966087 2024-04-25T05:43:53.210 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:53.229 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:52 smithi060 ceph-mon[30898]: pgmap v521: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 734 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:43:53.229 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3401277849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:43:53.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:53.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:53.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:53.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:53.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:43:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:43:53.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:53.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:53.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:53.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:53.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:53.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:43:53.877 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:53.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:53.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:54.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:54.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:43:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:43:54.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[27010]: osdmap e317: 8 total, 8 up, 6 in 2024-04-25T05:43:54.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:43:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:43:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[37492]: osdmap e317: 8 total, 8 up, 6 in 2024-04-25T05:43:54.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:54.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:54.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 64242 64226 102.909 82.125 0.0151778 0.00971121 2024-04-25T05:43:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:53 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:43:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:53 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:43:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:53 smithi060 ceph-mon[30898]: osdmap e317: 8 total, 8 up, 6 in 2024-04-25T05:43:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:43:54.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:54.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:54.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:54.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:54.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.556 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:54.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:54.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:54.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:54.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:54.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:54.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:54.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:54.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:55.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:55.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:55.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:55.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:43:55.206142+0000 min lat: 0.00146388 max lat: 1.02736 avg lat: 0.00977126 2024-04-25T05:43:55.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:43:55.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 65492 65476 102.289 78.125 0.00712248 0.00977126 2024-04-25T05:43:55.215 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:55.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:54 smithi060 ceph-mon[30898]: pgmap v523: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:43:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:54 smithi060 ceph-mon[30898]: osdmap e318: 8 total, 8 up, 6 in 2024-04-25T05:43:55.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:54 smithi043 ceph-mon[37492]: pgmap v523: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:43:55.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:54 smithi043 ceph-mon[37492]: osdmap e318: 8 total, 8 up, 6 in 2024-04-25T05:43:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:54 smithi043 ceph-mon[27010]: pgmap v523: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:43:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:54 smithi043 ceph-mon[27010]: osdmap e318: 8 total, 8 up, 6 in 2024-04-25T05:43:55.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:55.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:55.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:55.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:55.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:55.869 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:55.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:55.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:56.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:56.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:56.132 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:56.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 66709 66693 101.649 76.0625 0.0305248 0.00982879 2024-04-25T05:43:56.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:56.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:56.408 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:56.510 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:56.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:43:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:43:56.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:56.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:56.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:56.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:57.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:57.089 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:57.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:57.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 67849 67833 100.926 71.25 0.00754291 0.00990228 2024-04-25T05:43:57.215 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:57.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:56 smithi060 ceph-mon[30898]: pgmap v525: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-25T05:43:57.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:57.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:57.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:56 smithi043 ceph-mon[27010]: pgmap v525: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-25T05:43:57.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:56 smithi043 ceph-mon[37492]: pgmap v525: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-25T05:43:57.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:57.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.626 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:57.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:57.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:57.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:57.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:57.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:57.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:57.944 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:43:57.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:58.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:58.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:58.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:57 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1992624485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:57 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1992624485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:58.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:58.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 68975 68959 100.215 70.375 0.0091929 0.0099731 2024-04-25T05:43:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:57 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1992624485' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:43:58.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:58.291 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:58.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:43:58.422 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:43:58.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:58.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:58.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:58.678 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:58.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:58.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:58.844 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:43:58.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:58.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:58.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:59.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:43:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:43:59.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:43:59.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 642 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-25T05:43:59.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:59.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:59.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: osdmap e319: 8 total, 8 up, 6 in 2024-04-25T05:43:59.188 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:43:59.189 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:43:59.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 642 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-25T05:43:59.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:59.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:59.189 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: osdmap e319: 8 total, 8 up, 6 in 2024-04-25T05:43:59.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:43:59.190 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:43:58 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:43:59.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 15 70231 70216 99.7227 78.5625 0.00495646 0.0100226 2024-04-25T05:43:59.210 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 642 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: osdmap e319: 8 total, 8 up, 6 in 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:43:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:43:58 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:43:59.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:43:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:43:59.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:43:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:43:59.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:43:59.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:43:59.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:59.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:59.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:59.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:43:59.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:43:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:43:59.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:43:59.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:43:59.979 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:43:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:43:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:00.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:00.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:00.207 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 71456 71440 99.2059 76.5 0.0177102 0.0100749 2024-04-25T05:44:00.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:00.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:00.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:00.436 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:00.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:00.560 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:00.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:44:00.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:00.688 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:00.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:00.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:00.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:00.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:00.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:01.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:01.101 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:01.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: pgmap v528: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 737 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:44:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: osdmap e320: 8 total, 8 up, 6 in 2024-04-25T05:44:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:44:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:44:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:44:01.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: pgmap v528: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 737 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: osdmap e320: 8 total, 8 up, 6 in 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:44:01.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:00 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:44:01.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:01.206 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 72700 72684 98.7394 77.75 0.0070582 0.0101221 2024-04-25T05:44:01.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.233 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:01.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: pgmap v528: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 737 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: osdmap e320: 8 total, 8 up, 6 in 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:44:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:00 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:44:01.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:01.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:01.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-04-25T05:44:01.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:01.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:01.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.664 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:01.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:01.708 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:01.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:01.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:01.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:44:01.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:01.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:01.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:02.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:02.083 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:02.114 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:44:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:44:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:44:02.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:44:02.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:44:02.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:44:02.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:44:02.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:01 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:44:02.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:02.207 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 73798 73782 98.0962 68.625 0.0123824 0.0101884 2024-04-25T05:44:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.213 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:01 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:44:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:01 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:44:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:01 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:44:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:01 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:44:02.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:02.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:02.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:02.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:02.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:02.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:02.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:02.769 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:02.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:44:02.833 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:02.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:02.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:02.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:03.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.052 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: pgmap v530: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 734 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:44:03.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: pgmap v530: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 734 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:44:03.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:02 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:44:03.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:03.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:03.207 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 75057 75041 97.6916 78.6875 0.0104216 0.0102312 2024-04-25T05:44:03.240 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: pgmap v530: 158 pgs: 158 active+clean; 4.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 734 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-25T05:44:03.240 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:44:03.240 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:44:03.241 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:03.241 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:44:03.241 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:02 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:44:03.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:03.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:03.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:03.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:03.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:03.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.605 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:03.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:03.733 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:03.744 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:03.795 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:03.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:03.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:03.859 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:03.893 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:03.937 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:03.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:03.984 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:03.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:03.993 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/210772717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:44:03.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/210772717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:44:03.996 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:03 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:44:04.004 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:04.041 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:04.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:04.087 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:04.095 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:04.095 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:04.207 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 76247 76231 97.2157 74.375 0.0072896 0.0102813 2024-04-25T05:44:04.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.242 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:04.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/210772717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:44:04.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:03 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:44:04.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:04.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:04.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.523 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:04.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:04.527 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:04.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:04.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:04.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:04.792 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:04.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:04.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:04.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:05.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:05.064 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:05.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:05.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:05.207 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 15 77566 77551 96.9213 82.5 0.00897093 0.010311 2024-04-25T05:44:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:04 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:44:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:04 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:44:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:04 smithi060 ceph-mon[30898]: pgmap v531: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:04 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:44:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:04 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:44:05.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:05.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:44:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:44:05.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[27010]: pgmap v531: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[37492]: pgmap v531: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:44:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:04 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:44:05.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:05.470 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:05.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:05.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:05.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:05.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:05.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:05.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:06.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:06.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:06.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:06.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:05 smithi043 ceph-mon[27010]: 6.17 scrub starts 2024-04-25T05:44:06.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:05 smithi043 ceph-mon[27010]: 6.17 scrub ok 2024-04-25T05:44:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:05 smithi043 ceph-mon[37492]: 6.17 scrub starts 2024-04-25T05:44:06.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:05 smithi043 ceph-mon[37492]: 6.17 scrub ok 2024-04-25T05:44:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.174 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:06.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:06.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 78886 78870 96.6371 82.4375 0.00848977 0.0103404 2024-04-25T05:44:06.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:05 smithi060 ceph-mon[30898]: 6.17 scrub starts 2024-04-25T05:44:06.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:05 smithi060 ceph-mon[30898]: 6.17 scrub ok 2024-04-25T05:44:06.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:06.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:06.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:06.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:06.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:06.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:06.571 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:06.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.696 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:06.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.824 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:06.884 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:44:06.884 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:44:06.884 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:44:06.884 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:44:06.884 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:44:06.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:06.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:06.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:07.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:44:07.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:07.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:44:07.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:44:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: pgmap v532: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:44:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:44:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:44:07.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: pgmap v532: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:44:07.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:44:07.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:44:07.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:44:07.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:06 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:44:07.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:07.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 15 80103 80088 96.2425 76.125 0.0158095 0.0103852 2024-04-25T05:44:07.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: pgmap v532: 158 pgs: 158 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:44:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:06 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:44:07.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.366 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:07.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:07.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:07.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:07.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:07.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:07.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:07.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:07.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:07.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:07.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:08.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:08.051 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:08.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:44:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:44:08.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:07 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:44:08.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.183 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:08.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:08.209 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 15 81261 81246 95.7921 72.375 0.00731791 0.0104348 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:44:08.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:07 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:44:08.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:08.309 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:08.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:08.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:08.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:44:08.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:08.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:08.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:08.711 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:08.773 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:08.813 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:08.879 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:08.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:08.952 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[27010]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T05:44:08.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:09.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:09.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:09.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:09.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 15 82311 82296 95.2334 65.625 0.0117006 0.0104955 2024-04-25T05:44:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:08 smithi060 ceph-mon[30898]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T05:44:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:08 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:44:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:08 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:44:09.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:09.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:44:09.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:44:09.290 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[37492]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-25T05:44:09.290 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:44:09.290 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:08 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:44:09.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:09.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:09.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:09.537 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:09.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:09.581 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:09.628 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:09.670 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:09.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:09.715 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:09.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:09.759 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:09.807 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:09.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:09.847 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:09.857 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:09.864 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:09.870 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:09.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:09.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:10.092 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:44:10.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 5.13 scrub starts 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 5.13 scrub ok 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:44:10.115 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/653119532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 5.13 scrub starts 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 5.13 scrub ok 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:44:10.116 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:44:10.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:44:10.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:44:10.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:10.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:10.117 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/653119532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:10.123 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:10.198 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:10.198 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:10.209 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 83488 83472 94.8382 73.5 0.00976255 0.0105387 2024-04-25T05:44:10.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 5.13 scrub starts 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 5.13 scrub ok 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:44:10.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:44:10.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:10.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:10.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/653119532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:10.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:10.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:10.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:10.528 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:44:10.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:44:10.529 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-25T05:44:10.529 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-25T05:44:10.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:44:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.531 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:10.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:10.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:10.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:10.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:10.800 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:10.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:10.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:11.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:11.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:44:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:44:11.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: pgmap v534: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[27010]: osdmap e321: 8 total, 8 up, 6 in 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: pgmap v534: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:11.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:10 smithi043 ceph-mon[37492]: osdmap e321: 8 total, 8 up, 6 in 2024-04-25T05:44:11.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:11.202 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:11.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 84803 84787 94.6117 82.1875 0.0178916 0.0105644 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: pgmap v534: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3523575665' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:11.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:10 smithi060 ceph-mon[30898]: osdmap e321: 8 total, 8 up, 6 in 2024-04-25T05:44:11.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:44:11.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:11.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:11.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:11.529 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:11.631 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:11.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:11.775 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.816 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:11.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:11.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:11.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:11.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:12.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:12.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:12.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:12.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 86123 86107 94.399 82.5 0.00777067 0.0105886 2024-04-25T05:44:12.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:12.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:11 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:44:12.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:11 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:44:12.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:11 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:44:12.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:11 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:44:12.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:11 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:44:12.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:11 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:44:12.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:12.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:12.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:12.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:12.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:12.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:12.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:12.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:12.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:12.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:12.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:13.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:13.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:13.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:13.169 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:13.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 87455 87439 94.2067 83.25 0.0175232 0.0106097 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 5.14 scrub starts 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 5.14 scrub ok 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 5.3 deep-scrub starts 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 5.3 deep-scrub ok 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: pgmap v536: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:13.236 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 5.14 scrub starts 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 5.14 scrub ok 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 5.3 deep-scrub starts 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 5.3 deep-scrub ok 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: pgmap v536: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:44:13.237 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:12 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 5.14 scrub starts 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 5.14 scrub ok 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 5.3 deep-scrub starts 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 5.3 deep-scrub ok 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: pgmap v536: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 44 KiB/s rd, 77 MiB/s wr, 1.30k op/s 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:44:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:12 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:44:13.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:13.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:13.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:13.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:13.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:13.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:13.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:13.693 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:13.743 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:13.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:13.751 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:13.795 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:13.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:13.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:13.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:13.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:14.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:14.111 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:14.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 88689 88673 93.9171 77.125 0.010289 0.0106429 2024-04-25T05:44:14.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:14.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:44:14.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 4.17 scrub starts 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 4.17 scrub ok 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:44:14.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:13 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:14.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:14.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 4.17 scrub starts 2024-04-25T05:44:14.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 4.17 scrub ok 2024-04-25T05:44:14.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 4.17 scrub starts 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 4.17 scrub ok 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:44:14.285 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:13 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:14.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:14.497 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:14.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:14.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:14.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:14.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:14.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:14.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:14.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:14.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:15.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:15.061 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:15.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:15.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:15.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:44:15.209725+0000 min lat: 0.00146388 max lat: 1.02736 avg lat: 0.0106768 2024-04-25T05:44:15.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:44:15.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 89904 89888 93.6174 75.9375 0.0106546 0.0106768 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: pgmap v537: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 78 MiB/s wr, 1.31k op/s 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 4.11 scrub starts 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 4.11 scrub ok 2024-04-25T05:44:15.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 1.5 deep-scrub starts 2024-04-25T05:44:15.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 1.5 deep-scrub ok 2024-04-25T05:44:15.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:44:15.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:44:15.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:15.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:14 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:15.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:15.332 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: pgmap v537: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 78 MiB/s wr, 1.31k op/s 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 4.11 scrub starts 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 4.11 scrub ok 2024-04-25T05:44:15.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 1.5 deep-scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 1.5 deep-scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: pgmap v537: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 78 MiB/s wr, 1.31k op/s 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 4.11 scrub starts 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 4.11 scrub ok 2024-04-25T05:44:15.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 1.5 deep-scrub starts 2024-04-25T05:44:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 1.5 deep-scrub ok 2024-04-25T05:44:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:44:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:44:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:15.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:14 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:15.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:15.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:15.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.591 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:15.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:15.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:44:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:15.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:15.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:15.952 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:15.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:15.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:16.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:16.209 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 90931 90915 93.1348 64.1875 0.0119837 0.0107322 2024-04-25T05:44:16.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:16.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:16.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[27010]: osdmap e322: 8 total, 8 up, 6 in 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[37492]: osdmap e322: 8 total, 8 up, 6 in 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:44:16.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:44:16.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:44:16.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:16 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:44:16.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:16.424 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:15 smithi060 ceph-mon[30898]: osdmap e322: 8 total, 8 up, 6 in 2024-04-25T05:44:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:16 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:44:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:16 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:44:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:16 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:44:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:16 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:44:16.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:16.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:16.684 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:16.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:16.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:16.836 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:44:16.840 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:44:16.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:16.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:16.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:17.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:17.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:17.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:17.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 92171 92155 92.8807 77.5 0.00959665 0.0107615 2024-04-25T05:44:17.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:17.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:17 smithi043 ceph-mon[27010]: pgmap v539: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 52 KiB/s rd, 97 MiB/s wr, 1.64k op/s 2024-04-25T05:44:17.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:17 smithi043 ceph-mon[37492]: pgmap v539: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 52 KiB/s rd, 97 MiB/s wr, 1.64k op/s 2024-04-25T05:44:17.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:17.371 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:17.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:17.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:17 smithi060 ceph-mon[30898]: pgmap v539: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 52 KiB/s rd, 97 MiB/s wr, 1.64k op/s 2024-04-25T05:44:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:17.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:17.635 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:17.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:17.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:17.813 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":539,"stamp":"2024-04-25T05:44:15.867302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5746399771,"num_objects":87904,"num_object_clones":0,"num_object_copies":263712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87904,"num_whiteouts":0,"num_read":2020,"num_read_kb":1740,"num_write":88588,"num_write_kb":5611181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":337596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62141,"ondisk_log_size":62141,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19583116,"kb_used_data":16741292,"kb_used_omap":9,"kb_used_meta":2841398,"kb_avail":542887796,"statfs":{"total":575970213888,"available":555917103104,"internally_reserved":0,"allocated":17143083008,"data_stored":17140516259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":2909592230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":814153728,"num_objects":12423,"num_object_clones":0,"num_object_copies":37269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12423,"num_whiteouts":0,"num_read":418,"num_read_kb":418,"num_write":12695,"num_write_kb":795072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002805"},"pg_stats":[{"pgid":"9.1","version":"321'5539","reported_seq":5598,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.564005+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:44:13.564005+0000","last_peered":"2024-04-25T05:44:13.564005+0000","last_clean":"2024-04-25T05:44:13.564005+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:44:13.564005+0000","last_undegraded":"2024-04-25T05:44:13.564005+0000","last_fullsized":"2024-04-25T05:44:13.564005+0000","mapping_epoch":296,"log_start":"307'1900","ondisk_log_start":"307'1900","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:15.164933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363003904,"num_objects":5539,"num_object_clones":0,"num_object_copies":16617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":354496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":265,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.087095+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:44:10.087095+0000","last_peered":"2024-04-25T05:44:10.087095+0000","last_clean":"2024-04-25T05:44:10.087095+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:44:10.087095+0000","last_undegraded":"2024-04-25T05:44:10.087095+0000","last_fullsized":"2024-04-25T05:44:10.087095+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081719+0000","last_change":"2024-04-25T05:42:17.311411+0000","last_active":"2024-04-25T05:44:10.081719+0000","last_peered":"2024-04-25T05:44:10.081719+0000","last_clean":"2024-04-25T05:44:10.081719+0000","last_became_active":"2024-04-25T05:40:43.432950+0000","last_became_peered":"2024-04-25T05:40:43.432950+0000","last_unstale":"2024-04-25T05:44:10.081719+0000","last_undegraded":"2024-04-25T05:44:10.081719+0000","last_fullsized":"2024-04-25T05:44:10.081719+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"304'11","reported_seq":331,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082045+0000","last_change":"2024-04-25T05:43:16.218414+0000","last_active":"2024-04-25T05:44:10.082045+0000","last_peered":"2024-04-25T05:44:10.082045+0000","last_clean":"2024-04-25T05:44:10.082045+0000","last_became_active":"2024-04-25T05:43:16.202958+0000","last_became_peered":"2024-04-25T05:43:16.202958+0000","last_unstale":"2024-04-25T05:44:10.082045+0000","last_undegraded":"2024-04-25T05:44:10.082045+0000","last_fullsized":"2024-04-25T05:44:10.082045+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":228,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095326+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:44:10.095326+0000","last_peered":"2024-04-25T05:44:10.095326+0000","last_clean":"2024-04-25T05:44:10.095326+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:44:10.095326+0000","last_undegraded":"2024-04-25T05:44:10.095326+0000","last_fullsized":"2024-04-25T05:44:10.095326+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:28.523566+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":862,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082802+0000","last_change":"2024-04-25T05:43:19.210320+0000","last_active":"2024-04-25T05:44:10.082802+0000","last_peered":"2024-04-25T05:44:10.082802+0000","last_clean":"2024-04-25T05:44:10.082802+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:44:10.082802+0000","last_undegraded":"2024-04-25T05:44:10.082802+0000","last_fullsized":"2024-04-25T05:44:10.082802+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"321'5434","reported_seq":5493,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.540973+0000","last_change":"2024-04-25T05:43:10.483548+0000","last_active":"2024-04-25T05:44:13.540973+0000","last_peered":"2024-04-25T05:44:13.540973+0000","last_clean":"2024-04-25T05:44:13.540973+0000","last_became_active":"2024-04-25T05:43:10.483213+0000","last_became_peered":"2024-04-25T05:43:10.483213+0000","last_unstale":"2024-04-25T05:44:13.540973+0000","last_undegraded":"2024-04-25T05:44:13.540973+0000","last_fullsized":"2024-04-25T05:44:13.540973+0000","mapping_epoch":296,"log_start":"308'2000","ondisk_log_start":"308'2000","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:22.344956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356122624,"num_objects":5434,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":347776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":274,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095442+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:44:10.095442+0000","last_peered":"2024-04-25T05:44:10.095442+0000","last_clean":"2024-04-25T05:44:10.095442+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:44:10.095442+0000","last_undegraded":"2024-04-25T05:44:10.095442+0000","last_fullsized":"2024-04-25T05:44:10.095442+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":270,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091146+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:44:10.091146+0000","last_peered":"2024-04-25T05:44:10.091146+0000","last_clean":"2024-04-25T05:44:10.091146+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:44:10.091146+0000","last_undegraded":"2024-04-25T05:44:10.091146+0000","last_fullsized":"2024-04-25T05:44:10.091146+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.541034+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":556,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092102+0000","last_change":"2024-04-25T05:40:44.537276+0000","last_active":"2024-04-25T05:44:10.092102+0000","last_peered":"2024-04-25T05:44:10.092102+0000","last_clean":"2024-04-25T05:44:10.092102+0000","last_became_active":"2024-04-25T05:40:43.449807+0000","last_became_peered":"2024-04-25T05:40:43.449807+0000","last_unstale":"2024-04-25T05:44:10.092102+0000","last_undegraded":"2024-04-25T05:44:10.092102+0000","last_fullsized":"2024-04-25T05:44:10.092102+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":181,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083372+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:44:10.083372+0000","last_peered":"2024-04-25T05:44:10.083372+0000","last_clean":"2024-04-25T05:44:10.083372+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:44:10.083372+0000","last_undegraded":"2024-04-25T05:44:10.083372+0000","last_fullsized":"2024-04-25T05:44:10.083372+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.600058+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":462,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081468+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:44:10.081468+0000","last_peered":"2024-04-25T05:44:10.081468+0000","last_clean":"2024-04-25T05:44:10.081468+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:44:10.081468+0000","last_undegraded":"2024-04-25T05:44:10.081468+0000","last_fullsized":"2024-04-25T05:44:10.081468+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.377967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"321'5433","reported_seq":5492,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.889064+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:44:12.889064+0000","last_peered":"2024-04-25T05:44:12.889064+0000","last_clean":"2024-04-25T05:44:12.889064+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:44:12.889064+0000","last_undegraded":"2024-04-25T05:44:12.889064+0000","last_fullsized":"2024-04-25T05:44:12.889064+0000","mapping_epoch":296,"log_start":"306'1700","ondisk_log_start":"306'1700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3733,"log_dups_size":0,"ondisk_log_size":3733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:30.419007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356057088,"num_objects":5433,"num_object_clones":0,"num_object_copies":16299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5433,"num_write_kb":347712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":574,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092090+0000","last_change":"2024-04-25T05:42:53.501053+0000","last_active":"2024-04-25T05:44:10.092090+0000","last_peered":"2024-04-25T05:44:10.092090+0000","last_clean":"2024-04-25T05:44:10.092090+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:44:10.092090+0000","last_undegraded":"2024-04-25T05:44:10.092090+0000","last_fullsized":"2024-04-25T05:44:10.092090+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:42:53.500776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.500776+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:22.545533+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":560,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076016+0000","last_change":"2024-04-25T05:42:51.237692+0000","last_active":"2024-04-25T05:44:10.076016+0000","last_peered":"2024-04-25T05:44:10.076016+0000","last_clean":"2024-04-25T05:44:10.076016+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:44:10.076016+0000","last_undegraded":"2024-04-25T05:44:10.076016+0000","last_fullsized":"2024-04-25T05:44:10.076016+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.237570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.237570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:13.272604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":356,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076610+0000","last_change":"2024-04-25T05:43:24.227422+0000","last_active":"2024-04-25T05:44:10.076610+0000","last_peered":"2024-04-25T05:44:10.076610+0000","last_clean":"2024-04-25T05:44:10.076610+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:44:10.076610+0000","last_undegraded":"2024-04-25T05:44:10.076610+0000","last_fullsized":"2024-04-25T05:44:10.076610+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.227343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.227343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:28.047197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":544,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083414+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:44:10.083414+0000","last_peered":"2024-04-25T05:44:10.083414+0000","last_clean":"2024-04-25T05:44:10.083414+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:44:10.083414+0000","last_undegraded":"2024-04-25T05:44:10.083414+0000","last_fullsized":"2024-04-25T05:44:10.083414+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:23.864315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"321'5565","reported_seq":5623,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.565473+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:44:13.565473+0000","last_peered":"2024-04-25T05:44:13.565473+0000","last_clean":"2024-04-25T05:44:13.565473+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:44:13.565473+0000","last_undegraded":"2024-04-25T05:44:13.565473+0000","last_fullsized":"2024-04-25T05:44:13.565473+0000","mapping_epoch":296,"log_start":"307'1900","ondisk_log_start":"307'1900","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:16.108032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5565,"num_write_kb":356160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":592,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092908+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:44:10.092908+0000","last_peered":"2024-04-25T05:44:10.092908+0000","last_clean":"2024-04-25T05:44:10.092908+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:44:10.092908+0000","last_undegraded":"2024-04-25T05:44:10.092908+0000","last_fullsized":"2024-04-25T05:44:10.092908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:17.696734+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":546,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.053643+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:44:13.053643+0000","last_peered":"2024-04-25T05:44:13.053643+0000","last_clean":"2024-04-25T05:44:13.053643+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:44:13.053643+0000","last_undegraded":"2024-04-25T05:44:13.053643+0000","last_fullsized":"2024-04-25T05:44:13.053643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.125277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"304'12","reported_seq":126,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086711+0000","last_change":"2024-04-25T05:43:16.215807+0000","last_active":"2024-04-25T05:44:10.086711+0000","last_peered":"2024-04-25T05:44:10.086711+0000","last_clean":"2024-04-25T05:44:10.086711+0000","last_became_active":"2024-04-25T05:43:16.203616+0000","last_became_peered":"2024-04-25T05:43:16.203616+0000","last_unstale":"2024-04-25T05:44:10.086711+0000","last_undegraded":"2024-04-25T05:44:10.086711+0000","last_fullsized":"2024-04-25T05:44:10.086711+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":402,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095832+0000","last_change":"2024-04-25T05:43:16.207527+0000","last_active":"2024-04-25T05:44:10.095832+0000","last_peered":"2024-04-25T05:44:10.095832+0000","last_clean":"2024-04-25T05:44:10.095832+0000","last_became_active":"2024-04-25T05:43:16.207162+0000","last_became_peered":"2024-04-25T05:43:16.207162+0000","last_unstale":"2024-04-25T05:44:10.095832+0000","last_undegraded":"2024-04-25T05:44:10.095832+0000","last_fullsized":"2024-04-25T05:44:10.095832+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":529,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091040+0000","last_change":"2024-04-25T05:44:04.667653+0000","last_active":"2024-04-25T05:44:10.091040+0000","last_peered":"2024-04-25T05:44:10.091040+0000","last_clean":"2024-04-25T05:44:10.091040+0000","last_became_active":"2024-04-25T05:43:19.207294+0000","last_became_peered":"2024-04-25T05:43:19.207294+0000","last_unstale":"2024-04-25T05:44:10.091040+0000","last_undegraded":"2024-04-25T05:44:10.091040+0000","last_fullsized":"2024-04-25T05:44:10.091040+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:21.821857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"321'5483","reported_seq":5548,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.895764+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:44:12.895764+0000","last_peered":"2024-04-25T05:44:12.895764+0000","last_clean":"2024-04-25T05:44:12.895764+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:44:12.895764+0000","last_undegraded":"2024-04-25T05:44:12.895764+0000","last_fullsized":"2024-04-25T05:44:12.895764+0000","mapping_epoch":301,"log_start":"306'1800","ondisk_log_start":"306'1800","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:17.707548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359333888,"num_objects":5483,"num_object_clones":0,"num_object_copies":16449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5483,"num_write_kb":350912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":546,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.075908+0000","last_change":"2024-04-25T05:43:58.091032+0000","last_active":"2024-04-25T05:44:10.075908+0000","last_peered":"2024-04-25T05:44:10.075908+0000","last_clean":"2024-04-25T05:44:10.075908+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:44:10.075908+0000","last_undegraded":"2024-04-25T05:44:10.075908+0000","last_fullsized":"2024-04-25T05:44:10.075908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:58.090855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:58.090855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.355724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":665,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084000+0000","last_change":"2024-04-25T05:40:48.519454+0000","last_active":"2024-04-25T05:44:10.084000+0000","last_peered":"2024-04-25T05:44:10.084000+0000","last_clean":"2024-04-25T05:44:10.084000+0000","last_became_active":"2024-04-25T05:40:43.440689+0000","last_became_peered":"2024-04-25T05:40:43.440689+0000","last_unstale":"2024-04-25T05:44:10.084000+0000","last_undegraded":"2024-04-25T05:44:10.084000+0000","last_fullsized":"2024-04-25T05:44:10.084000+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'30","last_scrub_stamp":"2024-04-25T05:40:48.519288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:48.519288+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083521+0000","last_change":"2024-04-25T05:40:43.516969+0000","last_active":"2024-04-25T05:44:10.083521+0000","last_peered":"2024-04-25T05:44:10.083521+0000","last_clean":"2024-04-25T05:44:10.083521+0000","last_became_active":"2024-04-25T05:40:43.440253+0000","last_became_peered":"2024-04-25T05:40:43.440253+0000","last_unstale":"2024-04-25T05:44:10.083521+0000","last_undegraded":"2024-04-25T05:44:10.083521+0000","last_fullsized":"2024-04-25T05:44:10.083521+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:43.516897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:43.516897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":222,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084677+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:44:10.084677+0000","last_peered":"2024-04-25T05:44:10.084677+0000","last_clean":"2024-04-25T05:44:10.084677+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:44:10.084677+0000","last_undegraded":"2024-04-25T05:44:10.084677+0000","last_fullsized":"2024-04-25T05:44:10.084677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.232569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":749,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076517+0000","last_change":"2024-04-25T05:43:19.207448+0000","last_active":"2024-04-25T05:44:10.076517+0000","last_peered":"2024-04-25T05:44:10.076517+0000","last_clean":"2024-04-25T05:44:10.076517+0000","last_became_active":"2024-04-25T05:43:19.207030+0000","last_became_peered":"2024-04-25T05:43:19.207030+0000","last_unstale":"2024-04-25T05:44:10.076517+0000","last_undegraded":"2024-04-25T05:44:10.076517+0000","last_fullsized":"2024-04-25T05:44:10.076517+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:54.214835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.214835+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"321'5558","reported_seq":5617,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.538944+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:44:13.538944+0000","last_peered":"2024-04-25T05:44:13.538944+0000","last_clean":"2024-04-25T05:44:13.538944+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:44:13.538944+0000","last_undegraded":"2024-04-25T05:44:13.538944+0000","last_fullsized":"2024-04-25T05:44:13.538944+0000","mapping_epoch":296,"log_start":"307'1900","ondisk_log_start":"307'1900","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:15.613002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364249088,"num_objects":5558,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":662,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084105+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:44:10.084105+0000","last_peered":"2024-04-25T05:44:10.084105+0000","last_clean":"2024-04-25T05:44:10.084105+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:44:10.084105+0000","last_undegraded":"2024-04-25T05:44:10.084105+0000","last_fullsized":"2024-04-25T05:44:10.084105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":466,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091748+0000","last_change":"2024-04-25T05:43:24.628525+0000","last_active":"2024-04-25T05:44:10.091748+0000","last_peered":"2024-04-25T05:44:10.091748+0000","last_clean":"2024-04-25T05:44:10.091748+0000","last_became_active":"2024-04-25T05:43:16.199826+0000","last_became_peered":"2024-04-25T05:43:16.199826+0000","last_unstale":"2024-04-25T05:44:10.091748+0000","last_undegraded":"2024-04-25T05:44:10.091748+0000","last_fullsized":"2024-04-25T05:44:10.091748+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:43:24.628430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.628430+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.100187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":200,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086909+0000","last_change":"2024-04-25T05:43:23.347988+0000","last_active":"2024-04-25T05:44:10.086909+0000","last_peered":"2024-04-25T05:44:10.086909+0000","last_clean":"2024-04-25T05:44:10.086909+0000","last_became_active":"2024-04-25T05:40:43.443728+0000","last_became_peered":"2024-04-25T05:40:43.443728+0000","last_unstale":"2024-04-25T05:44:10.086909+0000","last_undegraded":"2024-04-25T05:44:10.086909+0000","last_fullsized":"2024-04-25T05:44:10.086909+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:38.605995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":179,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081517+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:44:10.081517+0000","last_peered":"2024-04-25T05:44:10.081517+0000","last_clean":"2024-04-25T05:44:10.081517+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:44:10.081517+0000","last_undegraded":"2024-04-25T05:44:10.081517+0000","last_fullsized":"2024-04-25T05:44:10.081517+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:25.031279+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":726,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082197+0000","last_change":"2024-04-25T05:43:19.215636+0000","last_active":"2024-04-25T05:44:10.082197+0000","last_peered":"2024-04-25T05:44:10.082197+0000","last_clean":"2024-04-25T05:44:10.082197+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:44:10.082197+0000","last_undegraded":"2024-04-25T05:44:10.082197+0000","last_fullsized":"2024-04-25T05:44:10.082197+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"321'5503","reported_seq":5553,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.547343+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:44:13.547343+0000","last_peered":"2024-04-25T05:44:13.547343+0000","last_clean":"2024-04-25T05:44:13.547343+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:44:13.547343+0000","last_undegraded":"2024-04-25T05:44:13.547343+0000","last_fullsized":"2024-04-25T05:44:13.547343+0000","mapping_epoch":301,"log_start":"308'2000","ondisk_log_start":"308'2000","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:26.841917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360644608,"num_objects":5503,"num_object_clones":0,"num_object_copies":16509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5503,"num_write_kb":352192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"321'14","reported_seq":253,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:11.494383+0000","last_change":"2024-04-25T05:44:11.494436+0000","last_active":"2024-04-25T05:44:11.494383+0000","last_peered":"2024-04-25T05:44:11.494383+0000","last_clean":"2024-04-25T05:44:11.494383+0000","last_became_active":"2024-04-25T05:43:16.206522+0000","last_became_peered":"2024-04-25T05:43:16.206522+0000","last_unstale":"2024-04-25T05:44:11.494383+0000","last_undegraded":"2024-04-25T05:44:11.494383+0000","last_fullsized":"2024-04-25T05:44:11.494383+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:20.615037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":228,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.093570+0000","last_change":"2024-04-25T05:43:16.308720+0000","last_active":"2024-04-25T05:44:10.093570+0000","last_peered":"2024-04-25T05:44:10.093570+0000","last_clean":"2024-04-25T05:44:10.093570+0000","last_became_active":"2024-04-25T05:43:16.210437+0000","last_became_peered":"2024-04-25T05:43:16.210437+0000","last_unstale":"2024-04-25T05:44:10.093570+0000","last_undegraded":"2024-04-25T05:44:10.093570+0000","last_fullsized":"2024-04-25T05:44:10.093570+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":433,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081609+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:44:10.081609+0000","last_peered":"2024-04-25T05:44:10.081609+0000","last_clean":"2024-04-25T05:44:10.081609+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:44:10.081609+0000","last_undegraded":"2024-04-25T05:44:10.081609+0000","last_fullsized":"2024-04-25T05:44:10.081609+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:21.585409+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":208,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090811+0000","last_change":"2024-04-25T05:43:16.213328+0000","last_active":"2024-04-25T05:44:10.090811+0000","last_peered":"2024-04-25T05:44:10.090811+0000","last_clean":"2024-04-25T05:44:10.090811+0000","last_became_active":"2024-04-25T05:43:16.213005+0000","last_became_peered":"2024-04-25T05:43:16.213005+0000","last_unstale":"2024-04-25T05:44:10.090811+0000","last_undegraded":"2024-04-25T05:44:10.090811+0000","last_fullsized":"2024-04-25T05:44:10.090811+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":725,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091806+0000","last_change":"2024-04-25T05:43:28.638040+0000","last_active":"2024-04-25T05:44:10.091806+0000","last_peered":"2024-04-25T05:44:10.091806+0000","last_clean":"2024-04-25T05:44:10.091806+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:44:10.091806+0000","last_undegraded":"2024-04-25T05:44:10.091806+0000","last_fullsized":"2024-04-25T05:44:10.091806+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:28.637882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:28.637882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.481823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"321'5423","reported_seq":5482,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.887717+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:44:12.887717+0000","last_peered":"2024-04-25T05:44:12.887717+0000","last_clean":"2024-04-25T05:44:12.887717+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:44:12.887717+0000","last_undegraded":"2024-04-25T05:44:12.887717+0000","last_fullsized":"2024-04-25T05:44:12.887717+0000","mapping_epoch":296,"log_start":"306'1700","ondisk_log_start":"306'1700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.350272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355401728,"num_objects":5423,"num_object_clones":0,"num_object_copies":16269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5423,"num_write_kb":347072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":282,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091194+0000","last_change":"2024-04-25T05:43:30.613961+0000","last_active":"2024-04-25T05:44:10.091194+0000","last_peered":"2024-04-25T05:44:10.091194+0000","last_clean":"2024-04-25T05:44:10.091194+0000","last_became_active":"2024-04-25T05:43:16.199833+0000","last_became_peered":"2024-04-25T05:43:16.199833+0000","last_unstale":"2024-04-25T05:44:10.091194+0000","last_undegraded":"2024-04-25T05:44:10.091194+0000","last_fullsized":"2024-04-25T05:44:10.091194+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:32.667203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":257,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086382+0000","last_change":"2024-04-25T05:43:16.300518+0000","last_active":"2024-04-25T05:44:10.086382+0000","last_peered":"2024-04-25T05:44:10.086382+0000","last_clean":"2024-04-25T05:44:10.086382+0000","last_became_active":"2024-04-25T05:43:16.215452+0000","last_became_peered":"2024-04-25T05:43:16.215452+0000","last_unstale":"2024-04-25T05:44:10.086382+0000","last_undegraded":"2024-04-25T05:44:10.086382+0000","last_fullsized":"2024-04-25T05:44:10.086382+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":566,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091777+0000","last_change":"2024-04-25T05:43:26.307795+0000","last_active":"2024-04-25T05:44:10.091777+0000","last_peered":"2024-04-25T05:44:10.091777+0000","last_clean":"2024-04-25T05:44:10.091777+0000","last_became_active":"2024-04-25T05:43:16.210243+0000","last_became_peered":"2024-04-25T05:43:16.210243+0000","last_unstale":"2024-04-25T05:44:10.091777+0000","last_undegraded":"2024-04-25T05:44:10.091777+0000","last_fullsized":"2024-04-25T05:44:10.091777+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:38.826125+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091164+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:44:10.091164+0000","last_peered":"2024-04-25T05:44:10.091164+0000","last_clean":"2024-04-25T05:44:10.091164+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:44:10.091164+0000","last_undegraded":"2024-04-25T05:44:10.091164+0000","last_fullsized":"2024-04-25T05:44:10.091164+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":734,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091878+0000","last_change":"2024-04-25T05:42:57.500822+0000","last_active":"2024-04-25T05:44:10.091878+0000","last_peered":"2024-04-25T05:44:10.091878+0000","last_clean":"2024-04-25T05:44:10.091878+0000","last_became_active":"2024-04-25T05:40:45.845535+0000","last_became_peered":"2024-04-25T05:40:45.845535+0000","last_unstale":"2024-04-25T05:44:10.091878+0000","last_undegraded":"2024-04-25T05:44:10.091878+0000","last_fullsized":"2024-04-25T05:44:10.091878+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.500683+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:17.122506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"321'5470","reported_seq":5536,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.374631+0000","last_change":"2024-04-25T05:43:16.214149+0000","last_active":"2024-04-25T05:44:13.374631+0000","last_peered":"2024-04-25T05:44:13.374631+0000","last_clean":"2024-04-25T05:44:13.374631+0000","last_became_active":"2024-04-25T05:43:16.213794+0000","last_became_peered":"2024-04-25T05:43:16.213794+0000","last_unstale":"2024-04-25T05:44:13.374631+0000","last_undegraded":"2024-04-25T05:44:13.374631+0000","last_fullsized":"2024-04-25T05:44:13.374631+0000","mapping_epoch":301,"log_start":"306'1200","ondisk_log_start":"306'1200","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:14.818306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358481920,"num_objects":5470,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5470,"num_write_kb":350080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":415,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.093333+0000","last_change":"2024-04-25T05:44:08.268300+0000","last_active":"2024-04-25T05:44:10.093333+0000","last_peered":"2024-04-25T05:44:10.093333+0000","last_clean":"2024-04-25T05:44:10.093333+0000","last_became_active":"2024-04-25T05:43:16.208005+0000","last_became_peered":"2024-04-25T05:43:16.208005+0000","last_unstale":"2024-04-25T05:44:10.093333+0000","last_undegraded":"2024-04-25T05:44:10.093333+0000","last_fullsized":"2024-04-25T05:44:10.093333+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.792885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":81,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086608+0000","last_change":"2024-04-25T05:43:16.251592+0000","last_active":"2024-04-25T05:44:10.086608+0000","last_peered":"2024-04-25T05:44:10.086608+0000","last_clean":"2024-04-25T05:44:10.086608+0000","last_became_active":"2024-04-25T05:43:16.209718+0000","last_became_peered":"2024-04-25T05:43:16.209718+0000","last_unstale":"2024-04-25T05:44:10.086608+0000","last_undegraded":"2024-04-25T05:44:10.086608+0000","last_fullsized":"2024-04-25T05:44:10.086608+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T05:40:12.370574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:12.370574+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1193,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.006605+0000","last_change":"2024-04-25T05:44:12.006605+0000","last_active":"2024-04-25T05:44:12.006605+0000","last_peered":"2024-04-25T05:44:12.006605+0000","last_clean":"2024-04-25T05:44:12.006605+0000","last_became_active":"2024-04-25T05:40:43.431415+0000","last_became_peered":"2024-04-25T05:40:43.431415+0000","last_unstale":"2024-04-25T05:44:12.006605+0000","last_undegraded":"2024-04-25T05:44:12.006605+0000","last_fullsized":"2024-04-25T05:44:12.006605+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:41.563201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":601,"num_read_kb":596,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":196,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086020+0000","last_change":"2024-04-25T05:44:06.421323+0000","last_active":"2024-04-25T05:44:10.086020+0000","last_peered":"2024-04-25T05:44:10.086020+0000","last_clean":"2024-04-25T05:44:10.086020+0000","last_became_active":"2024-04-25T05:40:43.434912+0000","last_became_peered":"2024-04-25T05:40:43.434912+0000","last_unstale":"2024-04-25T05:44:10.086020+0000","last_undegraded":"2024-04-25T05:44:10.086020+0000","last_fullsized":"2024-04-25T05:44:10.086020+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.421093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.421093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.839472+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":540,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091404+0000","last_change":"2024-04-25T05:43:16.189190+0000","last_active":"2024-04-25T05:44:10.091404+0000","last_peered":"2024-04-25T05:44:10.091404+0000","last_clean":"2024-04-25T05:44:10.091404+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:44:10.091404+0000","last_undegraded":"2024-04-25T05:44:10.091404+0000","last_fullsized":"2024-04-25T05:44:10.091404+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":728,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092168+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:44:10.092168+0000","last_peered":"2024-04-25T05:44:10.092168+0000","last_clean":"2024-04-25T05:44:10.092168+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:44:10.092168+0000","last_undegraded":"2024-04-25T05:44:10.092168+0000","last_fullsized":"2024-04-25T05:44:10.092168+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"321'5472","reported_seq":5537,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.562542+0000","last_change":"2024-04-25T05:43:16.198851+0000","last_active":"2024-04-25T05:44:13.562542+0000","last_peered":"2024-04-25T05:44:13.562542+0000","last_clean":"2024-04-25T05:44:13.562542+0000","last_became_active":"2024-04-25T05:43:16.198456+0000","last_became_peered":"2024-04-25T05:43:16.198456+0000","last_unstale":"2024-04-25T05:44:13.562542+0000","last_undegraded":"2024-04-25T05:44:13.562542+0000","last_fullsized":"2024-04-25T05:44:13.562542+0000","mapping_epoch":301,"log_start":"307'1800","ondisk_log_start":"307'1800","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:19.467166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358612992,"num_objects":5472,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":350208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":558,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082199+0000","last_change":"2024-04-25T05:44:06.090429+0000","last_active":"2024-04-25T05:44:10.082199+0000","last_peered":"2024-04-25T05:44:10.082199+0000","last_clean":"2024-04-25T05:44:10.082199+0000","last_became_active":"2024-04-25T05:43:16.204864+0000","last_became_peered":"2024-04-25T05:43:16.204864+0000","last_unstale":"2024-04-25T05:44:10.082199+0000","last_undegraded":"2024-04-25T05:44:10.082199+0000","last_fullsized":"2024-04-25T05:44:10.082199+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.724107+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":354,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091880+0000","last_change":"2024-04-25T05:43:22.635428+0000","last_active":"2024-04-25T05:44:10.091880+0000","last_peered":"2024-04-25T05:44:10.091880+0000","last_clean":"2024-04-25T05:44:10.091880+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:44:10.091880+0000","last_undegraded":"2024-04-25T05:44:10.091880+0000","last_fullsized":"2024-04-25T05:44:10.091880+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:43:22.635290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.635290+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:39.347414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":556,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084399+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:44:10.084399+0000","last_peered":"2024-04-25T05:44:10.084399+0000","last_clean":"2024-04-25T05:44:10.084399+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:44:10.084399+0000","last_undegraded":"2024-04-25T05:44:10.084399+0000","last_fullsized":"2024-04-25T05:44:10.084399+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":855,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081198+0000","last_change":"2024-04-25T05:42:18.235935+0000","last_active":"2024-04-25T05:44:10.081198+0000","last_peered":"2024-04-25T05:44:10.081198+0000","last_clean":"2024-04-25T05:44:10.081198+0000","last_became_active":"2024-04-25T05:40:45.847131+0000","last_became_peered":"2024-04-25T05:40:45.847131+0000","last_unstale":"2024-04-25T05:44:10.081198+0000","last_undegraded":"2024-04-25T05:44:10.081198+0000","last_fullsized":"2024-04-25T05:44:10.081198+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":50,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083706+0000","last_change":"2024-04-25T05:43:16.188274+0000","last_active":"2024-04-25T05:44:10.083706+0000","last_peered":"2024-04-25T05:44:10.083706+0000","last_clean":"2024-04-25T05:44:10.083706+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:44:10.083706+0000","last_undegraded":"2024-04-25T05:44:10.083706+0000","last_fullsized":"2024-04-25T05:44:10.083706+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"321'5535","reported_seq":5585,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.532265+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:44:13.532265+0000","last_peered":"2024-04-25T05:44:13.532265+0000","last_clean":"2024-04-25T05:44:13.532265+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:44:13.532265+0000","last_undegraded":"2024-04-25T05:44:13.532265+0000","last_fullsized":"2024-04-25T05:44:13.532265+0000","mapping_epoch":301,"log_start":"308'2100","ondisk_log_start":"308'2100","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:24.678864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362741760,"num_objects":5535,"num_object_clones":0,"num_object_copies":16605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5535,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":60,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083011+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:44:10.083011+0000","last_peered":"2024-04-25T05:44:10.083011+0000","last_clean":"2024-04-25T05:44:10.083011+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:44:10.083011+0000","last_undegraded":"2024-04-25T05:44:10.083011+0000","last_fullsized":"2024-04-25T05:44:10.083011+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:41.672026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":653,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092773+0000","last_change":"2024-04-25T05:43:22.350649+0000","last_active":"2024-04-25T05:44:10.092773+0000","last_peered":"2024-04-25T05:44:10.092773+0000","last_clean":"2024-04-25T05:44:10.092773+0000","last_became_active":"2024-04-25T05:43:16.207521+0000","last_became_peered":"2024-04-25T05:43:16.207521+0000","last_unstale":"2024-04-25T05:44:10.092773+0000","last_undegraded":"2024-04-25T05:44:10.092773+0000","last_fullsized":"2024-04-25T05:44:10.092773+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:36.531921+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":560,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084539+0000","last_change":"2024-04-25T05:42:55.459163+0000","last_active":"2024-04-25T05:44:10.084539+0000","last_peered":"2024-04-25T05:44:10.084539+0000","last_clean":"2024-04-25T05:44:10.084539+0000","last_became_active":"2024-04-25T05:40:55.990929+0000","last_became_peered":"2024-04-25T05:40:55.990929+0000","last_unstale":"2024-04-25T05:44:10.084539+0000","last_undegraded":"2024-04-25T05:44:10.084539+0000","last_fullsized":"2024-04-25T05:44:10.084539+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":579,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.364610+0000","last_change":"2024-04-25T05:44:13.364646+0000","last_active":"2024-04-25T05:44:13.364610+0000","last_peered":"2024-04-25T05:44:13.364610+0000","last_clean":"2024-04-25T05:44:13.364610+0000","last_became_active":"2024-04-25T05:40:55.966367+0000","last_became_peered":"2024-04-25T05:40:55.966367+0000","last_unstale":"2024-04-25T05:44:13.364610+0000","last_undegraded":"2024-04-25T05:44:13.364610+0000","last_fullsized":"2024-04-25T05:44:13.364610+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.831192+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":554,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082241+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:44:10.082241+0000","last_peered":"2024-04-25T05:44:10.082241+0000","last_clean":"2024-04-25T05:44:10.082241+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:44:10.082241+0000","last_undegraded":"2024-04-25T05:44:10.082241+0000","last_fullsized":"2024-04-25T05:44:10.082241+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:52.596050+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":853,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081802+0000","last_change":"2024-04-25T05:41:44.999875+0000","last_active":"2024-04-25T05:44:10.081802+0000","last_peered":"2024-04-25T05:44:10.081802+0000","last_clean":"2024-04-25T05:44:10.081802+0000","last_became_active":"2024-04-25T05:40:45.848289+0000","last_became_peered":"2024-04-25T05:40:45.848289+0000","last_unstale":"2024-04-25T05:44:10.081802+0000","last_undegraded":"2024-04-25T05:44:10.081802+0000","last_fullsized":"2024-04-25T05:44:10.081802+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"321'5490","reported_seq":5556,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.387323+0000","last_change":"2024-04-25T05:43:16.205376+0000","last_active":"2024-04-25T05:44:13.387323+0000","last_peered":"2024-04-25T05:44:13.387323+0000","last_clean":"2024-04-25T05:44:13.387323+0000","last_became_active":"2024-04-25T05:43:16.205062+0000","last_became_peered":"2024-04-25T05:43:16.205062+0000","last_unstale":"2024-04-25T05:44:13.387323+0000","last_undegraded":"2024-04-25T05:44:13.387323+0000","last_fullsized":"2024-04-25T05:44:13.387323+0000","mapping_epoch":301,"log_start":"306'1200","ondisk_log_start":"306'1200","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:24.824330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359792640,"num_objects":5490,"num_object_clones":0,"num_object_copies":16470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5490,"num_write_kb":351360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":188,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086596+0000","last_change":"2024-04-25T05:40:43.442114+0000","last_active":"2024-04-25T05:44:10.086596+0000","last_peered":"2024-04-25T05:44:10.086596+0000","last_clean":"2024-04-25T05:44:10.086596+0000","last_became_active":"2024-04-25T05:40:43.441816+0000","last_became_peered":"2024-04-25T05:40:43.441816+0000","last_unstale":"2024-04-25T05:44:10.086596+0000","last_undegraded":"2024-04-25T05:44:10.086596+0000","last_fullsized":"2024-04-25T05:44:10.086596+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":784,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091941+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:44:10.091941+0000","last_peered":"2024-04-25T05:44:10.091941+0000","last_clean":"2024-04-25T05:44:10.091941+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:44:10.091941+0000","last_undegraded":"2024-04-25T05:44:10.091941+0000","last_fullsized":"2024-04-25T05:44:10.091941+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:20.547718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":196,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086581+0000","last_change":"2024-04-25T05:44:08.448017+0000","last_active":"2024-04-25T05:44:10.086581+0000","last_peered":"2024-04-25T05:44:10.086581+0000","last_clean":"2024-04-25T05:44:10.086581+0000","last_became_active":"2024-04-25T05:40:43.434803+0000","last_became_peered":"2024-04-25T05:40:43.434803+0000","last_unstale":"2024-04-25T05:44:10.086581+0000","last_undegraded":"2024-04-25T05:44:10.086581+0000","last_fullsized":"2024-04-25T05:44:10.086581+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:36.795014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":76,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.093309+0000","last_change":"2024-04-25T05:43:16.281817+0000","last_active":"2024-04-25T05:44:10.093309+0000","last_peered":"2024-04-25T05:44:10.093309+0000","last_clean":"2024-04-25T05:44:10.093309+0000","last_became_active":"2024-04-25T05:43:16.210096+0000","last_became_peered":"2024-04-25T05:43:16.210096+0000","last_unstale":"2024-04-25T05:44:10.093309+0000","last_undegraded":"2024-04-25T05:44:10.093309+0000","last_fullsized":"2024-04-25T05:44:10.093309+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":526,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082959+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:44:10.082959+0000","last_peered":"2024-04-25T05:44:10.082959+0000","last_clean":"2024-04-25T05:44:10.082959+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:44:10.082959+0000","last_undegraded":"2024-04-25T05:44:10.082959+0000","last_fullsized":"2024-04-25T05:44:10.082959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":596,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.499549+0000","last_change":"2024-04-25T05:44:10.499596+0000","last_active":"2024-04-25T05:44:10.499549+0000","last_peered":"2024-04-25T05:44:10.499549+0000","last_clean":"2024-04-25T05:44:10.499549+0000","last_became_active":"2024-04-25T05:43:19.208047+0000","last_became_peered":"2024-04-25T05:43:19.208047+0000","last_unstale":"2024-04-25T05:44:10.499549+0000","last_undegraded":"2024-04-25T05:44:10.499549+0000","last_fullsized":"2024-04-25T05:44:10.499549+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.385958+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.6","version":"321'5437","reported_seq":5487,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.532136+0000","last_change":"2024-04-25T05:43:16.208811+0000","last_active":"2024-04-25T05:44:13.532136+0000","last_peered":"2024-04-25T05:44:13.532136+0000","last_clean":"2024-04-25T05:44:13.532136+0000","last_became_active":"2024-04-25T05:43:16.208505+0000","last_became_peered":"2024-04-25T05:43:16.208505+0000","last_unstale":"2024-04-25T05:44:13.532136+0000","last_undegraded":"2024-04-25T05:44:13.532136+0000","last_fullsized":"2024-04-25T05:44:13.532136+0000","mapping_epoch":301,"log_start":"308'2000","ondisk_log_start":"308'2000","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3437,"log_dups_size":0,"ondisk_log_size":3437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:30.172685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356319232,"num_objects":5437,"num_object_clones":0,"num_object_copies":16311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5437,"num_write_kb":347968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":253,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086675+0000","last_change":"2024-04-25T05:43:16.552278+0000","last_active":"2024-04-25T05:44:10.086675+0000","last_peered":"2024-04-25T05:44:10.086675+0000","last_clean":"2024-04-25T05:44:10.086675+0000","last_became_active":"2024-04-25T05:43:16.210660+0000","last_became_peered":"2024-04-25T05:43:16.210660+0000","last_unstale":"2024-04-25T05:44:10.086675+0000","last_undegraded":"2024-04-25T05:44:10.086675+0000","last_fullsized":"2024-04-25T05:44:10.086675+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:37.761552+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":356,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084290+0000","last_change":"2024-04-25T05:43:16.191937+0000","last_active":"2024-04-25T05:44:10.084290+0000","last_peered":"2024-04-25T05:44:10.084290+0000","last_clean":"2024-04-25T05:44:10.084290+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:44:10.084290+0000","last_undegraded":"2024-04-25T05:44:10.084290+0000","last_fullsized":"2024-04-25T05:44:10.084290+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":189,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076344+0000","last_change":"2024-04-25T05:44:08.186354+0000","last_active":"2024-04-25T05:44:10.076344+0000","last_peered":"2024-04-25T05:44:10.076344+0000","last_clean":"2024-04-25T05:44:10.076344+0000","last_became_active":"2024-04-25T05:40:55.981082+0000","last_became_peered":"2024-04-25T05:40:55.981082+0000","last_unstale":"2024-04-25T05:44:10.076344+0000","last_undegraded":"2024-04-25T05:44:10.076344+0000","last_fullsized":"2024-04-25T05:44:10.076344+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.186275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.186275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.359403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":208,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.085912+0000","last_change":"2024-04-25T05:42:54.545923+0000","last_active":"2024-04-25T05:44:10.085912+0000","last_peered":"2024-04-25T05:44:10.085912+0000","last_clean":"2024-04-25T05:44:10.085912+0000","last_became_active":"2024-04-25T05:42:34.577940+0000","last_became_peered":"2024-04-25T05:42:34.577940+0000","last_unstale":"2024-04-25T05:44:10.085912+0000","last_undegraded":"2024-04-25T05:44:10.085912+0000","last_fullsized":"2024-04-25T05:44:10.085912+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:18.163256+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"321'5339","reported_seq":5389,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.850064+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:44:12.850064+0000","last_peered":"2024-04-25T05:44:12.850064+0000","last_clean":"2024-04-25T05:44:12.850064+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:44:12.850064+0000","last_undegraded":"2024-04-25T05:44:12.850064+0000","last_fullsized":"2024-04-25T05:44:12.850064+0000","mapping_epoch":301,"log_start":"306'1600","ondisk_log_start":"306'1600","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3739,"log_dups_size":0,"ondisk_log_size":3739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:29.876089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349896704,"num_objects":5339,"num_object_clones":0,"num_object_copies":16017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":341696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":316,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091096+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:44:10.091096+0000","last_peered":"2024-04-25T05:44:10.091096+0000","last_clean":"2024-04-25T05:44:10.091096+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:44:10.091096+0000","last_undegraded":"2024-04-25T05:44:10.091096+0000","last_fullsized":"2024-04-25T05:44:10.091096+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:46.695440+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":590,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082140+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:44:10.082140+0000","last_peered":"2024-04-25T05:44:10.082140+0000","last_clean":"2024-04-25T05:44:10.082140+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:44:10.082140+0000","last_undegraded":"2024-04-25T05:44:10.082140+0000","last_fullsized":"2024-04-25T05:44:10.082140+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:16.999724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.085884+0000","last_change":"2024-04-25T05:44:03.424825+0000","last_active":"2024-04-25T05:44:10.085884+0000","last_peered":"2024-04-25T05:44:10.085884+0000","last_clean":"2024-04-25T05:44:10.085884+0000","last_became_active":"2024-04-25T05:43:16.215373+0000","last_became_peered":"2024-04-25T05:43:16.215373+0000","last_unstale":"2024-04-25T05:44:10.085884+0000","last_undegraded":"2024-04-25T05:44:10.085884+0000","last_fullsized":"2024-04-25T05:44:10.085884+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.606368+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":548,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081055+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:44:10.081055+0000","last_peered":"2024-04-25T05:44:10.081055+0000","last_clean":"2024-04-25T05:44:10.081055+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:44:10.081055+0000","last_undegraded":"2024-04-25T05:44:10.081055+0000","last_fullsized":"2024-04-25T05:44:10.081055+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.832260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"321'5501","reported_seq":5560,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.517657+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:44:13.517657+0000","last_peered":"2024-04-25T05:44:13.517657+0000","last_clean":"2024-04-25T05:44:13.517657+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:44:13.517657+0000","last_undegraded":"2024-04-25T05:44:13.517657+0000","last_fullsized":"2024-04-25T05:44:13.517657+0000","mapping_epoch":296,"log_start":"308'2000","ondisk_log_start":"308'2000","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:27.447307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360513536,"num_objects":5501,"num_object_clones":0,"num_object_copies":16503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5501,"num_write_kb":352064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":551,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090613+0000","last_change":"2024-04-25T05:43:29.464978+0000","last_active":"2024-04-25T05:44:10.090613+0000","last_peered":"2024-04-25T05:44:10.090613+0000","last_clean":"2024-04-25T05:44:10.090613+0000","last_became_active":"2024-04-25T05:43:16.213590+0000","last_became_peered":"2024-04-25T05:43:16.213590+0000","last_unstale":"2024-04-25T05:44:10.090613+0000","last_undegraded":"2024-04-25T05:44:10.090613+0000","last_fullsized":"2024-04-25T05:44:10.090613+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:29.957096+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":204,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.362219+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:44:12.362219+0000","last_peered":"2024-04-25T05:44:12.362219+0000","last_clean":"2024-04-25T05:44:12.362219+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:44:12.362219+0000","last_undegraded":"2024-04-25T05:44:12.362219+0000","last_fullsized":"2024-04-25T05:44:12.362219+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.771369+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":554,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082273+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:44:10.082273+0000","last_peered":"2024-04-25T05:44:10.082273+0000","last_clean":"2024-04-25T05:44:10.082273+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:44:10.082273+0000","last_undegraded":"2024-04-25T05:44:10.082273+0000","last_fullsized":"2024-04-25T05:44:10.082273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082327+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:44:10.082327+0000","last_peered":"2024-04-25T05:44:10.082327+0000","last_clean":"2024-04-25T05:44:10.082327+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:44:10.082327+0000","last_undegraded":"2024-04-25T05:44:10.082327+0000","last_fullsized":"2024-04-25T05:44:10.082327+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:33.061319+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"321'5492","reported_seq":5558,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:13.380222+0000","last_change":"2024-04-25T05:43:16.209790+0000","last_active":"2024-04-25T05:44:13.380222+0000","last_peered":"2024-04-25T05:44:13.380222+0000","last_clean":"2024-04-25T05:44:13.380222+0000","last_became_active":"2024-04-25T05:43:16.209459+0000","last_became_peered":"2024-04-25T05:43:16.209459+0000","last_unstale":"2024-04-25T05:44:13.380222+0000","last_undegraded":"2024-04-25T05:44:13.380222+0000","last_fullsized":"2024-04-25T05:44:13.380222+0000","mapping_epoch":301,"log_start":"306'1200","ondisk_log_start":"306'1200","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.628599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359923712,"num_objects":5492,"num_object_clones":0,"num_object_copies":16476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":351488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":701,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083087+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:44:10.083087+0000","last_peered":"2024-04-25T05:44:10.083087+0000","last_clean":"2024-04-25T05:44:10.083087+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:44:10.083087+0000","last_undegraded":"2024-04-25T05:44:10.083087+0000","last_fullsized":"2024-04-25T05:44:10.083087+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":156,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081328+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:44:10.081328+0000","last_peered":"2024-04-25T05:44:10.081328+0000","last_clean":"2024-04-25T05:44:10.081328+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:44:10.081328+0000","last_undegraded":"2024-04-25T05:44:10.081328+0000","last_fullsized":"2024-04-25T05:44:10.081328+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:21.393615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":590,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082005+0000","last_change":"2024-04-25T05:44:07.061732+0000","last_active":"2024-04-25T05:44:10.082005+0000","last_peered":"2024-04-25T05:44:10.082005+0000","last_clean":"2024-04-25T05:44:10.082005+0000","last_became_active":"2024-04-25T05:40:43.429608+0000","last_became_peered":"2024-04-25T05:40:43.429608+0000","last_unstale":"2024-04-25T05:44:10.082005+0000","last_undegraded":"2024-04-25T05:44:10.082005+0000","last_fullsized":"2024-04-25T05:44:10.082005+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:25.204462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":423,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090967+0000","last_change":"2024-04-25T05:43:26.676651+0000","last_active":"2024-04-25T05:44:10.090967+0000","last_peered":"2024-04-25T05:44:10.090967+0000","last_clean":"2024-04-25T05:44:10.090967+0000","last_became_active":"2024-04-25T05:43:16.199855+0000","last_became_peered":"2024-04-25T05:43:16.199855+0000","last_unstale":"2024-04-25T05:44:10.090967+0000","last_undegraded":"2024-04-25T05:44:10.090967+0000","last_fullsized":"2024-04-25T05:44:10.090967+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:50.459626+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":572,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:12.039020+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:44:12.039020+0000","last_peered":"2024-04-25T05:44:12.039020+0000","last_clean":"2024-04-25T05:44:12.039020+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:44:12.039020+0000","last_undegraded":"2024-04-25T05:44:12.039020+0000","last_fullsized":"2024-04-25T05:44:12.039020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.253497+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":282,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092163+0000","last_change":"2024-04-25T05:44:07.716378+0000","last_active":"2024-04-25T05:44:10.092163+0000","last_peered":"2024-04-25T05:44:10.092163+0000","last_clean":"2024-04-25T05:44:10.092163+0000","last_became_active":"2024-04-25T05:43:16.200632+0000","last_became_peered":"2024-04-25T05:43:16.200632+0000","last_unstale":"2024-04-25T05:44:10.092163+0000","last_undegraded":"2024-04-25T05:44:10.092163+0000","last_fullsized":"2024-04-25T05:44:10.092163+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.342200+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082436+0000","last_change":"2024-04-25T05:43:16.190945+0000","last_active":"2024-04-25T05:44:10.082436+0000","last_peered":"2024-04-25T05:44:10.082436+0000","last_clean":"2024-04-25T05:44:10.082436+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:44:10.082436+0000","last_undegraded":"2024-04-25T05:44:10.082436+0000","last_fullsized":"2024-04-25T05:44:10.082436+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:46.378394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:40:46.378394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":554,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081739+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:44:10.081739+0000","last_peered":"2024-04-25T05:44:10.081739+0000","last_clean":"2024-04-25T05:44:10.081739+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:44:10.081739+0000","last_undegraded":"2024-04-25T05:44:10.081739+0000","last_fullsized":"2024-04-25T05:44:10.081739+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:06.898722+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":219,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086808+0000","last_change":"2024-04-25T05:42:21.659494+0000","last_active":"2024-04-25T05:44:10.086808+0000","last_peered":"2024-04-25T05:44:10.086808+0000","last_clean":"2024-04-25T05:44:10.086808+0000","last_became_active":"2024-04-25T05:40:43.441558+0000","last_became_peered":"2024-04-25T05:40:43.441558+0000","last_unstale":"2024-04-25T05:44:10.086808+0000","last_undegraded":"2024-04-25T05:44:10.086808+0000","last_fullsized":"2024-04-25T05:44:10.086808+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":572,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082990+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:44:10.082990+0000","last_peered":"2024-04-25T05:44:10.082990+0000","last_clean":"2024-04-25T05:44:10.082990+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:44:10.082990+0000","last_undegraded":"2024-04-25T05:44:10.082990+0000","last_fullsized":"2024-04-25T05:44:10.082990+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:59.461950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":522,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083808+0000","last_change":"2024-04-25T05:40:38.559014+0000","last_active":"2024-04-25T05:44:10.083808+0000","last_peered":"2024-04-25T05:44:10.083808+0000","last_clean":"2024-04-25T05:44:10.083808+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:44:10.083808+0000","last_undegraded":"2024-04-25T05:44:10.083808+0000","last_fullsized":"2024-04-25T05:44:10.083808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:38.558895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:38.558895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":190,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.077093+0000","last_change":"2024-04-25T05:44:07.165732+0000","last_active":"2024-04-25T05:44:10.077093+0000","last_peered":"2024-04-25T05:44:10.077093+0000","last_clean":"2024-04-25T05:44:10.077093+0000","last_became_active":"2024-04-25T05:40:55.953924+0000","last_became_peered":"2024-04-25T05:40:55.953924+0000","last_unstale":"2024-04-25T05:44:10.077093+0000","last_undegraded":"2024-04-25T05:44:10.077093+0000","last_fullsized":"2024-04-25T05:44:10.077093+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:14.081959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":175,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082240+0000","last_change":"2024-04-25T05:43:16.229957+0000","last_active":"2024-04-25T05:44:10.082240+0000","last_peered":"2024-04-25T05:44:10.082240+0000","last_clean":"2024-04-25T05:44:10.082240+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:44:10.082240+0000","last_undegraded":"2024-04-25T05:44:10.082240+0000","last_fullsized":"2024-04-25T05:44:10.082240+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":196,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086150+0000","last_change":"2024-04-25T05:44:02.458219+0000","last_active":"2024-04-25T05:44:10.086150+0000","last_peered":"2024-04-25T05:44:10.086150+0000","last_clean":"2024-04-25T05:44:10.086150+0000","last_became_active":"2024-04-25T05:40:43.433933+0000","last_became_peered":"2024-04-25T05:40:43.433933+0000","last_unstale":"2024-04-25T05:44:10.086150+0000","last_undegraded":"2024-04-25T05:44:10.086150+0000","last_fullsized":"2024-04-25T05:44:10.086150+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.911945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":671,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:11.019834+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:44:11.019834+0000","last_peered":"2024-04-25T05:44:11.019834+0000","last_clean":"2024-04-25T05:44:11.019834+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:44:11.019834+0000","last_undegraded":"2024-04-25T05:44:11.019834+0000","last_fullsized":"2024-04-25T05:44:11.019834+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.732091+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086145+0000","last_change":"2024-04-25T05:44:04.540470+0000","last_active":"2024-04-25T05:44:10.086145+0000","last_peered":"2024-04-25T05:44:10.086145+0000","last_clean":"2024-04-25T05:44:10.086145+0000","last_became_active":"2024-04-25T05:43:16.197258+0000","last_became_peered":"2024-04-25T05:43:16.197258+0000","last_unstale":"2024-04-25T05:44:10.086145+0000","last_undegraded":"2024-04-25T05:44:10.086145+0000","last_fullsized":"2024-04-25T05:44:10.086145+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.222221+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":588,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083514+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:44:10.083514+0000","last_peered":"2024-04-25T05:44:10.083514+0000","last_clean":"2024-04-25T05:44:10.083514+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:44:10.083514+0000","last_undegraded":"2024-04-25T05:44:10.083514+0000","last_fullsized":"2024-04-25T05:44:10.083514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081729+0000","last_change":"2024-04-25T05:41:37.012236+0000","last_active":"2024-04-25T05:44:10.081729+0000","last_peered":"2024-04-25T05:44:10.081729+0000","last_clean":"2024-04-25T05:44:10.081729+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:44:10.081729+0000","last_undegraded":"2024-04-25T05:44:10.081729+0000","last_fullsized":"2024-04-25T05:44:10.081729+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:37.012159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:41:37.012159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"304'11","reported_seq":89,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095544+0000","last_change":"2024-04-25T05:43:16.255626+0000","last_active":"2024-04-25T05:44:10.095544+0000","last_peered":"2024-04-25T05:44:10.095544+0000","last_clean":"2024-04-25T05:44:10.095544+0000","last_became_active":"2024-04-25T05:43:16.207548+0000","last_became_peered":"2024-04-25T05:43:16.207548+0000","last_unstale":"2024-04-25T05:44:10.095544+0000","last_undegraded":"2024-04-25T05:44:10.095544+0000","last_fullsized":"2024-04-25T05:44:10.095544+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":196,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086914+0000","last_change":"2024-04-25T05:43:24.341293+0000","last_active":"2024-04-25T05:44:10.086914+0000","last_peered":"2024-04-25T05:44:10.086914+0000","last_clean":"2024-04-25T05:44:10.086914+0000","last_became_active":"2024-04-25T05:40:43.439460+0000","last_became_peered":"2024-04-25T05:40:43.439460+0000","last_unstale":"2024-04-25T05:44:10.086914+0000","last_undegraded":"2024-04-25T05:44:10.086914+0000","last_fullsized":"2024-04-25T05:44:10.086914+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.400587+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":658,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.077359+0000","last_change":"2024-04-25T05:42:49.238116+0000","last_active":"2024-04-25T05:44:10.077359+0000","last_peered":"2024-04-25T05:44:10.077359+0000","last_clean":"2024-04-25T05:44:10.077359+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:44:10.077359+0000","last_undegraded":"2024-04-25T05:44:10.077359+0000","last_fullsized":"2024-04-25T05:44:10.077359+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'16","last_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.238022+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:10.314687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":225,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081843+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:44:10.081843+0000","last_peered":"2024-04-25T05:44:10.081843+0000","last_clean":"2024-04-25T05:44:10.081843+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:44:10.081843+0000","last_undegraded":"2024-04-25T05:44:10.081843+0000","last_fullsized":"2024-04-25T05:44:10.081843+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":547,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.093668+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:44:10.093668+0000","last_peered":"2024-04-25T05:44:10.093668+0000","last_clean":"2024-04-25T05:44:10.093668+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:44:10.093668+0000","last_undegraded":"2024-04-25T05:44:10.093668+0000","last_fullsized":"2024-04-25T05:44:10.093668+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:42.530876+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084422+0000","last_change":"2024-04-25T05:43:24.480399+0000","last_active":"2024-04-25T05:44:10.084422+0000","last_peered":"2024-04-25T05:44:10.084422+0000","last_clean":"2024-04-25T05:44:10.084422+0000","last_became_active":"2024-04-25T05:43:16.194169+0000","last_became_peered":"2024-04-25T05:43:16.194169+0000","last_unstale":"2024-04-25T05:44:10.084422+0000","last_undegraded":"2024-04-25T05:44:10.084422+0000","last_fullsized":"2024-04-25T05:44:10.084422+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:33.168704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":68,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092699+0000","last_change":"2024-04-25T05:43:16.278532+0000","last_active":"2024-04-25T05:44:10.092699+0000","last_peered":"2024-04-25T05:44:10.092699+0000","last_clean":"2024-04-25T05:44:10.092699+0000","last_became_active":"2024-04-25T05:43:16.209844+0000","last_became_peered":"2024-04-25T05:43:16.209844+0000","last_unstale":"2024-04-25T05:44:10.092699+0000","last_undegraded":"2024-04-25T05:44:10.092699+0000","last_fullsized":"2024-04-25T05:44:10.092699+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":554,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081854+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:44:10.081854+0000","last_peered":"2024-04-25T05:44:10.081854+0000","last_clean":"2024-04-25T05:44:10.081854+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:44:10.081854+0000","last_undegraded":"2024-04-25T05:44:10.081854+0000","last_fullsized":"2024-04-25T05:44:10.081854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":542,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081829+0000","last_change":"2024-04-25T05:44:07.993587+0000","last_active":"2024-04-25T05:44:10.081829+0000","last_peered":"2024-04-25T05:44:10.081829+0000","last_clean":"2024-04-25T05:44:10.081829+0000","last_became_active":"2024-04-25T05:40:43.433240+0000","last_became_peered":"2024-04-25T05:40:43.433240+0000","last_unstale":"2024-04-25T05:44:10.081829+0000","last_undegraded":"2024-04-25T05:44:10.081829+0000","last_fullsized":"2024-04-25T05:44:10.081829+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.460346+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":202,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090504+0000","last_change":"2024-04-25T05:43:16.210785+0000","last_active":"2024-04-25T05:44:10.090504+0000","last_peered":"2024-04-25T05:44:10.090504+0000","last_clean":"2024-04-25T05:44:10.090504+0000","last_became_active":"2024-04-25T05:43:16.210171+0000","last_became_peered":"2024-04-25T05:43:16.210171+0000","last_unstale":"2024-04-25T05:44:10.090504+0000","last_undegraded":"2024-04-25T05:44:10.090504+0000","last_fullsized":"2024-04-25T05:44:10.090504+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":617,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084947+0000","last_change":"2024-04-25T05:40:49.558496+0000","last_active":"2024-04-25T05:44:10.084947+0000","last_peered":"2024-04-25T05:44:10.084947+0000","last_clean":"2024-04-25T05:44:10.084947+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:44:10.084947+0000","last_undegraded":"2024-04-25T05:44:10.084947+0000","last_fullsized":"2024-04-25T05:44:10.084947+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:40:49.558354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:49.558354+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082238+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:44:10.082238+0000","last_peered":"2024-04-25T05:44:10.082238+0000","last_clean":"2024-04-25T05:44:10.082238+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:44:10.082238+0000","last_undegraded":"2024-04-25T05:44:10.082238+0000","last_fullsized":"2024-04-25T05:44:10.082238+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:42.578695+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":564,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081436+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:44:10.081436+0000","last_peered":"2024-04-25T05:44:10.081436+0000","last_clean":"2024-04-25T05:44:10.081436+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:44:10.081436+0000","last_undegraded":"2024-04-25T05:44:10.081436+0000","last_fullsized":"2024-04-25T05:44:10.081436+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.164571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083627+0000","last_change":"2024-04-25T05:42:20.379921+0000","last_active":"2024-04-25T05:44:10.083627+0000","last_peered":"2024-04-25T05:44:10.083627+0000","last_clean":"2024-04-25T05:44:10.083627+0000","last_became_active":"2024-04-25T05:40:43.435420+0000","last_became_peered":"2024-04-25T05:40:43.435420+0000","last_unstale":"2024-04-25T05:44:10.083627+0000","last_undegraded":"2024-04-25T05:44:10.083627+0000","last_fullsized":"2024-04-25T05:44:10.083627+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":570,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082449+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:44:10.082449+0000","last_peered":"2024-04-25T05:44:10.082449+0000","last_clean":"2024-04-25T05:44:10.082449+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:44:10.082449+0000","last_undegraded":"2024-04-25T05:44:10.082449+0000","last_fullsized":"2024-04-25T05:44:10.082449+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":202,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086476+0000","last_change":"2024-04-25T05:43:26.399573+0000","last_active":"2024-04-25T05:44:10.086476+0000","last_peered":"2024-04-25T05:44:10.086476+0000","last_clean":"2024-04-25T05:44:10.086476+0000","last_became_active":"2024-04-25T05:43:16.207279+0000","last_became_peered":"2024-04-25T05:43:16.207279+0000","last_unstale":"2024-04-25T05:44:10.086476+0000","last_undegraded":"2024-04-25T05:44:10.086476+0000","last_fullsized":"2024-04-25T05:44:10.086476+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:38.112644+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":262,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092168+0000","last_change":"2024-04-25T05:43:29.610487+0000","last_active":"2024-04-25T05:44:10.092168+0000","last_peered":"2024-04-25T05:44:10.092168+0000","last_clean":"2024-04-25T05:44:10.092168+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:44:10.092168+0000","last_undegraded":"2024-04-25T05:44:10.092168+0000","last_fullsized":"2024-04-25T05:44:10.092168+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.610411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.610411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:31.076176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":545,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081924+0000","last_change":"2024-04-25T05:41:39.031952+0000","last_active":"2024-04-25T05:44:10.081924+0000","last_peered":"2024-04-25T05:44:10.081924+0000","last_clean":"2024-04-25T05:44:10.081924+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:44:10.081924+0000","last_undegraded":"2024-04-25T05:44:10.081924+0000","last_fullsized":"2024-04-25T05:44:10.081924+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:41:39.031784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.031784+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":322,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091948+0000","last_change":"2024-04-25T05:44:03.712052+0000","last_active":"2024-04-25T05:44:10.091948+0000","last_peered":"2024-04-25T05:44:10.091948+0000","last_clean":"2024-04-25T05:44:10.091948+0000","last_became_active":"2024-04-25T05:43:16.200717+0000","last_became_peered":"2024-04-25T05:43:16.200717+0000","last_unstale":"2024-04-25T05:44:10.091948+0000","last_undegraded":"2024-04-25T05:44:10.091948+0000","last_fullsized":"2024-04-25T05:44:10.091948+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.510376+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":550,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082750+0000","last_change":"2024-04-25T05:40:50.562106+0000","last_active":"2024-04-25T05:44:10.082750+0000","last_peered":"2024-04-25T05:44:10.082750+0000","last_clean":"2024-04-25T05:44:10.082750+0000","last_became_active":"2024-04-25T05:40:43.440363+0000","last_became_peered":"2024-04-25T05:40:43.440363+0000","last_unstale":"2024-04-25T05:44:10.082750+0000","last_undegraded":"2024-04-25T05:44:10.082750+0000","last_fullsized":"2024-04-25T05:44:10.082750+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.560317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.560317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":98,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092163+0000","last_change":"2024-04-25T05:42:48.418293+0000","last_active":"2024-04-25T05:44:10.092163+0000","last_peered":"2024-04-25T05:44:10.092163+0000","last_clean":"2024-04-25T05:44:10.092163+0000","last_became_active":"2024-04-25T05:42:34.575586+0000","last_became_peered":"2024-04-25T05:42:34.575586+0000","last_unstale":"2024-04-25T05:44:10.092163+0000","last_undegraded":"2024-04-25T05:44:10.092163+0000","last_fullsized":"2024-04-25T05:44:10.092163+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.418180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.418180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:12.177812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":540,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095350+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:44:10.095350+0000","last_peered":"2024-04-25T05:44:10.095350+0000","last_clean":"2024-04-25T05:44:10.095350+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:44:10.095350+0000","last_undegraded":"2024-04-25T05:44:10.095350+0000","last_fullsized":"2024-04-25T05:44:10.095350+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:50.241779+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":225,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086784+0000","last_change":"2024-04-25T05:43:18.467174+0000","last_active":"2024-04-25T05:44:10.086784+0000","last_peered":"2024-04-25T05:44:10.086784+0000","last_clean":"2024-04-25T05:44:10.086784+0000","last_became_active":"2024-04-25T05:40:43.442720+0000","last_became_peered":"2024-04-25T05:40:43.442720+0000","last_unstale":"2024-04-25T05:44:10.086784+0000","last_undegraded":"2024-04-25T05:44:10.086784+0000","last_fullsized":"2024-04-25T05:44:10.086784+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:19.904994+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":547,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090997+0000","last_change":"2024-04-25T05:40:40.500687+0000","last_active":"2024-04-25T05:44:10.090997+0000","last_peered":"2024-04-25T05:44:10.090997+0000","last_clean":"2024-04-25T05:44:10.090997+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:44:10.090997+0000","last_undegraded":"2024-04-25T05:44:10.090997+0000","last_fullsized":"2024-04-25T05:44:10.090997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:40:40.500174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:40.500174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":330,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091756+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:44:10.091756+0000","last_peered":"2024-04-25T05:44:10.091756+0000","last_clean":"2024-04-25T05:44:10.091756+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:44:10.091756+0000","last_undegraded":"2024-04-25T05:44:10.091756+0000","last_fullsized":"2024-04-25T05:44:10.091756+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.185065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":563,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082194+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:44:10.082194+0000","last_peered":"2024-04-25T05:44:10.082194+0000","last_clean":"2024-04-25T05:44:10.082194+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:44:10.082194+0000","last_undegraded":"2024-04-25T05:44:10.082194+0000","last_fullsized":"2024-04-25T05:44:10.082194+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":578,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081719+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:44:10.081719+0000","last_peered":"2024-04-25T05:44:10.081719+0000","last_clean":"2024-04-25T05:44:10.081719+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:44:10.081719+0000","last_undegraded":"2024-04-25T05:44:10.081719+0000","last_fullsized":"2024-04-25T05:44:10.081719+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:44.866351+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":560,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076291+0000","last_change":"2024-04-25T05:43:27.192557+0000","last_active":"2024-04-25T05:44:10.076291+0000","last_peered":"2024-04-25T05:44:10.076291+0000","last_clean":"2024-04-25T05:44:10.076291+0000","last_became_active":"2024-04-25T05:40:55.964565+0000","last_became_peered":"2024-04-25T05:40:55.964565+0000","last_unstale":"2024-04-25T05:44:10.076291+0000","last_undegraded":"2024-04-25T05:44:10.076291+0000","last_fullsized":"2024-04-25T05:44:10.076291+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.192432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.192432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:42.924009+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":223,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084385+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:44:10.084385+0000","last_peered":"2024-04-25T05:44:10.084385+0000","last_clean":"2024-04-25T05:44:10.084385+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:44:10.084385+0000","last_undegraded":"2024-04-25T05:44:10.084385+0000","last_fullsized":"2024-04-25T05:44:10.084385+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":544,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.083916+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:44:10.083916+0000","last_peered":"2024-04-25T05:44:10.083916+0000","last_clean":"2024-04-25T05:44:10.083916+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:44:10.083916+0000","last_undegraded":"2024-04-25T05:44:10.083916+0000","last_fullsized":"2024-04-25T05:44:10.083916+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":423,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091680+0000","last_change":"2024-04-25T05:42:52.924801+0000","last_active":"2024-04-25T05:44:10.091680+0000","last_peered":"2024-04-25T05:44:10.091680+0000","last_clean":"2024-04-25T05:44:10.091680+0000","last_became_active":"2024-04-25T05:40:55.991291+0000","last_became_peered":"2024-04-25T05:40:55.991291+0000","last_unstale":"2024-04-25T05:44:10.091680+0000","last_undegraded":"2024-04-25T05:44:10.091680+0000","last_fullsized":"2024-04-25T05:44:10.091680+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":188,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086489+0000","last_change":"2024-04-25T05:40:43.435522+0000","last_active":"2024-04-25T05:44:10.086489+0000","last_peered":"2024-04-25T05:44:10.086489+0000","last_clean":"2024-04-25T05:44:10.086489+0000","last_became_active":"2024-04-25T05:40:43.434830+0000","last_became_peered":"2024-04-25T05:40:43.434830+0000","last_unstale":"2024-04-25T05:44:10.086489+0000","last_undegraded":"2024-04-25T05:44:10.086489+0000","last_fullsized":"2024-04-25T05:44:10.086489+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.750137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.750137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":558,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.095694+0000","last_change":"2024-04-25T05:44:02.270962+0000","last_active":"2024-04-25T05:44:10.095694+0000","last_peered":"2024-04-25T05:44:10.095694+0000","last_clean":"2024-04-25T05:44:10.095694+0000","last_became_active":"2024-04-25T05:40:43.449712+0000","last_became_peered":"2024-04-25T05:40:43.449712+0000","last_unstale":"2024-04-25T05:44:10.095694+0000","last_undegraded":"2024-04-25T05:44:10.095694+0000","last_fullsized":"2024-04-25T05:44:10.095694+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:07.080383+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":561,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076877+0000","last_change":"2024-04-25T05:44:00.169219+0000","last_active":"2024-04-25T05:44:10.076877+0000","last_peered":"2024-04-25T05:44:10.076877+0000","last_clean":"2024-04-25T05:44:10.076877+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:44:10.076877+0000","last_undegraded":"2024-04-25T05:44:10.076877+0000","last_fullsized":"2024-04-25T05:44:10.076877+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.169057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.169057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.657106+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":288,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081611+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:44:10.081611+0000","last_peered":"2024-04-25T05:44:10.081611+0000","last_clean":"2024-04-25T05:44:10.081611+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:44:10.081611+0000","last_undegraded":"2024-04-25T05:44:10.081611+0000","last_fullsized":"2024-04-25T05:44:10.081611+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":58,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.076646+0000","last_change":"2024-04-25T05:43:17.283718+0000","last_active":"2024-04-25T05:44:10.076646+0000","last_peered":"2024-04-25T05:44:10.076646+0000","last_clean":"2024-04-25T05:44:10.076646+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:44:10.076646+0000","last_undegraded":"2024-04-25T05:44:10.076646+0000","last_fullsized":"2024-04-25T05:44:10.076646+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.283618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.283618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:39.318852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081595+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:44:10.081595+0000","last_peered":"2024-04-25T05:44:10.081595+0000","last_clean":"2024-04-25T05:44:10.081595+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:44:10.081595+0000","last_undegraded":"2024-04-25T05:44:10.081595+0000","last_fullsized":"2024-04-25T05:44:10.081595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":50,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091256+0000","last_change":"2024-04-25T05:43:16.207872+0000","last_active":"2024-04-25T05:44:10.091256+0000","last_peered":"2024-04-25T05:44:10.091256+0000","last_clean":"2024-04-25T05:44:10.091256+0000","last_became_active":"2024-04-25T05:43:16.207529+0000","last_became_peered":"2024-04-25T05:43:16.207529+0000","last_unstale":"2024-04-25T05:44:10.091256+0000","last_undegraded":"2024-04-25T05:44:10.091256+0000","last_fullsized":"2024-04-25T05:44:10.091256+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":305,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.084533+0000","last_change":"2024-04-25T05:42:19.252275+0000","last_active":"2024-04-25T05:44:10.084533+0000","last_peered":"2024-04-25T05:44:10.084533+0000","last_clean":"2024-04-25T05:44:10.084533+0000","last_became_active":"2024-04-25T05:40:43.433465+0000","last_became_peered":"2024-04-25T05:40:43.433465+0000","last_unstale":"2024-04-25T05:44:10.084533+0000","last_undegraded":"2024-04-25T05:44:10.084533+0000","last_fullsized":"2024-04-25T05:44:10.084533+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":364,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092020+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:44:10.092020+0000","last_peered":"2024-04-25T05:44:10.092020+0000","last_clean":"2024-04-25T05:44:10.092020+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:44:10.092020+0000","last_undegraded":"2024-04-25T05:44:10.092020+0000","last_fullsized":"2024-04-25T05:44:10.092020+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.453574+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":406,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.085975+0000","last_change":"2024-04-25T05:43:16.204706+0000","last_active":"2024-04-25T05:44:10.085975+0000","last_peered":"2024-04-25T05:44:10.085975+0000","last_clean":"2024-04-25T05:44:10.085975+0000","last_became_active":"2024-04-25T05:43:16.204375+0000","last_became_peered":"2024-04-25T05:43:16.204375+0000","last_unstale":"2024-04-25T05:44:10.085975+0000","last_undegraded":"2024-04-25T05:44:10.085975+0000","last_fullsized":"2024-04-25T05:44:10.085975+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":196,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.090709+0000","last_change":"2024-04-25T05:42:49.433877+0000","last_active":"2024-04-25T05:44:10.090709+0000","last_peered":"2024-04-25T05:44:10.090709+0000","last_clean":"2024-04-25T05:44:10.090709+0000","last_became_active":"2024-04-25T05:40:43.441308+0000","last_became_peered":"2024-04-25T05:40:43.441308+0000","last_unstale":"2024-04-25T05:44:10.090709+0000","last_undegraded":"2024-04-25T05:44:10.090709+0000","last_fullsized":"2024-04-25T05:44:10.090709+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:14.115420+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":745,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.085041+0000","last_change":"2024-04-25T05:43:16.273904+0000","last_active":"2024-04-25T05:44:10.085041+0000","last_peered":"2024-04-25T05:44:10.085041+0000","last_clean":"2024-04-25T05:44:10.085041+0000","last_became_active":"2024-04-25T05:43:16.195968+0000","last_became_peered":"2024-04-25T05:43:16.195968+0000","last_unstale":"2024-04-25T05:44:10.085041+0000","last_undegraded":"2024-04-25T05:44:10.085041+0000","last_fullsized":"2024-04-25T05:44:10.085041+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:18.293905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":557,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.081962+0000","last_change":"2024-04-25T05:40:55.982555+0000","last_active":"2024-04-25T05:44:10.081962+0000","last_peered":"2024-04-25T05:44:10.081962+0000","last_clean":"2024-04-25T05:44:10.081962+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:44:10.081962+0000","last_undegraded":"2024-04-25T05:44:10.081962+0000","last_fullsized":"2024-04-25T05:44:10.081962+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.485053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":513,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.092289+0000","last_change":"2024-04-25T05:43:18.685398+0000","last_active":"2024-04-25T05:44:10.092289+0000","last_peered":"2024-04-25T05:44:10.092289+0000","last_clean":"2024-04-25T05:44:10.092289+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:44:10.092289+0000","last_undegraded":"2024-04-25T05:44:10.092289+0000","last_fullsized":"2024-04-25T05:44:10.092289+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.685299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.685299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:31.947123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":538,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.082204+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:44:10.082204+0000","last_peered":"2024-04-25T05:44:10.082204+0000","last_clean":"2024-04-25T05:44:10.082204+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:44:10.082204+0000","last_undegraded":"2024-04-25T05:44:10.082204+0000","last_fullsized":"2024-04-25T05:44:10.082204+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":628,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.075621+0000","last_change":"2024-04-25T05:44:02.120412+0000","last_active":"2024-04-25T05:44:10.075621+0000","last_peered":"2024-04-25T05:44:10.075621+0000","last_clean":"2024-04-25T05:44:10.075621+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:44:10.075621+0000","last_undegraded":"2024-04-25T05:44:10.075621+0000","last_fullsized":"2024-04-25T05:44:10.075621+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:44:02.120258+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.120258+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.452057+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":208,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.086577+0000","last_change":"2024-04-25T05:43:16.208668+0000","last_active":"2024-04-25T05:44:10.086577+0000","last_peered":"2024-04-25T05:44:10.086577+0000","last_clean":"2024-04-25T05:44:10.086577+0000","last_became_active":"2024-04-25T05:43:16.207717+0000","last_became_peered":"2024-04-25T05:43:16.207717+0000","last_unstale":"2024-04-25T05:44:10.086577+0000","last_undegraded":"2024-04-25T05:44:10.086577+0000","last_fullsized":"2024-04-25T05:44:10.086577+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:57.544063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.544063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:26.369960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":552,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.091086+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:44:10.091086+0000","last_peered":"2024-04-25T05:44:10.091086+0000","last_clean":"2024-04-25T05:44:10.091086+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:44:10.091086+0000","last_undegraded":"2024-04-25T05:44:10.091086+0000","last_fullsized":"2024-04-25T05:44:10.091086+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.182548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-04-25T05:44:10.093146+0000","last_change":"2024-04-25T05:43:16.198408+0000","last_active":"2024-04-25T05:44:10.093146+0000","last_peered":"2024-04-25T05:44:10.093146+0000","last_clean":"2024-04-25T05:44:10.093146+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:44:10.093146+0000","last_undegraded":"2024-04-25T05:44:10.093146+0000","last_fullsized":"2024-04-25T05:44:10.093146+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5745803264,"num_objects":87674,"num_object_clones":0,"num_object_copies":263022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87674,"num_write_kb":5611136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17212833792,"data_stored":17212833792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59674,"ondisk_log_size":59674,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":601,"num_read_kb":596,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":797,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1402,"num_read_kb":1127,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":6716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":516096,"data_stored":13664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070592,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4223704,"kb_used_data":3787152,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":89521448,"statfs":{"total":95995035648,"available":91669962752,"internally_reserved":0,"allocated":3878043648,"data_stored":3877618583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":447019466},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364932,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1783108,"kb_used_data":1386628,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91962044,"statfs":{"total":95995035648,"available":94169133056,"internally_reserved":0,"allocated":1419907072,"data_stored":1419493879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405797318},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4745524,"kb_used_data":4200684,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":88999628,"statfs":{"total":95995035648,"available":91135619072,"internally_reserved":0,"allocated":4301500416,"data_stored":4301045827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738531,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2532168,"kb_used_data":2067656,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91212984,"statfs":{"total":95995035648,"available":93402095616,"internally_reserved":0,"allocated":2117279744,"data_stored":2116861047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4189084,"kb_used_data":3572052,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":89556068,"statfs":{"total":95995035648,"available":91705413632,"internally_reserved":0,"allocated":3657781248,"data_stored":3657351407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986317,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109528,"kb_used_data":1727120,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91635624,"statfs":{"total":95995035648,"available":93834878976,"internally_reserved":0,"allocated":1768570880,"data_stored":1768145516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2146304000,"data_stored":2146304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4314628096,"data_stored":4314628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3609264128,"data_stored":3609264128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1788346368,"data_stored":1788346368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1422524416,"data_stored":1422524416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3931766784,"data_stored":3931766784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:44:17.820 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-04-25T05:44:17.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:17.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:17.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:18.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:18.149 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:18.167 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:18.167 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:18.195 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:18.196 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:18.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 93508 93492 92.7326 83.5625 0.0150473 0.010778 2024-04-25T05:44:18.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:18.281 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:18.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[27010]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:18.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[27010]: 3.1e scrub starts 2024-04-25T05:44:18.319 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[27010]: 3.1e scrub ok 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[37492]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[37492]: 3.1e scrub starts 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[37492]: 3.1e scrub ok 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:18.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:18.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:18 smithi060 ceph-mon[30898]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:18 smithi060 ceph-mon[30898]: 3.1e scrub starts 2024-04-25T05:44:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:18 smithi060 ceph-mon[30898]: 3.1e scrub ok 2024-04-25T05:44:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:18.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.541 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:18.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:18.670 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:18.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:18.808 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:18.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:18.938 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:18.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:19.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:19.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:19.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:44:19.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:19.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 94876 94860 92.6193 85.5 0.0196609 0.0107922 2024-04-25T05:44:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:19.328 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:19.354 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[27010]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-04-25T05:44:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[27010]: 5.17 scrub starts 2024-04-25T05:44:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[27010]: 5.17 scrub ok 2024-04-25T05:44:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[37492]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-04-25T05:44:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[37492]: 5.17 scrub starts 2024-04-25T05:44:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-mon[37492]: 5.17 scrub ok 2024-04-25T05:44:19.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:19.484 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:19.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:19 smithi060 ceph-mon[30898]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-04-25T05:44:19.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:19 smithi060 ceph-mon[30898]: 5.17 scrub starts 2024-04-25T05:44:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:19 smithi060 ceph-mon[30898]: 5.17 scrub ok 2024-04-25T05:44:19.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:19.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:19.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:19.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:19.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:19.869 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:19.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:19.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:19.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:20.045 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:44:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:44:20.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:44:20.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:20.125 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:20.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 15 96039 96024 92.3135 72.75 0.0198352 0.0108274 2024-04-25T05:44:20.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:20.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:20.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:20.382 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:44:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 5.16 scrub starts 2024-04-25T05:44:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[27010]: 5.16 scrub ok 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 5.16 scrub starts 2024-04-25T05:44:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:20 smithi043 ceph-mon[37492]: 5.16 scrub ok 2024-04-25T05:44:20.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:20.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:44:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:44:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:44:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:44:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 5.16 scrub starts 2024-04-25T05:44:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:20 smithi060 ceph-mon[30898]: 5.16 scrub ok 2024-04-25T05:44:20.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:20.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:20.648 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:20.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:20.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:20.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:20.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:20.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:21.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:21.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:21.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:21.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:21.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 97380 97364 92.1836 83.75 0.0123758 0.0108433 2024-04-25T05:44:21.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:21.323 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[27010]: pgmap v541: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:44:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[37492]: pgmap v541: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:44:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:21 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:44:21.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:21.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:21.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:21 smithi060 ceph-mon[30898]: pgmap v541: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:44:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:21 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:44:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:21 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:44:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:21 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:44:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:21 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:44:21.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:21.588 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:21.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:21.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:21.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:21.946 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:21.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:21.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:22.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:22.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:22.130 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:22.172 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:22.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 98603 98587 91.9485 76.4375 0.0113186 0.0108706 2024-04-25T05:44:22.217 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:22.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:22.263 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:22.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:22 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:44:22.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:22 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:44:22.265 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:22.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:22 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:44:22.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:22 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:44:22.306 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:22.350 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:22.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:22.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:22.393 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:22.435 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:22.449 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:22.450 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:22.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:22.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:22 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:44:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:22 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:44:22.528 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:22.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:22.670 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:22.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:22.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:22.822 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:44:22.822 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:44:22.822 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 3, 4, 7, 6, 0] 2024-04-25T05:44:22.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd out 0 2024-04-25T05:44:22.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:22.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:22.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:23.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:23.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:23.077 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:44:23.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:23.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 99692 99676 91.5972 68.0625 0.01699 0.0109126 2024-04-25T05:44:23.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:23.318 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:23.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: pgmap v542: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1345123319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/581181767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: pgmap v542: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1345123319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/581181767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:23 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.491 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:23.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:23.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: pgmap v542: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1345123319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/581181767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:23 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:44:23.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:23.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:23.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:23.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:23.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:23.909 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:23.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:24.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:24.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:24.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:24.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 15 101062 101047 91.5114 85.6875 0.00453128 0.0109222 2024-04-25T05:44:24.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: 3.18 scrub starts 2024-04-25T05:44:24.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: 3.18 scrub ok 2024-04-25T05:44:24.281 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: osdmap e323: 8 total, 8 up, 6 in 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: 3.18 scrub starts 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: 3.18 scrub ok 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: osdmap e323: 8 total, 8 up, 6 in 2024-04-25T05:44:24.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:24.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:24.342 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:24.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: 3.18 scrub starts 2024-04-25T05:44:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: 3.18 scrub ok 2024-04-25T05:44:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:44:24.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: osdmap e323: 8 total, 8 up, 6 in 2024-04-25T05:44:24.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:24.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:24.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:24.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:24.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:24.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:24.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:24.989 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:24.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:25.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:44:25.071 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:44:25.071 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:44:25.071 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:44:25.071 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:44:25.149 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:25.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 102254 102238 91.2674 74.4375 0.00667001 0.0109519 2024-04-25T05:44:25.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:25 smithi043 ceph-mon[37492]: pgmap v544: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:44:25.218 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:25 smithi043 ceph-mon[27010]: pgmap v544: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:44:25.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:25.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:25.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:25.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:25 smithi060 ceph-mon[30898]: pgmap v544: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:44:25.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:44:25.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.649 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:25.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.790 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:25.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:25.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:25.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:26.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:26.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:26.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 103460 103444 91.0434 75.375 0.00676712 0.0109792 2024-04-25T05:44:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:26.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:26.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:44:26.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:26.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:26.490 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:26.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:26.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:26.672 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:26.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:44:26.738 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:26.764 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:26.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:26 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:44:26.537+0000 7f173ec68700 -1 mon.a@0(leader).osd e323 definitely_dead 0 2024-04-25T05:44:26.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:26.890 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:26.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:26.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:27.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:27.120 INFO:teuthology.orchestra.run.smithi043.stderr:marked out osd.0. 2024-04-25T05:44:27.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:27.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:27.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 104470 104454 90.6556 63.125 0.00266365 0.0110152 2024-04-25T05:44:27.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:27.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:27.314 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:27.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:27.354 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:27.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:27.374 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[27010]: pgmap v545: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[27010]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[37492]: pgmap v545: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:44:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:27 smithi043 ceph-mon[37492]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:44:27.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:27.397 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:27.417 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:27.438 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:27.458 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:27.459 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:27.459 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:27.460 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:27.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:27.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:27 smithi060 ceph-mon[30898]: pgmap v545: 158 pgs: 158 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-25T05:44:27.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-25T05:44:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:27 smithi060 ceph-mon[30898]: Client client.admin marked osd.0 out, while it was still marked up 2024-04-25T05:44:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:27.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:27.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:27.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:27.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:27.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:27.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:27.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:27.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:27.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:28.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:28.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:28.134 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:28.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:44:28.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:44:28.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 104470 104454 89.4138 0 - 0.0110152 2024-04-25T05:44:28.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:28.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:28.382 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:28.484 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:28.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3169027922' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-25T05:44:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:28 smithi060 ceph-mon[30898]: osdmap e324: 8 total, 8 up, 5 in 2024-04-25T05:44:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3958466479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:28 smithi060 ceph-mon[30898]: osdmap e325: 8 total, 8 up, 5 in 2024-04-25T05:44:28.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:28.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:28.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[27010]: osdmap e324: 8 total, 8 up, 5 in 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3958466479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[27010]: osdmap e325: 8 total, 8 up, 5 in 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[37492]: osdmap e324: 8 total, 8 up, 5 in 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3958466479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:28.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:28 smithi043 ceph-mon[37492]: osdmap e325: 8 total, 8 up, 5 in 2024-04-25T05:44:28.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:28.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:28.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:28.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:28.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.003 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:29.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:29.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:29.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 104479 104463 88.2133 0.28125 2.06024 0.0111725 2024-04-25T05:44:29.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:29.280 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:29.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[37492]: pgmap v547: 158 pgs: 5 peering, 1 unknown, 152 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 776 op/s 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[37492]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[37492]: osdmap e326: 8 total, 8 up, 5 in 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[27010]: pgmap v547: 158 pgs: 5 peering, 1 unknown, 152 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 776 op/s 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[27010]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T05:44:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-mon[27010]: osdmap e326: 8 total, 8 up, 5 in 2024-04-25T05:44:29.427 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:29 smithi060 ceph-mon[30898]: pgmap v547: 158 pgs: 5 peering, 1 unknown, 152 active+clean; 6.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 639 B/s rd, 48 MiB/s wr, 776 op/s 2024-04-25T05:44:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:29 smithi060 ceph-mon[30898]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T05:44:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:29 smithi060 ceph-mon[30898]: osdmap e326: 8 total, 8 up, 5 in 2024-04-25T05:44:29.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:29.557 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:29.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:29.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:29.728 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:29.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:29.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:29.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:29.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:29.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:30.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:30.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:30.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 105319 105303 87.7371 52.5 0.0295257 0.0113927 2024-04-25T05:44:30.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:30.257 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:30.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.0 starting backfill to osd.4 from (0'0,0'0] MAX to 323'6471 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 323'6534 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.9 starting backfill to osd.6 from (0'0,0'0] MAX to 323'6533 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 323'6499 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 323'6518 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 323'6475 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[37492]: osdmap e327: 8 total, 8 up, 5 in 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.0 starting backfill to osd.4 from (0'0,0'0] MAX to 323'6471 2024-04-25T05:44:30.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 323'6534 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.9 starting backfill to osd.6 from (0'0,0'0] MAX to 323'6533 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 323'6499 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 323'6518 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 323'6475 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:30.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:30 smithi043 ceph-mon[27010]: osdmap e327: 8 total, 8 up, 5 in 2024-04-25T05:44:30.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.0 starting backfill to osd.4 from (0'0,0'0] MAX to 323'6471 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 323'6534 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.9 starting backfill to osd.6 from (0'0,0'0] MAX to 323'6533 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.b starting backfill to osd.3 from (0'0,0'0] MAX to 323'6499 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.5 starting backfill to osd.3 from (0'0,0'0] MAX to 323'6518 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: 9.6 starting backfill to osd.7 from (0'0,0'0] MAX to 323'6475 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:30 smithi060 ceph-mon[30898]: osdmap e327: 8 total, 8 up, 5 in 2024-04-25T05:44:30.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:30.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:30.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:30.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:30.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:30.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:30.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:30.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:31.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:31.116 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:31.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:31.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 106258 106242 87.3521 58.6875 0.0144376 0.0114236 2024-04-25T05:44:31.239 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:31.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:31.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:31.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:31.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:31 smithi043 ceph-mon[27010]: pgmap v550: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 47 MiB/s wr, 750 op/s; 204 B/s, 11 objects/s recovering 2024-04-25T05:44:31.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:31 smithi043 ceph-mon[27010]: osdmap e328: 8 total, 8 up, 5 in 2024-04-25T05:44:31.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:31 smithi043 ceph-mon[37492]: pgmap v550: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 47 MiB/s wr, 750 op/s; 204 B/s, 11 objects/s recovering 2024-04-25T05:44:31.381 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:31 smithi043 ceph-mon[37492]: osdmap e328: 8 total, 8 up, 5 in 2024-04-25T05:44:31.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:31.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:31 smithi060 ceph-mon[30898]: pgmap v550: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 47 MiB/s wr, 750 op/s; 204 B/s, 11 objects/s recovering 2024-04-25T05:44:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:31 smithi060 ceph-mon[30898]: osdmap e328: 8 total, 8 up, 5 in 2024-04-25T05:44:31.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:31.542 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:31.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:31.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:31.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:31.807 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:31.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:31.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:31.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:32.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:32.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:32.162 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:44:32.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:32.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 107145 107129 86.9376 55.4375 0.0145073 0.0114972 2024-04-25T05:44:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:32.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:32.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:32.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:44:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2047647971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2047647971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:32 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:32.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:32.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2047647971' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:32 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:32.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:32.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:32.672 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:32.672 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:32.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:32.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:32.887 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:32.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:32.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:33.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:33.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:33.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.190 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:33.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:33.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 108175 108159 86.6483 64.375 0.0181061 0.0115362 2024-04-25T05:44:33.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:33.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:33.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:33.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:33.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: pgmap v553: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 307 B/s, 17 objects/s recovering 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: osdmap e329: 8 total, 8 up, 5 in 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:33.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:33.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: pgmap v553: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 307 B/s, 17 objects/s recovering 2024-04-25T05:44:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: osdmap e329: 8 total, 8 up, 5 in 2024-04-25T05:44:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: pgmap v553: 158 pgs: 3 remapped+peering, 31 peering, 4 unknown, 120 active+clean; 6.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 70 MiB/s wr, 1.13k op/s; 307 B/s, 17 objects/s recovering 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: osdmap e329: 8 total, 8 up, 5 in 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:33.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:33.725 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:33.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:33.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:33.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:33.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:34.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:34.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.169 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:34.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:34.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 15 109210 109195 86.371 64.75 0.00976197 0.0115728 2024-04-25T05:44:34.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:34.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:34.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:34.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:34.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:34.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:44:34.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:34.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:34.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:34.769 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:34.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:34.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:35.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:35.056 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:35.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:35.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:35.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:44:35.216045+0000 min lat: 0.00146388 max lat: 2.14494 avg lat: 0.0115911 2024-04-25T05:44:35.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:44:35.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 110423 110407 86.2376 75.75 0.0101996 0.0115911 2024-04-25T05:44:35.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:35.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:35.474 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:35 smithi060 ceph-mon[30898]: pgmap v555: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 798 op/s; 36024/325470 objects misplaced (11.068%); 57 MiB/s, 913 objects/s recovering 2024-04-25T05:44:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:35 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:44:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:35 smithi060 ceph-mon[30898]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 11 pgs peering) 2024-04-25T05:44:35.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[27010]: pgmap v555: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 798 op/s; 36024/325470 objects misplaced (11.068%); 57 MiB/s, 913 objects/s recovering 2024-04-25T05:44:35.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:44:35.535 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[27010]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 11 pgs peering) 2024-04-25T05:44:35.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[37492]: pgmap v555: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 798 op/s; 36024/325470 objects misplaced (11.068%); 57 MiB/s, 913 objects/s recovering 2024-04-25T05:44:35.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:44:35.536 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:35 smithi043 ceph-mon[37492]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 11 pgs peering) 2024-04-25T05:44:35.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:35.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:35.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:35.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:35.941 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:44:35.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:44:35.941 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-25T05:44:35.941 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-25T05:44:35.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:44:36.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:36.034 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:36.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:36.190 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:36.191 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:36.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:36.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 111398 111382 85.925 60.9375 0.0316186 0.0116329 2024-04-25T05:44:36.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:36.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:36.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:36.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:36.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:36.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:36.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:36.895 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:36.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:36.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:37.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:37.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:37.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:37.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 112564 112548 85.7658 72.875 0.00987682 0.0116547 2024-04-25T05:44:37.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:37 smithi043 ceph-mon[27010]: pgmap v556: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 40 MiB/s wr, 634 op/s; 36024/325470 objects misplaced (11.068%); 45 MiB/s, 725 objects/s recovering 2024-04-25T05:44:37.237 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/556417415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:37.238 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:37 smithi043 ceph-mon[37492]: pgmap v556: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 40 MiB/s wr, 634 op/s; 36024/325470 objects misplaced (11.068%); 45 MiB/s, 725 objects/s recovering 2024-04-25T05:44:37.238 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/556417415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:37.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:37.290 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:37.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:44:37.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:37.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:37 smithi060 ceph-mon[30898]: pgmap v556: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 40 MiB/s wr, 634 op/s; 36024/325470 objects misplaced (11.068%); 45 MiB/s, 725 objects/s recovering 2024-04-25T05:44:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/556417415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:37.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:37.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:37.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:37.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:37.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:37.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:37.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:37.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:37.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:38.083 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:38.120 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:38.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 113650 113634 85.5502 67.875 0.0096149 0.011684 2024-04-25T05:44:38.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:38.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:38.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:38.415 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:38.416 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:38.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:38.533 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:38.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:38.666 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:38.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:38.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:38.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:38.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:38.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:39.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:39.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:39.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:39.207 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:39.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 114658 114642 85.2818 63 0.0105879 0.0117205 2024-04-25T05:44:39.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:39.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:39.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:39 smithi060 ceph-mon[30898]: pgmap v557: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 30 MiB/s wr, 477 op/s; 36024/325470 objects misplaced (11.068%); 34 MiB/s, 546 objects/s recovering 2024-04-25T05:44:39.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:39.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:39 smithi043 ceph-mon[27010]: pgmap v557: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 30 MiB/s wr, 477 op/s; 36024/325470 objects misplaced (11.068%); 34 MiB/s, 546 objects/s recovering 2024-04-25T05:44:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:39 smithi043 ceph-mon[37492]: pgmap v557: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 30 MiB/s wr, 477 op/s; 36024/325470 objects misplaced (11.068%); 34 MiB/s, 546 objects/s recovering 2024-04-25T05:44:39.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:39.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:39.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:39.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:39.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:39.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:40.035 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:40.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:40.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:40.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:40.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 115720 115704 85.0593 66.375 0.0126723 0.0117517 2024-04-25T05:44:40.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:40.308 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:40.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:40.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:40.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:40.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:40.836 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:40.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:40.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:40.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:41.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:41.107 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:41.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:41.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 116879 116863 84.9125 72.4375 0.0520845 0.011771 2024-04-25T05:44:41.246 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:41.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:41.378 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:41.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:41.435 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:41 smithi043 ceph-mon[27010]: pgmap v558: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 33040/341814 objects misplaced (9.666%); 63 MiB/s, 1.01k objects/s recovering 2024-04-25T05:44:41.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:41 smithi043 ceph-mon[37492]: pgmap v558: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 33040/341814 objects misplaced (9.666%); 63 MiB/s, 1.01k objects/s recovering 2024-04-25T05:44:41.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:41.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:41 smithi060 ceph-mon[30898]: pgmap v558: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 33040/341814 objects misplaced (9.666%); 63 MiB/s, 1.01k objects/s recovering 2024-04-25T05:44:41.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:41.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:41.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:41.758 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:41.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:41.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:41.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:41.992 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:41.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:42.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:42.147 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:42.217 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 117892 117876 84.6629 63.3125 0.0135915 0.0118065 2024-04-25T05:44:42.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:42.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:42.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.412 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:42.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:42.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:42.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:42.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:44:42.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:42.670 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:44:42.703 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:42.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:42.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:42.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:42.968 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:42.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:43.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.102 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:43.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:43.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:43.217 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 118975 118959 84.4699 67.6875 0.011066 0.0118331 2024-04-25T05:44:43.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:44:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:43.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:43.365 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:43.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:43 smithi060 ceph-mon[30898]: pgmap v559: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 58 MiB/s wr, 925 op/s; 33040/341814 objects misplaced (9.666%); 56 MiB/s, 906 objects/s recovering 2024-04-25T05:44:43.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.501 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:43.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:43.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:43.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:43 smithi043 ceph-mon[27010]: pgmap v559: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 58 MiB/s wr, 925 op/s; 33040/341814 objects misplaced (9.666%); 56 MiB/s, 906 objects/s recovering 2024-04-25T05:44:43.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:43 smithi043 ceph-mon[37492]: pgmap v559: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 6.9 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 58 MiB/s wr, 925 op/s; 33040/341814 objects misplaced (9.666%); 56 MiB/s, 906 objects/s recovering 2024-04-25T05:44:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:43.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:43.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:43.780 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:43.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:43.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:43.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:44.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:44.098 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2024-04-25T05:44:41.878851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7452563995,"num_objects":113938,"num_object_clones":0,"num_object_copies":341814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33040,"num_objects_unfound":0,"num_objects_dirty":113938,"num_whiteouts":0,"num_read":2035,"num_read_kb":1755,"num_write":114622,"num_write_kb":7277357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9528,"num_bytes_recovered":592391970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52723,"ondisk_log_size":52723,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":22161628,"kb_used_data":19633140,"kb_used_omap":7,"kb_used_meta":2528440,"kb_avail":446564132,"statfs":{"total":479975178240,"available":457281671168,"internally_reserved":0,"allocated":20104335360,"data_stored":20102121439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2589122785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":606208000,"num_objects":9250,"num_object_clones":0,"num_object_copies":27750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33040,"num_objects_unfound":0,"num_objects_dirty":9250,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9250,"num_write_kb":592000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9068,"num_bytes_recovered":592053145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005523"},"pg_stats":[{"pgid":"9.1","version":"329'7137","reported_seq":7212,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.605819+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:44:38.605819+0000","last_peered":"2024-04-25T05:44:38.605819+0000","last_clean":"2024-04-25T05:44:38.605819+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:44:38.605819+0000","last_undegraded":"2024-04-25T05:44:38.605819+0000","last_fullsized":"2024-04-25T05:44:38.605819+0000","mapping_epoch":296,"log_start":"317'3987","ondisk_log_start":"317'3987","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3150,"log_dups_size":0,"ondisk_log_size":3150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467730432,"num_objects":7137,"num_object_clones":0,"num_object_copies":21411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7137,"num_write_kb":456768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":281,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.175526+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:44:32.175526+0000","last_peered":"2024-04-25T05:44:32.175526+0000","last_clean":"2024-04-25T05:44:32.175526+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:44:32.175526+0000","last_undegraded":"2024-04-25T05:44:32.175526+0000","last_fullsized":"2024-04-25T05:44:32.175526+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":572,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168630+0000","last_change":"2024-04-25T05:44:28.173941+0000","last_active":"2024-04-25T05:44:32.168630+0000","last_peered":"2024-04-25T05:44:32.168630+0000","last_clean":"2024-04-25T05:44:32.168630+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:44:32.168630+0000","last_undegraded":"2024-04-25T05:44:32.168630+0000","last_fullsized":"2024-04-25T05:44:32.168630+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"325'12","reported_seq":363,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171370+0000","last_change":"2024-04-25T05:44:28.198744+0000","last_active":"2024-04-25T05:44:32.171370+0000","last_peered":"2024-04-25T05:44:32.171370+0000","last_clean":"2024-04-25T05:44:32.171370+0000","last_became_active":"2024-04-25T05:44:28.171810+0000","last_became_peered":"2024-04-25T05:44:28.171810+0000","last_unstale":"2024-04-25T05:44:32.171370+0000","last_undegraded":"2024-04-25T05:44:32.171370+0000","last_fullsized":"2024-04-25T05:44:32.171370+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":244,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166717+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:44:32.166717+0000","last_peered":"2024-04-25T05:44:32.166717+0000","last_clean":"2024-04-25T05:44:32.166717+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:44:32.166717+0000","last_undegraded":"2024-04-25T05:44:32.166717+0000","last_fullsized":"2024-04-25T05:44:32.166717+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":878,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171637+0000","last_change":"2024-04-25T05:43:19.210320+0000","last_active":"2024-04-25T05:44:32.171637+0000","last_peered":"2024-04-25T05:44:32.171637+0000","last_clean":"2024-04-25T05:44:32.171637+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:44:32.171637+0000","last_undegraded":"2024-04-25T05:44:32.171637+0000","last_fullsized":"2024-04-25T05:44:32.171637+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"329'7062","reported_seq":7143,"reported_epoch":329,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:44:38.588950+0000","last_change":"2024-04-25T05:44:29.211276+0000","last_active":"2024-04-25T05:44:38.588950+0000","last_peered":"2024-04-25T05:44:38.588950+0000","last_clean":"2024-04-25T05:44:27.095992+0000","last_became_active":"2024-04-25T05:44:29.189111+0000","last_became_peered":"2024-04-25T05:44:29.189111+0000","last_unstale":"2024-04-25T05:44:38.588950+0000","last_undegraded":"2024-04-25T05:44:38.588950+0000","last_fullsized":"2024-04-25T05:44:38.588950+0000","mapping_epoch":325,"log_start":"318'4087","ondisk_log_start":"318'4087","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":615,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462815232,"num_objects":7062,"num_object_clones":0,"num_object_copies":21186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7062,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":451968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":7062}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":290,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.164766+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:44:32.164766+0000","last_peered":"2024-04-25T05:44:32.164766+0000","last_clean":"2024-04-25T05:44:32.164766+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:44:32.164766+0000","last_undegraded":"2024-04-25T05:44:32.164766+0000","last_fullsized":"2024-04-25T05:44:32.164766+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":286,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172206+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:44:32.172206+0000","last_peered":"2024-04-25T05:44:32.172206+0000","last_clean":"2024-04-25T05:44:32.172206+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:44:32.172206+0000","last_undegraded":"2024-04-25T05:44:32.172206+0000","last_fullsized":"2024-04-25T05:44:32.172206+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.541034+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":578,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170262+0000","last_change":"2024-04-25T05:44:28.168158+0000","last_active":"2024-04-25T05:44:32.170262+0000","last_peered":"2024-04-25T05:44:32.170262+0000","last_clean":"2024-04-25T05:44:32.170262+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:44:32.170262+0000","last_undegraded":"2024-04-25T05:44:32.170262+0000","last_fullsized":"2024-04-25T05:44:32.170262+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":197,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168768+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:44:32.168768+0000","last_peered":"2024-04-25T05:44:32.168768+0000","last_clean":"2024-04-25T05:44:32.168768+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:44:32.168768+0000","last_undegraded":"2024-04-25T05:44:32.168768+0000","last_fullsized":"2024-04-25T05:44:32.168768+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.600058+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":478,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166796+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:44:32.166796+0000","last_peered":"2024-04-25T05:44:32.166796+0000","last_clean":"2024-04-25T05:44:32.166796+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:44:32.166796+0000","last_undegraded":"2024-04-25T05:44:32.166796+0000","last_fullsized":"2024-04-25T05:44:32.166796+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.377967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"329'7050","reported_seq":7125,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:37.924246+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:44:37.924246+0000","last_peered":"2024-04-25T05:44:37.924246+0000","last_clean":"2024-04-25T05:44:37.924246+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:44:37.924246+0000","last_undegraded":"2024-04-25T05:44:37.924246+0000","last_fullsized":"2024-04-25T05:44:37.924246+0000","mapping_epoch":296,"log_start":"316'3777","ondisk_log_start":"316'3777","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462028800,"num_objects":7050,"num_object_clones":0,"num_object_copies":21150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7050,"num_write_kb":451200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":590,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172627+0000","last_change":"2024-04-25T05:42:53.501053+0000","last_active":"2024-04-25T05:44:32.172627+0000","last_peered":"2024-04-25T05:44:32.172627+0000","last_clean":"2024-04-25T05:44:32.172627+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:44:32.172627+0000","last_undegraded":"2024-04-25T05:44:32.172627+0000","last_fullsized":"2024-04-25T05:44:32.172627+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:42:53.500776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.500776+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":576,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.151361+0000","last_change":"2024-04-25T05:42:51.237692+0000","last_active":"2024-04-25T05:44:32.151361+0000","last_peered":"2024-04-25T05:44:32.151361+0000","last_clean":"2024-04-25T05:44:32.151361+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:44:32.151361+0000","last_undegraded":"2024-04-25T05:44:32.151361+0000","last_fullsized":"2024-04-25T05:44:32.151361+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.237570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.237570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:32.333607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":372,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.154045+0000","last_change":"2024-04-25T05:43:24.227422+0000","last_active":"2024-04-25T05:44:32.154045+0000","last_peered":"2024-04-25T05:44:32.154045+0000","last_clean":"2024-04-25T05:44:32.154045+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:44:32.154045+0000","last_undegraded":"2024-04-25T05:44:32.154045+0000","last_fullsized":"2024-04-25T05:44:32.154045+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.227343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.227343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:33.320292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":560,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169123+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:44:32.169123+0000","last_peered":"2024-04-25T05:44:32.169123+0000","last_clean":"2024-04-25T05:44:32.169123+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:44:32.169123+0000","last_undegraded":"2024-04-25T05:44:32.169123+0000","last_fullsized":"2024-04-25T05:44:32.169123+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"329'7240","reported_seq":7314,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.587693+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:44:38.587693+0000","last_peered":"2024-04-25T05:44:38.587693+0000","last_clean":"2024-04-25T05:44:38.587693+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:44:38.587693+0000","last_undegraded":"2024-04-25T05:44:38.587693+0000","last_fullsized":"2024-04-25T05:44:38.587693+0000","mapping_epoch":296,"log_start":"317'4081","ondisk_log_start":"317'4081","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":608,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.173672+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:44:32.173672+0000","last_peered":"2024-04-25T05:44:32.173672+0000","last_clean":"2024-04-25T05:44:32.173672+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:44:32.173672+0000","last_undegraded":"2024-04-25T05:44:32.173672+0000","last_fullsized":"2024-04-25T05:44:32.173672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":562,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171709+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:44:32.171709+0000","last_peered":"2024-04-25T05:44:32.171709+0000","last_clean":"2024-04-25T05:44:32.171709+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:44:32.171709+0000","last_undegraded":"2024-04-25T05:44:32.171709+0000","last_fullsized":"2024-04-25T05:44:32.171709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.125277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"325'13","reported_seq":90,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167063+0000","last_change":"2024-04-25T05:44:28.188020+0000","last_active":"2024-04-25T05:44:32.167063+0000","last_peered":"2024-04-25T05:44:32.167063+0000","last_clean":"2024-04-25T05:44:32.167063+0000","last_became_active":"2024-04-25T05:44:28.178692+0000","last_became_peered":"2024-04-25T05:44:28.178692+0000","last_unstale":"2024-04-25T05:44:32.167063+0000","last_undegraded":"2024-04-25T05:44:32.167063+0000","last_fullsized":"2024-04-25T05:44:32.167063+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":424,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166018+0000","last_change":"2024-04-25T05:44:28.186072+0000","last_active":"2024-04-25T05:44:32.166018+0000","last_peered":"2024-04-25T05:44:32.166018+0000","last_clean":"2024-04-25T05:44:32.166018+0000","last_became_active":"2024-04-25T05:44:28.177729+0000","last_became_peered":"2024-04-25T05:44:28.177729+0000","last_unstale":"2024-04-25T05:44:32.166018+0000","last_undegraded":"2024-04-25T05:44:32.166018+0000","last_fullsized":"2024-04-25T05:44:32.166018+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":554,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172104+0000","last_change":"2024-04-25T05:44:30.932605+0000","last_active":"2024-04-25T05:44:32.172104+0000","last_peered":"2024-04-25T05:44:32.172104+0000","last_clean":"2024-04-25T05:44:32.172104+0000","last_became_active":"2024-04-25T05:44:30.932221+0000","last_became_peered":"2024-04-25T05:44:30.932221+0000","last_unstale":"2024-04-25T05:44:32.172104+0000","last_undegraded":"2024-04-25T05:44:32.172104+0000","last_fullsized":"2024-04-25T05:44:32.172104+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.344182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"329'7112","reported_seq":7193,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:37.921589+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:44:37.921589+0000","last_peered":"2024-04-25T05:44:37.921589+0000","last_clean":"2024-04-25T05:44:37.921589+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:44:37.921589+0000","last_undegraded":"2024-04-25T05:44:37.921589+0000","last_fullsized":"2024-04-25T05:44:37.921589+0000","mapping_epoch":301,"log_start":"316'3853","ondisk_log_start":"316'3853","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466092032,"num_objects":7112,"num_object_clones":0,"num_object_copies":21336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":563,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.151282+0000","last_change":"2024-04-25T05:43:58.091032+0000","last_active":"2024-04-25T05:44:32.151282+0000","last_peered":"2024-04-25T05:44:32.151282+0000","last_clean":"2024-04-25T05:44:32.151282+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:44:32.151282+0000","last_undegraded":"2024-04-25T05:44:32.151282+0000","last_fullsized":"2024-04-25T05:44:32.151282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:58.090855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:58.090855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.355724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":717,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171811+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:44:32.171811+0000","last_peered":"2024-04-25T05:44:32.171811+0000","last_clean":"2024-04-25T05:44:32.171811+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:44:32.171811+0000","last_undegraded":"2024-04-25T05:44:32.171811+0000","last_fullsized":"2024-04-25T05:44:32.171811+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.991166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":580,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169126+0000","last_change":"2024-04-25T05:44:28.180828+0000","last_active":"2024-04-25T05:44:32.169126+0000","last_peered":"2024-04-25T05:44:32.169126+0000","last_clean":"2024-04-25T05:44:32.169126+0000","last_became_active":"2024-04-25T05:44:28.178969+0000","last_became_peered":"2024-04-25T05:44:28.178969+0000","last_unstale":"2024-04-25T05:44:32.169126+0000","last_undegraded":"2024-04-25T05:44:32.169126+0000","last_fullsized":"2024-04-25T05:44:32.169126+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:15.041538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.041538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.837433+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":239,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172534+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:44:32.172534+0000","last_peered":"2024-04-25T05:44:32.172534+0000","last_clean":"2024-04-25T05:44:32.172534+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:44:32.172534+0000","last_undegraded":"2024-04-25T05:44:32.172534+0000","last_fullsized":"2024-04-25T05:44:32.172534+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.232569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":782,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.153329+0000","last_change":"2024-04-25T05:44:30.916141+0000","last_active":"2024-04-25T05:44:32.153329+0000","last_peered":"2024-04-25T05:44:32.153329+0000","last_clean":"2024-04-25T05:44:32.153329+0000","last_became_active":"2024-04-25T05:44:30.915718+0000","last_became_peered":"2024-04-25T05:44:30.915718+0000","last_unstale":"2024-04-25T05:44:32.153329+0000","last_undegraded":"2024-04-25T05:44:32.153329+0000","last_fullsized":"2024-04-25T05:44:32.153329+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.223066+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.958082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"329'7199","reported_seq":7274,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.622232+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:44:38.622232+0000","last_peered":"2024-04-25T05:44:38.622232+0000","last_clean":"2024-04-25T05:44:38.622232+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:44:38.622232+0000","last_undegraded":"2024-04-25T05:44:38.622232+0000","last_fullsized":"2024-04-25T05:44:38.622232+0000","mapping_epoch":296,"log_start":"317'4116","ondisk_log_start":"317'4116","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471793664,"num_objects":7199,"num_object_clones":0,"num_object_copies":21597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":683,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172032+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:44:32.172032+0000","last_peered":"2024-04-25T05:44:32.172032+0000","last_clean":"2024-04-25T05:44:32.172032+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:44:32.172032+0000","last_undegraded":"2024-04-25T05:44:32.172032+0000","last_fullsized":"2024-04-25T05:44:32.172032+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":506,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171249+0000","last_change":"2024-04-25T05:44:28.134148+0000","last_active":"2024-04-25T05:44:32.171249+0000","last_peered":"2024-04-25T05:44:32.171249+0000","last_clean":"2024-04-25T05:44:32.171249+0000","last_became_active":"2024-04-25T05:44:28.124913+0000","last_became_peered":"2024-04-25T05:44:28.124913+0000","last_unstale":"2024-04-25T05:44:32.171249+0000","last_undegraded":"2024-04-25T05:44:32.171249+0000","last_fullsized":"2024-04-25T05:44:32.171249+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:43:24.628430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.628430+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":267,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166792+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:44:32.166792+0000","last_peered":"2024-04-25T05:44:32.166792+0000","last_clean":"2024-04-25T05:44:32.166792+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:44:32.166792+0000","last_undegraded":"2024-04-25T05:44:32.166792+0000","last_fullsized":"2024-04-25T05:44:32.166792+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:50.274438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":195,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169229+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:44:32.169229+0000","last_peered":"2024-04-25T05:44:32.169229+0000","last_clean":"2024-04-25T05:44:32.169229+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:44:32.169229+0000","last_undegraded":"2024-04-25T05:44:32.169229+0000","last_fullsized":"2024-04-25T05:44:32.169229+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":742,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168407+0000","last_change":"2024-04-25T05:43:19.215636+0000","last_active":"2024-04-25T05:44:32.168407+0000","last_peered":"2024-04-25T05:44:32.168407+0000","last_clean":"2024-04-25T05:44:32.168407+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:44:32.168407+0000","last_undegraded":"2024-04-25T05:44:32.168407+0000","last_fullsized":"2024-04-25T05:44:32.168407+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"329'7129","reported_seq":7195,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.597269+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:44:38.597269+0000","last_peered":"2024-04-25T05:44:38.597269+0000","last_clean":"2024-04-25T05:44:38.597269+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:44:38.597269+0000","last_undegraded":"2024-04-25T05:44:38.597269+0000","last_fullsized":"2024-04-25T05:44:38.597269+0000","mapping_epoch":301,"log_start":"318'4238","ondisk_log_start":"318'4238","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2891,"log_dups_size":715,"ondisk_log_size":2891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467206144,"num_objects":7129,"num_object_clones":0,"num_object_copies":21387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":456256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":274,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171948+0000","last_change":"2024-04-25T05:44:28.202828+0000","last_active":"2024-04-25T05:44:32.171948+0000","last_peered":"2024-04-25T05:44:32.171948+0000","last_clean":"2024-04-25T05:44:32.171948+0000","last_became_active":"2024-04-25T05:44:28.182043+0000","last_became_peered":"2024-04-25T05:44:28.182043+0000","last_unstale":"2024-04-25T05:44:32.171948+0000","last_undegraded":"2024-04-25T05:44:32.171948+0000","last_fullsized":"2024-04-25T05:44:32.171948+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.008065+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":264,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171447+0000","last_change":"2024-04-25T05:44:28.217831+0000","last_active":"2024-04-25T05:44:32.171447+0000","last_peered":"2024-04-25T05:44:32.171447+0000","last_clean":"2024-04-25T05:44:32.171447+0000","last_became_active":"2024-04-25T05:44:28.170458+0000","last_became_peered":"2024-04-25T05:44:28.170458+0000","last_unstale":"2024-04-25T05:44:32.171447+0000","last_undegraded":"2024-04-25T05:44:32.171447+0000","last_fullsized":"2024-04-25T05:44:32.171447+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":449,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168571+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:44:32.168571+0000","last_peered":"2024-04-25T05:44:32.168571+0000","last_clean":"2024-04-25T05:44:32.168571+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:44:32.168571+0000","last_undegraded":"2024-04-25T05:44:32.168571+0000","last_fullsized":"2024-04-25T05:44:32.168571+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":181,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167000+0000","last_change":"2024-04-25T05:44:28.168971+0000","last_active":"2024-04-25T05:44:32.167000+0000","last_peered":"2024-04-25T05:44:32.167000+0000","last_clean":"2024-04-25T05:44:32.167000+0000","last_became_active":"2024-04-25T05:44:28.168333+0000","last_became_peered":"2024-04-25T05:44:28.168333+0000","last_unstale":"2024-04-25T05:44:32.167000+0000","last_undegraded":"2024-04-25T05:44:32.167000+0000","last_fullsized":"2024-04-25T05:44:32.167000+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":741,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172830+0000","last_change":"2024-04-25T05:43:28.638040+0000","last_active":"2024-04-25T05:44:32.172830+0000","last_peered":"2024-04-25T05:44:32.172830+0000","last_clean":"2024-04-25T05:44:32.172830+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:44:32.172830+0000","last_undegraded":"2024-04-25T05:44:32.172830+0000","last_fullsized":"2024-04-25T05:44:32.172830+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:28.637882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:28.637882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:40.481823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"329'7014","reported_seq":7089,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:37.891573+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:44:37.891573+0000","last_peered":"2024-04-25T05:44:37.891573+0000","last_clean":"2024-04-25T05:44:37.891573+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:44:37.891573+0000","last_undegraded":"2024-04-25T05:44:37.891573+0000","last_fullsized":"2024-04-25T05:44:37.891573+0000","mapping_epoch":296,"log_start":"316'3730","ondisk_log_start":"316'3730","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3284,"log_dups_size":0,"ondisk_log_size":3284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459669504,"num_objects":7014,"num_object_clones":0,"num_object_copies":21042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":448896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":304,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172309+0000","last_change":"2024-04-25T05:44:28.125054+0000","last_active":"2024-04-25T05:44:32.172309+0000","last_peered":"2024-04-25T05:44:32.172309+0000","last_clean":"2024-04-25T05:44:32.172309+0000","last_became_active":"2024-04-25T05:44:28.124674+0000","last_became_peered":"2024-04-25T05:44:28.124674+0000","last_unstale":"2024-04-25T05:44:32.172309+0000","last_undegraded":"2024-04-25T05:44:32.172309+0000","last_fullsized":"2024-04-25T05:44:32.172309+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:58.303045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":293,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170066+0000","last_change":"2024-04-25T05:44:28.198134+0000","last_active":"2024-04-25T05:44:32.170066+0000","last_peered":"2024-04-25T05:44:32.170066+0000","last_clean":"2024-04-25T05:44:32.170066+0000","last_became_active":"2024-04-25T05:44:28.168410+0000","last_became_peered":"2024-04-25T05:44:28.168410+0000","last_unstale":"2024-04-25T05:44:32.170066+0000","last_undegraded":"2024-04-25T05:44:32.170066+0000","last_fullsized":"2024-04-25T05:44:32.170066+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":589,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169292+0000","last_change":"2024-04-25T05:44:28.167276+0000","last_active":"2024-04-25T05:44:32.169292+0000","last_peered":"2024-04-25T05:44:32.169292+0000","last_clean":"2024-04-25T05:44:32.169292+0000","last_became_active":"2024-04-25T05:44:28.166968+0000","last_became_peered":"2024-04-25T05:44:28.166968+0000","last_unstale":"2024-04-25T05:44:32.169292+0000","last_undegraded":"2024-04-25T05:44:32.169292+0000","last_fullsized":"2024-04-25T05:44:32.169292+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:45.827541+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":569,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.162606+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:44:32.162606+0000","last_peered":"2024-04-25T05:44:32.162606+0000","last_clean":"2024-04-25T05:44:32.162606+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:44:32.162606+0000","last_undegraded":"2024-04-25T05:44:32.162606+0000","last_fullsized":"2024-04-25T05:44:32.162606+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":767,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172470+0000","last_change":"2024-04-25T05:44:30.918290+0000","last_active":"2024-04-25T05:44:32.172470+0000","last_peered":"2024-04-25T05:44:32.172470+0000","last_clean":"2024-04-25T05:44:32.172470+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:44:32.172470+0000","last_undegraded":"2024-04-25T05:44:32.172470+0000","last_fullsized":"2024-04-25T05:44:32.172470+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:20.780175+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.780175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:42.506221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"329'7016","reported_seq":9988,"reported_epoch":329,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:37.915899+0000","last_change":"2024-04-25T05:44:29.210942+0000","last_active":"2024-04-25T05:44:37.915899+0000","last_peered":"2024-04-25T05:44:37.915899+0000","last_clean":"2024-04-25T05:44:27.036002+0000","last_became_active":"2024-04-25T05:44:29.183071+0000","last_became_peered":"2024-04-25T05:44:29.183071+0000","last_unstale":"2024-04-25T05:44:37.915899+0000","last_undegraded":"2024-04-25T05:44:37.915899+0000","last_fullsized":"2024-04-25T05:44:37.915899+0000","mapping_epoch":325,"log_start":"315'3511","ondisk_log_start":"315'3511","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459800576,"num_objects":7016,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5518,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":449024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1443,"num_bytes_recovered":94568448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":7016}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":437,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.174984+0000","last_change":"2024-04-25T05:44:28.180356+0000","last_active":"2024-04-25T05:44:32.174984+0000","last_peered":"2024-04-25T05:44:32.174984+0000","last_clean":"2024-04-25T05:44:32.174984+0000","last_became_active":"2024-04-25T05:44:28.180059+0000","last_became_peered":"2024-04-25T05:44:28.180059+0000","last_unstale":"2024-04-25T05:44:32.174984+0000","last_undegraded":"2024-04-25T05:44:32.174984+0000","last_fullsized":"2024-04-25T05:44:32.174984+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.755552+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":111,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167057+0000","last_change":"2024-04-25T05:44:28.234082+0000","last_active":"2024-04-25T05:44:32.167057+0000","last_peered":"2024-04-25T05:44:32.167057+0000","last_clean":"2024-04-25T05:44:32.167057+0000","last_became_active":"2024-04-25T05:44:28.177715+0000","last_became_peered":"2024-04-25T05:44:28.177715+0000","last_unstale":"2024-04-25T05:44:32.167057+0000","last_undegraded":"2024-04-25T05:44:32.167057+0000","last_fullsized":"2024-04-25T05:44:32.167057+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:44.249745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1243,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:36.187156+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:44:36.187156+0000","last_peered":"2024-04-25T05:44:36.187156+0000","last_clean":"2024-04-25T05:44:36.187156+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:44:36.187156+0000","last_undegraded":"2024-04-25T05:44:36.187156+0000","last_fullsized":"2024-04-25T05:44:36.187156+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.162472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":621,"num_read_kb":616,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.151438+0000","last_change":"2024-04-25T05:44:28.172816+0000","last_active":"2024-04-25T05:44:32.151438+0000","last_peered":"2024-04-25T05:44:32.151438+0000","last_clean":"2024-04-25T05:44:32.151438+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:44:32.151438+0000","last_undegraded":"2024-04-25T05:44:32.151438+0000","last_fullsized":"2024-04-25T05:44:32.151438+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.421093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.421093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.272940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":557,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169064+0000","last_change":"2024-04-25T05:43:16.189190+0000","last_active":"2024-04-25T05:44:32.169064+0000","last_peered":"2024-04-25T05:44:32.169064+0000","last_clean":"2024-04-25T05:44:32.169064+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:44:32.169064+0000","last_undegraded":"2024-04-25T05:44:32.169064+0000","last_fullsized":"2024-04-25T05:44:32.169064+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":744,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170267+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:44:32.170267+0000","last_peered":"2024-04-25T05:44:32.170267+0000","last_clean":"2024-04-25T05:44:32.170267+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:44:32.170267+0000","last_undegraded":"2024-04-25T05:44:32.170267+0000","last_fullsized":"2024-04-25T05:44:32.170267+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"329'7131","reported_seq":9098,"reported_epoch":329,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:38.619873+0000","last_change":"2024-04-25T05:44:29.184637+0000","last_active":"2024-04-25T05:44:38.619873+0000","last_peered":"2024-04-25T05:44:38.619873+0000","last_clean":"2024-04-25T05:44:27.101546+0000","last_became_active":"2024-04-25T05:44:29.168753+0000","last_became_peered":"2024-04-25T05:44:29.168753+0000","last_unstale":"2024-04-25T05:44:38.619873+0000","last_undegraded":"2024-04-25T05:44:38.619873+0000","last_fullsized":"2024-04-25T05:44:38.619873+0000","mapping_epoch":325,"log_start":"317'3973","ondisk_log_start":"317'3973","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467337216,"num_objects":7131,"num_object_clones":0,"num_object_copies":21393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6147,"num_objects_unfound":0,"num_objects_dirty":7131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":456384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":61603840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":7131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":581,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170178+0000","last_change":"2024-04-25T05:44:28.178984+0000","last_active":"2024-04-25T05:44:32.170178+0000","last_peered":"2024-04-25T05:44:32.170178+0000","last_clean":"2024-04-25T05:44:32.170178+0000","last_became_active":"2024-04-25T05:44:28.178614+0000","last_became_peered":"2024-04-25T05:44:28.178614+0000","last_unstale":"2024-04-25T05:44:32.170178+0000","last_undegraded":"2024-04-25T05:44:32.170178+0000","last_fullsized":"2024-04-25T05:44:32.170178+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:23.945519+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":375,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172572+0000","last_change":"2024-04-25T05:43:22.635428+0000","last_active":"2024-04-25T05:44:32.172572+0000","last_peered":"2024-04-25T05:44:32.172572+0000","last_clean":"2024-04-25T05:44:32.172572+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:44:32.172572+0000","last_undegraded":"2024-04-25T05:44:32.172572+0000","last_fullsized":"2024-04-25T05:44:32.172572+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:43:22.635290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.635290+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:39.347414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":573,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172386+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:44:32.172386+0000","last_peered":"2024-04-25T05:44:32.172386+0000","last_clean":"2024-04-25T05:44:32.172386+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:44:32.172386+0000","last_undegraded":"2024-04-25T05:44:32.172386+0000","last_fullsized":"2024-04-25T05:44:32.172386+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":880,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168769+0000","last_change":"2024-04-25T05:44:30.948007+0000","last_active":"2024-04-25T05:44:32.168769+0000","last_peered":"2024-04-25T05:44:32.168769+0000","last_clean":"2024-04-25T05:44:32.168769+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:44:32.168769+0000","last_undegraded":"2024-04-25T05:44:32.168769+0000","last_fullsized":"2024-04-25T05:44:32.168769+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":66,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169305+0000","last_change":"2024-04-25T05:43:16.188274+0000","last_active":"2024-04-25T05:44:32.169305+0000","last_peered":"2024-04-25T05:44:32.169305+0000","last_clean":"2024-04-25T05:44:32.169305+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:44:32.169305+0000","last_undegraded":"2024-04-25T05:44:32.169305+0000","last_fullsized":"2024-04-25T05:44:32.169305+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"329'7269","reported_seq":7334,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.583785+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:44:38.583785+0000","last_peered":"2024-04-25T05:44:38.583785+0000","last_clean":"2024-04-25T05:44:38.583785+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:44:38.583785+0000","last_undegraded":"2024-04-25T05:44:38.583785+0000","last_fullsized":"2024-04-25T05:44:38.583785+0000","mapping_epoch":301,"log_start":"319'4370","ondisk_log_start":"319'4370","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":715,"ondisk_log_size":2899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476381184,"num_objects":7269,"num_object_clones":0,"num_object_copies":21807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":465216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171448+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:44:32.171448+0000","last_peered":"2024-04-25T05:44:32.171448+0000","last_clean":"2024-04-25T05:44:32.171448+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:44:32.171448+0000","last_undegraded":"2024-04-25T05:44:32.171448+0000","last_fullsized":"2024-04-25T05:44:32.171448+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:41.672026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":689,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.173563+0000","last_change":"2024-04-25T05:44:28.221414+0000","last_active":"2024-04-25T05:44:32.173563+0000","last_peered":"2024-04-25T05:44:32.173563+0000","last_clean":"2024-04-25T05:44:32.173563+0000","last_became_active":"2024-04-25T05:44:28.179973+0000","last_became_peered":"2024-04-25T05:44:28.179973+0000","last_unstale":"2024-04-25T05:44:32.173563+0000","last_undegraded":"2024-04-25T05:44:32.173563+0000","last_fullsized":"2024-04-25T05:44:32.173563+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:44.447685+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":582,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166828+0000","last_change":"2024-04-25T05:44:28.186329+0000","last_active":"2024-04-25T05:44:32.166828+0000","last_peered":"2024-04-25T05:44:32.166828+0000","last_clean":"2024-04-25T05:44:32.166828+0000","last_became_active":"2024-04-25T05:44:28.184251+0000","last_became_peered":"2024-04-25T05:44:28.184251+0000","last_unstale":"2024-04-25T05:44:32.166828+0000","last_undegraded":"2024-04-25T05:44:32.166828+0000","last_fullsized":"2024-04-25T05:44:32.166828+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":601,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.164913+0000","last_change":"2024-04-25T05:44:28.183966+0000","last_active":"2024-04-25T05:44:32.164913+0000","last_peered":"2024-04-25T05:44:32.164913+0000","last_clean":"2024-04-25T05:44:32.164913+0000","last_became_active":"2024-04-25T05:44:28.183528+0000","last_became_peered":"2024-04-25T05:44:28.183528+0000","last_unstale":"2024-04-25T05:44:32.164913+0000","last_undegraded":"2024-04-25T05:44:32.164913+0000","last_fullsized":"2024-04-25T05:44:32.164913+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.104924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":571,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171680+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:44:32.171680+0000","last_peered":"2024-04-25T05:44:32.171680+0000","last_clean":"2024-04-25T05:44:32.171680+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:44:32.171680+0000","last_undegraded":"2024-04-25T05:44:32.171680+0000","last_fullsized":"2024-04-25T05:44:32.171680+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:52.596050+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":878,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169278+0000","last_change":"2024-04-25T05:44:30.947331+0000","last_active":"2024-04-25T05:44:32.169278+0000","last_peered":"2024-04-25T05:44:32.169278+0000","last_clean":"2024-04-25T05:44:32.169278+0000","last_became_active":"2024-04-25T05:44:30.946674+0000","last_became_peered":"2024-04-25T05:44:30.946674+0000","last_unstale":"2024-04-25T05:44:32.169278+0000","last_undegraded":"2024-04-25T05:44:32.169278+0000","last_fullsized":"2024-04-25T05:44:32.169278+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"329'7105","reported_seq":15040,"reported_epoch":329,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:37.924144+0000","last_change":"2024-04-25T05:44:29.232879+0000","last_active":"2024-04-25T05:44:37.924144+0000","last_peered":"2024-04-25T05:44:37.924144+0000","last_clean":"2024-04-25T05:44:27.070032+0000","last_became_active":"2024-04-25T05:44:29.214548+0000","last_became_peered":"2024-04-25T05:44:29.214548+0000","last_unstale":"2024-04-25T05:44:37.924144+0000","last_undegraded":"2024-04-25T05:44:37.924144+0000","last_fullsized":"2024-04-25T05:44:37.924144+0000","mapping_epoch":325,"log_start":"316'3808","ondisk_log_start":"316'3808","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465633280,"num_objects":7105,"num_object_clones":0,"num_object_copies":21315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3023,"num_objects_unfound":0,"num_objects_dirty":7105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":454720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3924,"num_bytes_recovered":257163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":7105}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":188,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167577+0000","last_change":"2024-04-25T05:44:28.188205+0000","last_active":"2024-04-25T05:44:32.167577+0000","last_peered":"2024-04-25T05:44:32.167577+0000","last_clean":"2024-04-25T05:44:32.167577+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:44:32.167577+0000","last_undegraded":"2024-04-25T05:44:32.167577+0000","last_fullsized":"2024-04-25T05:44:32.167577+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":806,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169489+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:44:32.169489+0000","last_peered":"2024-04-25T05:44:32.169489+0000","last_clean":"2024-04-25T05:44:32.169489+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:44:32.169489+0000","last_undegraded":"2024-04-25T05:44:32.169489+0000","last_fullsized":"2024-04-25T05:44:32.169489+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":358,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171744+0000","last_change":"2024-04-25T05:44:28.160664+0000","last_active":"2024-04-25T05:44:32.171744+0000","last_peered":"2024-04-25T05:44:32.171744+0000","last_clean":"2024-04-25T05:44:32.171744+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:44:32.171744+0000","last_undegraded":"2024-04-25T05:44:32.171744+0000","last_fullsized":"2024-04-25T05:44:32.171744+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.320979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":102,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171321+0000","last_change":"2024-04-25T05:44:28.205511+0000","last_active":"2024-04-25T05:44:32.171321+0000","last_peered":"2024-04-25T05:44:32.171321+0000","last_clean":"2024-04-25T05:44:32.171321+0000","last_became_active":"2024-04-25T05:44:28.176104+0000","last_became_peered":"2024-04-25T05:44:28.176104+0000","last_unstale":"2024-04-25T05:44:32.171321+0000","last_undegraded":"2024-04-25T05:44:32.171321+0000","last_fullsized":"2024-04-25T05:44:32.171321+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":542,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168756+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:44:32.168756+0000","last_peered":"2024-04-25T05:44:32.168756+0000","last_clean":"2024-04-25T05:44:32.168756+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:44:32.168756+0000","last_undegraded":"2024-04-25T05:44:32.168756+0000","last_fullsized":"2024-04-25T05:44:32.168756+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":583,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.151072+0000","last_change":"2024-04-25T05:44:30.918203+0000","last_active":"2024-04-25T05:44:32.151072+0000","last_peered":"2024-04-25T05:44:32.151072+0000","last_clean":"2024-04-25T05:44:32.151072+0000","last_became_active":"2024-04-25T05:44:30.917713+0000","last_became_peered":"2024-04-25T05:44:30.917713+0000","last_unstale":"2024-04-25T05:44:32.151072+0000","last_undegraded":"2024-04-25T05:44:32.151072+0000","last_fullsized":"2024-04-25T05:44:32.151072+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.424874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"329'7089","reported_seq":9577,"reported_epoch":329,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:38.622375+0000","last_change":"2024-04-25T05:44:29.205531+0000","last_active":"2024-04-25T05:44:38.622375+0000","last_peered":"2024-04-25T05:44:38.622375+0000","last_clean":"2024-04-25T05:44:27.108028+0000","last_became_active":"2024-04-25T05:44:29.173771+0000","last_became_peered":"2024-04-25T05:44:29.173771+0000","last_unstale":"2024-04-25T05:44:38.622375+0000","last_undegraded":"2024-04-25T05:44:38.622375+0000","last_fullsized":"2024-04-25T05:44:38.622375+0000","mapping_epoch":325,"log_start":"318'4191","ondisk_log_start":"318'4191","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2898,"log_dups_size":715,"ondisk_log_size":2898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464584704,"num_objects":7089,"num_object_clones":0,"num_object_copies":21267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5826,"num_objects_unfound":0,"num_objects_dirty":7089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":453696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":79101952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":7089}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":281,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167576+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:44:32.167576+0000","last_peered":"2024-04-25T05:44:32.167576+0000","last_clean":"2024-04-25T05:44:32.167576+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:44:32.167576+0000","last_undegraded":"2024-04-25T05:44:32.167576+0000","last_fullsized":"2024-04-25T05:44:32.167576+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":372,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169368+0000","last_change":"2024-04-25T05:43:16.191937+0000","last_active":"2024-04-25T05:44:32.169368+0000","last_peered":"2024-04-25T05:44:32.169368+0000","last_clean":"2024-04-25T05:44:32.169368+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:44:32.169368+0000","last_undegraded":"2024-04-25T05:44:32.169368+0000","last_fullsized":"2024-04-25T05:44:32.169368+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":211,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.152230+0000","last_change":"2024-04-25T05:44:28.171402+0000","last_active":"2024-04-25T05:44:32.152230+0000","last_peered":"2024-04-25T05:44:32.152230+0000","last_clean":"2024-04-25T05:44:32.152230+0000","last_became_active":"2024-04-25T05:44:28.171004+0000","last_became_peered":"2024-04-25T05:44:28.171004+0000","last_unstale":"2024-04-25T05:44:32.152230+0000","last_undegraded":"2024-04-25T05:44:32.152230+0000","last_fullsized":"2024-04-25T05:44:32.152230+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.186275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.186275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:20.209466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171697+0000","last_change":"2024-04-25T05:44:28.169508+0000","last_active":"2024-04-25T05:44:32.171697+0000","last_peered":"2024-04-25T05:44:32.171697+0000","last_clean":"2024-04-25T05:44:32.171697+0000","last_became_active":"2024-04-25T05:44:28.169146+0000","last_became_peered":"2024-04-25T05:44:28.169146+0000","last_unstale":"2024-04-25T05:44:32.171697+0000","last_undegraded":"2024-04-25T05:44:32.171697+0000","last_fullsized":"2024-04-25T05:44:32.171697+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"329'6927","reported_seq":6993,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:37.857392+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:44:37.857392+0000","last_peered":"2024-04-25T05:44:37.857392+0000","last_clean":"2024-04-25T05:44:37.857392+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:44:37.857392+0000","last_undegraded":"2024-04-25T05:44:37.857392+0000","last_fullsized":"2024-04-25T05:44:37.857392+0000","mapping_epoch":301,"log_start":"316'3655","ondisk_log_start":"316'3655","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453967872,"num_objects":6927,"num_object_clones":0,"num_object_copies":20781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":332,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172186+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:44:32.172186+0000","last_peered":"2024-04-25T05:44:32.172186+0000","last_clean":"2024-04-25T05:44:32.172186+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:44:32.172186+0000","last_undegraded":"2024-04-25T05:44:32.172186+0000","last_fullsized":"2024-04-25T05:44:32.172186+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:46.695440+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":606,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171427+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:44:32.171427+0000","last_peered":"2024-04-25T05:44:32.171427+0000","last_clean":"2024-04-25T05:44:32.171427+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:44:32.171427+0000","last_undegraded":"2024-04-25T05:44:32.171427+0000","last_fullsized":"2024-04-25T05:44:32.171427+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":515,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169116+0000","last_change":"2024-04-25T05:44:28.180755+0000","last_active":"2024-04-25T05:44:32.169116+0000","last_peered":"2024-04-25T05:44:32.169116+0000","last_clean":"2024-04-25T05:44:32.169116+0000","last_became_active":"2024-04-25T05:44:28.180393+0000","last_became_peered":"2024-04-25T05:44:28.180393+0000","last_unstale":"2024-04-25T05:44:32.169116+0000","last_undegraded":"2024-04-25T05:44:32.169116+0000","last_fullsized":"2024-04-25T05:44:32.169116+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.534093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":564,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168589+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:44:32.168589+0000","last_peered":"2024-04-25T05:44:32.168589+0000","last_clean":"2024-04-25T05:44:32.168589+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:44:32.168589+0000","last_undegraded":"2024-04-25T05:44:32.168589+0000","last_fullsized":"2024-04-25T05:44:32.168589+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.832260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"329'7192","reported_seq":7267,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:38.563995+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:44:38.563995+0000","last_peered":"2024-04-25T05:44:38.563995+0000","last_clean":"2024-04-25T05:44:38.563995+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:44:38.563995+0000","last_undegraded":"2024-04-25T05:44:38.563995+0000","last_fullsized":"2024-04-25T05:44:38.563995+0000","mapping_epoch":296,"log_start":"319'4286","ondisk_log_start":"319'4286","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":715,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471334912,"num_objects":7192,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7192,"num_write_kb":460288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170593+0000","last_change":"2024-04-25T05:44:28.169954+0000","last_active":"2024-04-25T05:44:32.170593+0000","last_peered":"2024-04-25T05:44:32.170593+0000","last_clean":"2024-04-25T05:44:32.170593+0000","last_became_active":"2024-04-25T05:44:28.168753+0000","last_became_peered":"2024-04-25T05:44:28.168753+0000","last_unstale":"2024-04-25T05:44:32.170593+0000","last_undegraded":"2024-04-25T05:44:32.170593+0000","last_fullsized":"2024-04-25T05:44:32.170593+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:41.236874+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":220,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.162293+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:44:32.162293+0000","last_peered":"2024-04-25T05:44:32.162293+0000","last_clean":"2024-04-25T05:44:32.162293+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:44:32.162293+0000","last_undegraded":"2024-04-25T05:44:32.162293+0000","last_fullsized":"2024-04-25T05:44:32.162293+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.771369+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":570,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170955+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:44:32.170955+0000","last_peered":"2024-04-25T05:44:32.170955+0000","last_clean":"2024-04-25T05:44:32.170955+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:44:32.170955+0000","last_undegraded":"2024-04-25T05:44:32.170955+0000","last_fullsized":"2024-04-25T05:44:32.170955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":554,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168830+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:44:32.168830+0000","last_peered":"2024-04-25T05:44:32.168830+0000","last_clean":"2024-04-25T05:44:32.168830+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:44:32.168830+0000","last_undegraded":"2024-04-25T05:44:32.168830+0000","last_fullsized":"2024-04-25T05:44:32.168830+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:33.061319+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"329'7036","reported_seq":10165,"reported_epoch":329,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:37.926491+0000","last_change":"2024-04-25T05:44:29.255024+0000","last_active":"2024-04-25T05:44:37.926491+0000","last_peered":"2024-04-25T05:44:37.926491+0000","last_clean":"2024-04-25T05:44:27.090244+0000","last_became_active":"2024-04-25T05:44:29.209023+0000","last_became_peered":"2024-04-25T05:44:29.209023+0000","last_unstale":"2024-04-25T05:44:37.926491+0000","last_undegraded":"2024-04-25T05:44:37.926491+0000","last_fullsized":"2024-04-25T05:44:37.926491+0000","mapping_epoch":325,"log_start":"316'3793","ondisk_log_start":"316'3793","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461111296,"num_objects":7036,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5464,"num_objects_unfound":0,"num_objects_dirty":7036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7036,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1520,"num_bytes_recovered":99614720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":7036}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":722,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172017+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:44:32.172017+0000","last_peered":"2024-04-25T05:44:32.172017+0000","last_clean":"2024-04-25T05:44:32.172017+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:44:32.172017+0000","last_undegraded":"2024-04-25T05:44:32.172017+0000","last_fullsized":"2024-04-25T05:44:32.172017+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":172,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168831+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:44:32.168831+0000","last_peered":"2024-04-25T05:44:32.168831+0000","last_clean":"2024-04-25T05:44:32.168831+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:44:32.168831+0000","last_undegraded":"2024-04-25T05:44:32.168831+0000","last_fullsized":"2024-04-25T05:44:32.168831+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":618,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169849+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:44:32.169849+0000","last_peered":"2024-04-25T05:44:32.169849+0000","last_clean":"2024-04-25T05:44:32.169849+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:44:32.169849+0000","last_undegraded":"2024-04-25T05:44:32.169849+0000","last_fullsized":"2024-04-25T05:44:32.169849+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.858926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":445,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172088+0000","last_change":"2024-04-25T05:44:28.174865+0000","last_active":"2024-04-25T05:44:32.172088+0000","last_peered":"2024-04-25T05:44:32.172088+0000","last_clean":"2024-04-25T05:44:32.172088+0000","last_became_active":"2024-04-25T05:44:28.174499+0000","last_became_peered":"2024-04-25T05:44:28.174499+0000","last_unstale":"2024-04-25T05:44:32.172088+0000","last_undegraded":"2024-04-25T05:44:32.172088+0000","last_fullsized":"2024-04-25T05:44:32.172088+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:50.432715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":588,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172746+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:44:32.172746+0000","last_peered":"2024-04-25T05:44:32.172746+0000","last_clean":"2024-04-25T05:44:32.172746+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:44:32.172746+0000","last_undegraded":"2024-04-25T05:44:32.172746+0000","last_fullsized":"2024-04-25T05:44:32.172746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.253497+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":304,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170931+0000","last_change":"2024-04-25T05:44:28.191586+0000","last_active":"2024-04-25T05:44:32.170931+0000","last_peered":"2024-04-25T05:44:32.170931+0000","last_clean":"2024-04-25T05:44:32.170931+0000","last_became_active":"2024-04-25T05:44:28.191288+0000","last_became_peered":"2024-04-25T05:44:28.191288+0000","last_unstale":"2024-04-25T05:44:32.170931+0000","last_undegraded":"2024-04-25T05:44:32.170931+0000","last_fullsized":"2024-04-25T05:44:32.170931+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.539924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":74,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168641+0000","last_change":"2024-04-25T05:44:19.931291+0000","last_active":"2024-04-25T05:44:32.168641+0000","last_peered":"2024-04-25T05:44:32.168641+0000","last_clean":"2024-04-25T05:44:32.168641+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:44:32.168641+0000","last_undegraded":"2024-04-25T05:44:32.168641+0000","last_fullsized":"2024-04-25T05:44:32.168641+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.670336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":570,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171048+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:44:32.171048+0000","last_peered":"2024-04-25T05:44:32.171048+0000","last_clean":"2024-04-25T05:44:32.171048+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:44:32.171048+0000","last_undegraded":"2024-04-25T05:44:32.171048+0000","last_fullsized":"2024-04-25T05:44:32.171048+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:06.898722+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":249,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168707+0000","last_change":"2024-04-25T05:44:28.203127+0000","last_active":"2024-04-25T05:44:32.168707+0000","last_peered":"2024-04-25T05:44:32.168707+0000","last_clean":"2024-04-25T05:44:32.168707+0000","last_became_active":"2024-04-25T05:44:28.184563+0000","last_became_peered":"2024-04-25T05:44:28.184563+0000","last_unstale":"2024-04-25T05:44:32.168707+0000","last_undegraded":"2024-04-25T05:44:32.168707+0000","last_fullsized":"2024-04-25T05:44:32.168707+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":588,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171861+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:44:32.171861+0000","last_peered":"2024-04-25T05:44:32.171861+0000","last_clean":"2024-04-25T05:44:32.171861+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:44:32.171861+0000","last_undegraded":"2024-04-25T05:44:32.171861+0000","last_fullsized":"2024-04-25T05:44:32.171861+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:59.461950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":546,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169391+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:44:32.169391+0000","last_peered":"2024-04-25T05:44:32.169391+0000","last_clean":"2024-04-25T05:44:32.169391+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:44:32.169391+0000","last_undegraded":"2024-04-25T05:44:32.169391+0000","last_fullsized":"2024-04-25T05:44:32.169391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.522496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":212,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.154404+0000","last_change":"2024-04-25T05:44:28.120205+0000","last_active":"2024-04-25T05:44:32.154404+0000","last_peered":"2024-04-25T05:44:32.154404+0000","last_clean":"2024-04-25T05:44:32.154404+0000","last_became_active":"2024-04-25T05:44:28.119749+0000","last_became_peered":"2024-04-25T05:44:28.119749+0000","last_unstale":"2024-04-25T05:44:32.154404+0000","last_undegraded":"2024-04-25T05:44:32.154404+0000","last_fullsized":"2024-04-25T05:44:32.154404+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.285907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":191,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171534+0000","last_change":"2024-04-25T05:43:16.229957+0000","last_active":"2024-04-25T05:44:32.171534+0000","last_peered":"2024-04-25T05:44:32.171534+0000","last_clean":"2024-04-25T05:44:32.171534+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:44:32.171534+0000","last_undegraded":"2024-04-25T05:44:32.171534+0000","last_fullsized":"2024-04-25T05:44:32.171534+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169017+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:44:32.169017+0000","last_peered":"2024-04-25T05:44:32.169017+0000","last_clean":"2024-04-25T05:44:32.169017+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:44:32.169017+0000","last_undegraded":"2024-04-25T05:44:32.169017+0000","last_fullsized":"2024-04-25T05:44:32.169017+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.919590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":692,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.173359+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:44:32.173359+0000","last_peered":"2024-04-25T05:44:32.173359+0000","last_clean":"2024-04-25T05:44:32.173359+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:44:32.173359+0000","last_undegraded":"2024-04-25T05:44:32.173359+0000","last_fullsized":"2024-04-25T05:44:32.173359+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.732091+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":26,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169618+0000","last_change":"2024-04-25T05:44:28.167266+0000","last_active":"2024-04-25T05:44:32.169618+0000","last_peered":"2024-04-25T05:44:32.169618+0000","last_clean":"2024-04-25T05:44:32.169618+0000","last_became_active":"2024-04-25T05:44:28.166838+0000","last_became_peered":"2024-04-25T05:44:28.166838+0000","last_unstale":"2024-04-25T05:44:32.169618+0000","last_undegraded":"2024-04-25T05:44:32.169618+0000","last_fullsized":"2024-04-25T05:44:32.169618+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.460917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":604,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169254+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:44:32.169254+0000","last_peered":"2024-04-25T05:44:32.169254+0000","last_clean":"2024-04-25T05:44:32.169254+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:44:32.169254+0000","last_undegraded":"2024-04-25T05:44:32.169254+0000","last_fullsized":"2024-04-25T05:44:32.169254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":576,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169262+0000","last_change":"2024-04-25T05:44:13.015441+0000","last_active":"2024-04-25T05:44:32.169262+0000","last_peered":"2024-04-25T05:44:32.169262+0000","last_clean":"2024-04-25T05:44:32.169262+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:44:32.169262+0000","last_undegraded":"2024-04-25T05:44:32.169262+0000","last_fullsized":"2024-04-25T05:44:32.169262+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:27.074191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":121,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166297+0000","last_change":"2024-04-25T05:44:28.236616+0000","last_active":"2024-04-25T05:44:32.166297+0000","last_peered":"2024-04-25T05:44:32.166297+0000","last_clean":"2024-04-25T05:44:32.166297+0000","last_became_active":"2024-04-25T05:44:28.184915+0000","last_became_peered":"2024-04-25T05:44:28.184915+0000","last_unstale":"2024-04-25T05:44:32.166297+0000","last_undegraded":"2024-04-25T05:44:32.166297+0000","last_fullsized":"2024-04-25T05:44:32.166297+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170720+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:44:32.170720+0000","last_peered":"2024-04-25T05:44:32.170720+0000","last_clean":"2024-04-25T05:44:32.170720+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:44:32.170720+0000","last_undegraded":"2024-04-25T05:44:32.170720+0000","last_fullsized":"2024-04-25T05:44:32.170720+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:49.527243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":682,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.152873+0000","last_change":"2024-04-25T05:44:19.254245+0000","last_active":"2024-04-25T05:44:32.152873+0000","last_peered":"2024-04-25T05:44:32.152873+0000","last_clean":"2024-04-25T05:44:32.152873+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:44:32.152873+0000","last_undegraded":"2024-04-25T05:44:32.152873+0000","last_fullsized":"2024-04-25T05:44:32.152873+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:19.254124+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.254124+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:41.554540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":241,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169479+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:44:32.169479+0000","last_peered":"2024-04-25T05:44:32.169479+0000","last_clean":"2024-04-25T05:44:32.169479+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:44:32.169479+0000","last_undegraded":"2024-04-25T05:44:32.169479+0000","last_fullsized":"2024-04-25T05:44:32.169479+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":563,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171488+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:44:32.171488+0000","last_peered":"2024-04-25T05:44:32.171488+0000","last_clean":"2024-04-25T05:44:32.171488+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:44:32.171488+0000","last_undegraded":"2024-04-25T05:44:32.171488+0000","last_fullsized":"2024-04-25T05:44:32.171488+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:42.530876+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":574,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166733+0000","last_change":"2024-04-25T05:44:28.186508+0000","last_active":"2024-04-25T05:44:32.166733+0000","last_peered":"2024-04-25T05:44:32.166733+0000","last_clean":"2024-04-25T05:44:32.166733+0000","last_became_active":"2024-04-25T05:44:28.184914+0000","last_became_peered":"2024-04-25T05:44:28.184914+0000","last_unstale":"2024-04-25T05:44:32.166733+0000","last_undegraded":"2024-04-25T05:44:32.166733+0000","last_fullsized":"2024-04-25T05:44:32.166733+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:54.248226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":100,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167381+0000","last_change":"2024-04-25T05:44:28.208196+0000","last_active":"2024-04-25T05:44:32.167381+0000","last_peered":"2024-04-25T05:44:32.167381+0000","last_clean":"2024-04-25T05:44:32.167381+0000","last_became_active":"2024-04-25T05:44:28.160392+0000","last_became_peered":"2024-04-25T05:44:28.160392+0000","last_unstale":"2024-04-25T05:44:32.167381+0000","last_undegraded":"2024-04-25T05:44:32.167381+0000","last_fullsized":"2024-04-25T05:44:32.167381+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":570,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171155+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:44:32.171155+0000","last_peered":"2024-04-25T05:44:32.171155+0000","last_clean":"2024-04-25T05:44:32.171155+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:44:32.171155+0000","last_undegraded":"2024-04-25T05:44:32.171155+0000","last_fullsized":"2024-04-25T05:44:32.171155+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":564,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168853+0000","last_change":"2024-04-25T05:44:28.174271+0000","last_active":"2024-04-25T05:44:32.168853+0000","last_peered":"2024-04-25T05:44:32.168853+0000","last_clean":"2024-04-25T05:44:32.168853+0000","last_became_active":"2024-04-25T05:44:28.172179+0000","last_became_peered":"2024-04-25T05:44:28.172179+0000","last_unstale":"2024-04-25T05:44:32.168853+0000","last_undegraded":"2024-04-25T05:44:32.168853+0000","last_fullsized":"2024-04-25T05:44:32.168853+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.204906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":175,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170575+0000","last_change":"2024-04-25T05:44:28.179459+0000","last_active":"2024-04-25T05:44:32.170575+0000","last_peered":"2024-04-25T05:44:32.170575+0000","last_clean":"2024-04-25T05:44:32.170575+0000","last_became_active":"2024-04-25T05:44:28.176134+0000","last_became_peered":"2024-04-25T05:44:28.176134+0000","last_unstale":"2024-04-25T05:44:32.170575+0000","last_undegraded":"2024-04-25T05:44:32.170575+0000","last_fullsized":"2024-04-25T05:44:32.170575+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":641,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167040+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:44:32.167040+0000","last_peered":"2024-04-25T05:44:32.167040+0000","last_clean":"2024-04-25T05:44:32.167040+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:44:32.167040+0000","last_undegraded":"2024-04-25T05:44:32.167040+0000","last_fullsized":"2024-04-25T05:44:32.167040+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.001968+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":74,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170870+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:44:32.170870+0000","last_peered":"2024-04-25T05:44:32.170870+0000","last_clean":"2024-04-25T05:44:32.170870+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:44:32.170870+0000","last_undegraded":"2024-04-25T05:44:32.170870+0000","last_fullsized":"2024-04-25T05:44:32.170870+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:42.578695+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":580,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169005+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:44:32.169005+0000","last_peered":"2024-04-25T05:44:32.169005+0000","last_clean":"2024-04-25T05:44:32.169005+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:44:32.169005+0000","last_undegraded":"2024-04-25T05:44:32.169005+0000","last_fullsized":"2024-04-25T05:44:32.169005+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:35.164571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":572,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168959+0000","last_change":"2024-04-25T05:44:28.179442+0000","last_active":"2024-04-25T05:44:32.168959+0000","last_peered":"2024-04-25T05:44:32.168959+0000","last_clean":"2024-04-25T05:44:32.168959+0000","last_became_active":"2024-04-25T05:44:28.179103+0000","last_became_peered":"2024-04-25T05:44:28.179103+0000","last_unstale":"2024-04-25T05:44:32.168959+0000","last_undegraded":"2024-04-25T05:44:32.168959+0000","last_fullsized":"2024-04-25T05:44:32.168959+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":586,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169440+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:44:32.169440+0000","last_peered":"2024-04-25T05:44:32.169440+0000","last_clean":"2024-04-25T05:44:32.169440+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:44:32.169440+0000","last_undegraded":"2024-04-25T05:44:32.169440+0000","last_fullsized":"2024-04-25T05:44:32.169440+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":167,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169027+0000","last_change":"2024-04-25T05:44:28.188135+0000","last_active":"2024-04-25T05:44:32.169027+0000","last_peered":"2024-04-25T05:44:32.169027+0000","last_clean":"2024-04-25T05:44:32.169027+0000","last_became_active":"2024-04-25T05:44:28.187806+0000","last_became_peered":"2024-04-25T05:44:28.187806+0000","last_unstale":"2024-04-25T05:44:32.169027+0000","last_undegraded":"2024-04-25T05:44:32.169027+0000","last_fullsized":"2024-04-25T05:44:32.169027+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:38.153293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":278,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171418+0000","last_change":"2024-04-25T05:43:29.610487+0000","last_active":"2024-04-25T05:44:32.171418+0000","last_peered":"2024-04-25T05:44:32.171418+0000","last_clean":"2024-04-25T05:44:32.171418+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:44:32.171418+0000","last_undegraded":"2024-04-25T05:44:32.171418+0000","last_fullsized":"2024-04-25T05:44:32.171418+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.610411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.610411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":569,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169337+0000","last_change":"2024-04-25T05:44:19.960606+0000","last_active":"2024-04-25T05:44:32.169337+0000","last_peered":"2024-04-25T05:44:32.169337+0000","last_clean":"2024-04-25T05:44:32.169337+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:44:32.169337+0000","last_undegraded":"2024-04-25T05:44:32.169337+0000","last_fullsized":"2024-04-25T05:44:32.169337+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:24.771833+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":358,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172941+0000","last_change":"2024-04-25T05:44:28.212080+0000","last_active":"2024-04-25T05:44:32.172941+0000","last_peered":"2024-04-25T05:44:32.172941+0000","last_clean":"2024-04-25T05:44:32.172941+0000","last_became_active":"2024-04-25T05:44:28.181996+0000","last_became_peered":"2024-04-25T05:44:28.181996+0000","last_unstale":"2024-04-25T05:44:32.172941+0000","last_undegraded":"2024-04-25T05:44:32.172941+0000","last_fullsized":"2024-04-25T05:44:32.172941+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.299319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":580,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167102+0000","last_change":"2024-04-25T05:44:28.172975+0000","last_active":"2024-04-25T05:44:32.167102+0000","last_peered":"2024-04-25T05:44:32.167102+0000","last_clean":"2024-04-25T05:44:32.167102+0000","last_became_active":"2024-04-25T05:44:28.171311+0000","last_became_peered":"2024-04-25T05:44:28.171311+0000","last_unstale":"2024-04-25T05:44:32.167102+0000","last_undegraded":"2024-04-25T05:44:32.167102+0000","last_fullsized":"2024-04-25T05:44:32.167102+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:37.398601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171175+0000","last_change":"2024-04-25T05:44:28.179599+0000","last_active":"2024-04-25T05:44:32.171175+0000","last_peered":"2024-04-25T05:44:32.171175+0000","last_clean":"2024-04-25T05:44:32.171175+0000","last_became_active":"2024-04-25T05:44:28.178710+0000","last_became_peered":"2024-04-25T05:44:28.178710+0000","last_unstale":"2024-04-25T05:44:32.171175+0000","last_undegraded":"2024-04-25T05:44:32.171175+0000","last_fullsized":"2024-04-25T05:44:32.171175+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:32.286921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":556,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166127+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:44:32.166127+0000","last_peered":"2024-04-25T05:44:32.166127+0000","last_clean":"2024-04-25T05:44:32.166127+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:44:32.166127+0000","last_undegraded":"2024-04-25T05:44:32.166127+0000","last_fullsized":"2024-04-25T05:44:32.166127+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:50.241779+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":255,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.171341+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:44:32.171341+0000","last_peered":"2024-04-25T05:44:32.171341+0000","last_clean":"2024-04-25T05:44:32.171341+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:44:32.171341+0000","last_undegraded":"2024-04-25T05:44:32.171341+0000","last_fullsized":"2024-04-25T05:44:32.171341+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:37.615379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":571,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.162395+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:44:32.162395+0000","last_peered":"2024-04-25T05:44:32.162395+0000","last_clean":"2024-04-25T05:44:32.162395+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:44:32.162395+0000","last_undegraded":"2024-04-25T05:44:32.162395+0000","last_fullsized":"2024-04-25T05:44:32.162395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:40.500720+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":351,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172816+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:44:32.172816+0000","last_peered":"2024-04-25T05:44:32.172816+0000","last_clean":"2024-04-25T05:44:32.172816+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:44:32.172816+0000","last_undegraded":"2024-04-25T05:44:32.172816+0000","last_fullsized":"2024-04-25T05:44:32.172816+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.185065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":579,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170228+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:44:32.170228+0000","last_peered":"2024-04-25T05:44:32.170228+0000","last_clean":"2024-04-25T05:44:32.170228+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:44:32.170228+0000","last_undegraded":"2024-04-25T05:44:32.170228+0000","last_fullsized":"2024-04-25T05:44:32.170228+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":594,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168769+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:44:32.168769+0000","last_peered":"2024-04-25T05:44:32.168769+0000","last_clean":"2024-04-25T05:44:32.168769+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:44:32.168769+0000","last_undegraded":"2024-04-25T05:44:32.168769+0000","last_fullsized":"2024-04-25T05:44:32.168769+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:44.866351+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":582,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.152048+0000","last_change":"2024-04-25T05:44:28.167895+0000","last_active":"2024-04-25T05:44:32.152048+0000","last_peered":"2024-04-25T05:44:32.152048+0000","last_clean":"2024-04-25T05:44:32.152048+0000","last_became_active":"2024-04-25T05:44:28.167563+0000","last_became_peered":"2024-04-25T05:44:28.167563+0000","last_unstale":"2024-04-25T05:44:32.152048+0000","last_undegraded":"2024-04-25T05:44:32.152048+0000","last_fullsized":"2024-04-25T05:44:32.152048+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.192432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.192432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:44.936516+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":239,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.173093+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:44:32.173093+0000","last_peered":"2024-04-25T05:44:32.173093+0000","last_clean":"2024-04-25T05:44:32.173093+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:44:32.173093+0000","last_undegraded":"2024-04-25T05:44:32.173093+0000","last_fullsized":"2024-04-25T05:44:32.173093+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":561,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172652+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:44:32.172652+0000","last_peered":"2024-04-25T05:44:32.172652+0000","last_clean":"2024-04-25T05:44:32.172652+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:44:32.172652+0000","last_undegraded":"2024-04-25T05:44:32.172652+0000","last_fullsized":"2024-04-25T05:44:32.172652+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":461,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169183+0000","last_change":"2024-04-25T05:44:28.237495+0000","last_active":"2024-04-25T05:44:32.169183+0000","last_peered":"2024-04-25T05:44:32.169183+0000","last_clean":"2024-04-25T05:44:32.169183+0000","last_became_active":"2024-04-25T05:44:28.182958+0000","last_became_peered":"2024-04-25T05:44:28.182958+0000","last_unstale":"2024-04-25T05:44:32.169183+0000","last_undegraded":"2024-04-25T05:44:32.169183+0000","last_fullsized":"2024-04-25T05:44:32.169183+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169518+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:44:32.169518+0000","last_peered":"2024-04-25T05:44:32.169518+0000","last_clean":"2024-04-25T05:44:32.169518+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:44:32.169518+0000","last_undegraded":"2024-04-25T05:44:32.169518+0000","last_fullsized":"2024-04-25T05:44:32.169518+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:21.442272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":580,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.166414+0000","last_change":"2024-04-25T05:44:28.189092+0000","last_active":"2024-04-25T05:44:32.166414+0000","last_peered":"2024-04-25T05:44:32.166414+0000","last_clean":"2024-04-25T05:44:32.166414+0000","last_became_active":"2024-04-25T05:44:28.185597+0000","last_became_peered":"2024-04-25T05:44:28.185597+0000","last_unstale":"2024-04-25T05:44:32.166414+0000","last_undegraded":"2024-04-25T05:44:32.166414+0000","last_fullsized":"2024-04-25T05:44:32.166414+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.532823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":577,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.153834+0000","last_change":"2024-04-25T05:44:00.169219+0000","last_active":"2024-04-25T05:44:32.153834+0000","last_peered":"2024-04-25T05:44:32.153834+0000","last_clean":"2024-04-25T05:44:32.153834+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:44:32.153834+0000","last_undegraded":"2024-04-25T05:44:32.153834+0000","last_fullsized":"2024-04-25T05:44:32.153834+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.169057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.169057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.657106+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":304,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168597+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:44:32.168597+0000","last_peered":"2024-04-25T05:44:32.168597+0000","last_clean":"2024-04-25T05:44:32.168597+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:44:32.168597+0000","last_undegraded":"2024-04-25T05:44:32.168597+0000","last_fullsized":"2024-04-25T05:44:32.168597+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":74,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.154099+0000","last_change":"2024-04-25T05:43:17.283718+0000","last_active":"2024-04-25T05:44:32.154099+0000","last_peered":"2024-04-25T05:44:32.154099+0000","last_clean":"2024-04-25T05:44:32.154099+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:44:32.154099+0000","last_undegraded":"2024-04-25T05:44:32.154099+0000","last_fullsized":"2024-04-25T05:44:32.154099+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.283618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.283618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:39.318852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":554,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169169+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:44:32.169169+0000","last_peered":"2024-04-25T05:44:32.169169+0000","last_clean":"2024-04-25T05:44:32.169169+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:44:32.169169+0000","last_undegraded":"2024-04-25T05:44:32.169169+0000","last_fullsized":"2024-04-25T05:44:32.169169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":72,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.162654+0000","last_change":"2024-04-25T05:44:28.201019+0000","last_active":"2024-04-25T05:44:32.162654+0000","last_peered":"2024-04-25T05:44:32.162654+0000","last_clean":"2024-04-25T05:44:32.162654+0000","last_became_active":"2024-04-25T05:44:28.200623+0000","last_became_peered":"2024-04-25T05:44:28.200623+0000","last_unstale":"2024-04-25T05:44:32.162654+0000","last_undegraded":"2024-04-25T05:44:32.162654+0000","last_fullsized":"2024-04-25T05:44:32.162654+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":341,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172480+0000","last_change":"2024-04-25T05:44:28.198166+0000","last_active":"2024-04-25T05:44:32.172480+0000","last_peered":"2024-04-25T05:44:32.172480+0000","last_clean":"2024-04-25T05:44:32.172480+0000","last_became_active":"2024-04-25T05:44:28.179156+0000","last_became_peered":"2024-04-25T05:44:28.179156+0000","last_unstale":"2024-04-25T05:44:32.172480+0000","last_undegraded":"2024-04-25T05:44:32.172480+0000","last_fullsized":"2024-04-25T05:44:32.172480+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":380,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172343+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:44:32.172343+0000","last_peered":"2024-04-25T05:44:32.172343+0000","last_clean":"2024-04-25T05:44:32.172343+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:44:32.172343+0000","last_undegraded":"2024-04-25T05:44:32.172343+0000","last_fullsized":"2024-04-25T05:44:32.172343+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.453574+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":378,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.169421+0000","last_change":"2024-04-25T05:44:28.126200+0000","last_active":"2024-04-25T05:44:32.169421+0000","last_peered":"2024-04-25T05:44:32.169421+0000","last_clean":"2024-04-25T05:44:32.169421+0000","last_became_active":"2024-04-25T05:44:28.125552+0000","last_became_peered":"2024-04-25T05:44:28.125552+0000","last_unstale":"2024-04-25T05:44:32.169421+0000","last_undegraded":"2024-04-25T05:44:32.169421+0000","last_fullsized":"2024-04-25T05:44:32.169421+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":182,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170665+0000","last_change":"2024-04-25T05:44:28.169494+0000","last_active":"2024-04-25T05:44:32.170665+0000","last_peered":"2024-04-25T05:44:32.170665+0000","last_clean":"2024-04-25T05:44:32.170665+0000","last_became_active":"2024-04-25T05:44:28.168461+0000","last_became_peered":"2024-04-25T05:44:28.168461+0000","last_unstale":"2024-04-25T05:44:32.170665+0000","last_undegraded":"2024-04-25T05:44:32.170665+0000","last_fullsized":"2024-04-25T05:44:32.170665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":791,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167146+0000","last_change":"2024-04-25T05:44:28.225524+0000","last_active":"2024-04-25T05:44:32.167146+0000","last_peered":"2024-04-25T05:44:32.167146+0000","last_clean":"2024-04-25T05:44:32.167146+0000","last_became_active":"2024-04-25T05:44:28.182800+0000","last_became_peered":"2024-04-25T05:44:28.182800+0000","last_unstale":"2024-04-25T05:44:32.167146+0000","last_undegraded":"2024-04-25T05:44:32.167146+0000","last_fullsized":"2024-04-25T05:44:32.167146+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":581,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168369+0000","last_change":"2024-04-25T05:44:16.995982+0000","last_active":"2024-04-25T05:44:32.168369+0000","last_peered":"2024-04-25T05:44:32.168369+0000","last_clean":"2024-04-25T05:44:32.168369+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:44:32.168369+0000","last_undegraded":"2024-04-25T05:44:32.168369+0000","last_fullsized":"2024-04-25T05:44:32.168369+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:38.671823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":529,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.170774+0000","last_change":"2024-04-25T05:43:18.685398+0000","last_active":"2024-04-25T05:44:32.170774+0000","last_peered":"2024-04-25T05:44:32.170774+0000","last_clean":"2024-04-25T05:44:32.170774+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:44:32.170774+0000","last_undegraded":"2024-04-25T05:44:32.170774+0000","last_fullsized":"2024-04-25T05:44:32.170774+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.685299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.685299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":554,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.168731+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:44:32.168731+0000","last_peered":"2024-04-25T05:44:32.168731+0000","last_clean":"2024-04-25T05:44:32.168731+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:44:32.168731+0000","last_undegraded":"2024-04-25T05:44:32.168731+0000","last_fullsized":"2024-04-25T05:44:32.168731+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":644,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.151072+0000","last_change":"2024-04-25T05:44:02.120412+0000","last_active":"2024-04-25T05:44:32.151072+0000","last_peered":"2024-04-25T05:44:32.151072+0000","last_clean":"2024-04-25T05:44:32.151072+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:44:32.151072+0000","last_undegraded":"2024-04-25T05:44:32.151072+0000","last_fullsized":"2024-04-25T05:44:32.151072+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:44:02.120258+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.120258+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.452057+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":181,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.172662+0000","last_change":"2024-04-25T05:44:28.182297+0000","last_active":"2024-04-25T05:44:32.172662+0000","last_peered":"2024-04-25T05:44:32.172662+0000","last_clean":"2024-04-25T05:44:32.172662+0000","last_became_active":"2024-04-25T05:44:28.181951+0000","last_became_peered":"2024-04-25T05:44:28.181951+0000","last_unstale":"2024-04-25T05:44:32.172662+0000","last_undegraded":"2024-04-25T05:44:32.172662+0000","last_fullsized":"2024-04-25T05:44:32.172662+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:57.544063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.544063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":568,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.162491+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:44:32.162491+0000","last_peered":"2024-04-25T05:44:32.162491+0000","last_clean":"2024-04-25T05:44:32.162491+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:44:32.162491+0000","last_undegraded":"2024-04-25T05:44:32.162491+0000","last_fullsized":"2024-04-25T05:44:32.162491+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.182548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":68,"reported_epoch":329,"state":"active+clean","last_fresh":"2024-04-25T05:44:32.167838+0000","last_change":"2024-04-25T05:43:16.198408+0000","last_active":"2024-04-25T05:44:32.167838+0000","last_peered":"2024-04-25T05:44:32.167838+0000","last_clean":"2024-04-25T05:44:32.167838+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:44:32.167838+0000","last_undegraded":"2024-04-25T05:44:32.167838+0000","last_fullsized":"2024-04-25T05:44:32.167838+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7451967488,"num_objects":113708,"num_object_clones":0,"num_object_copies":341124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":33040,"num_objects_unfound":0,"num_objects_dirty":113708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113708,"num_write_kb":7277312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9034,"num_bytes_recovered":592052224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22977118208,"data_stored":22977118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50252,"ondisk_log_size":50252,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":621,"num_read_kb":616,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":8366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":14104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070597,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5505504,"kb_used_data":5032856,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":88239648,"statfs":{"total":95995035648,"available":90357399552,"internally_reserved":0,"allocated":5153644544,"data_stored":5153214214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364937,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2479256,"kb_used_data":2055184,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91265896,"statfs":{"total":95995035648,"available":93456277504,"internally_reserved":0,"allocated":2104508416,"data_stored":2104086036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6111620,"kb_used_data":5520956,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":87633532,"statfs":{"total":95995035648,"available":89736736768,"internally_reserved":0,"allocated":5653458944,"data_stored":5652991054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411485,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5405944,"kb_used_data":4765296,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":88339208,"statfs":{"total":95995035648,"available":90459348992,"internally_reserved":0,"allocated":4879663104,"data_stored":4879208320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":656013767},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986322,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2659304,"kb_used_data":2258848,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91085848,"statfs":{"total":95995035648,"available":93271908352,"internally_reserved":0,"allocated":2313060352,"data_stored":2312621815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2786197504,"data_stored":2786197504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5661196288,"data_stored":5661196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4903272448,"data_stored":4903272448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2324103168,"data_stored":2324103168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2113732608,"data_stored":2113732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5188616192,"data_stored":5188616192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:44:44.143 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-04-25T05:44:44.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:44.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:44:44.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 120090 120074 84.3031 69.6875 0.0107838 0.011856 2024-04-25T05:44:44.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:44.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[27010]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:44.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1091410150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[37492]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1091410150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:44 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:44.467 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:44 smithi060 ceph-mon[30898]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:44:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:44 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1091410150' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:44 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:44:44.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.602 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:44.703 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.738 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:44.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:44.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:44.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:44.874 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:44:44.874 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:44:44.874 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:44:44.874 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:44:44.874 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:44:44.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:45.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:45.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.143 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:45.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0089 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 121311 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 84.2354 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 30.2704 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 186.875 2024-04-25T05:44:45.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 0 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1347 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 484.342 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 2990 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 0 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0118674 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0277132 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 2.14494 2024-04-25T05:44:45.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00146388 2024-04-25T05:44:45.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-04-25T05:44:45.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:45.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:45.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.417 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:45.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: pgmap v560: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: osdmap e330: 8 total, 8 up, 5 in 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:45 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:45.518 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:45.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: pgmap v560: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: osdmap e330: 8 total, 8 up, 5 in 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: pgmap v560: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: osdmap e330: 8 total, 8 up, 5 in 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:45 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:45.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:45.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:45.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:45.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:45.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:45.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:45.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:46.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:46.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:44:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:46.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:46.214 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:46.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:46.241 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:46.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:46 smithi043 ceph-mon[27010]: osdmap e331: 8 total, 8 up, 5 in 2024-04-25T05:44:46.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:46 smithi043 ceph-mon[37492]: osdmap e331: 8 total, 8 up, 5 in 2024-04-25T05:44:46.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:46.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:46.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:46.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:46.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:46 smithi060 ceph-mon[30898]: osdmap e331: 8 total, 8 up, 5 in 2024-04-25T05:44:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:46.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:46.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:46.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:46.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:46.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:46.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:46.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:46.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:47.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:47.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:47.148 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:47.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:47.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:47.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.418 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:47.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[27010]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[27010]: osdmap e332: 8 total, 8 up, 5 in 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[37492]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[37492]: osdmap e332: 8 total, 8 up, 5 in 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:44:47.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:47 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:44:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:47 smithi060 ceph-mon[30898]: pgmap v563: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s; 29895/357948 objects misplaced (8.352%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:47 smithi060 ceph-mon[30898]: osdmap e332: 8 total, 8 up, 5 in 2024-04-25T05:44:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:47 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:44:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:47 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:44:47.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:47.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:47.550 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:44:47.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:47.601 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:44:47.644 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:44:47.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:47.691 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:44:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:47.734 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:44:47.774 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:44:47.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:47.813 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:44:47.826 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:47.852 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:44:47.860 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:44:47.861 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:44:47.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:47.954 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:47.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:48.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:48.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:48.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:48.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:48.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:48.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:48.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:48 smithi060 ceph-mon[30898]: osdmap e333: 8 total, 8 up, 5 in 2024-04-25T05:44:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/550680698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:48.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:48.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[27010]: osdmap e333: 8 total, 8 up, 5 in 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/550680698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[37492]: osdmap e333: 8 total, 8 up, 5 in 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/550680698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:44:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:44:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.644 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:48.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:48.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:48.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:48.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:44:48.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:48.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:48.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:49.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:49.123 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:49.123 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:49.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:49.144 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:44:49.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:44:49.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:44:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:49.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:49.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:49.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:49.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:49 smithi043 ceph-mon[27010]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 29895/357948 objects misplaced (8.352%) 2024-04-25T05:44:49.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:49 smithi043 ceph-mon[37492]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 29895/357948 objects misplaced (8.352%) 2024-04-25T05:44:49.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.491 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:49.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:49.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:49 smithi060 ceph-mon[30898]: pgmap v566: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 29895/357948 objects misplaced (8.352%) 2024-04-25T05:44:49.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:49.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:49.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:49.764 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:49.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:49.884 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:44:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:49.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:49.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:44:49.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.027 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:50.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:50.070 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:50.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:50.139 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:44:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.162 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:50.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:50.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:50.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:44:50.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:50.443 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:50.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:50.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:50.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.704 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:50.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:50.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:44:50.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.828 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:50.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:50.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:50.965 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:50.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:51.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:50 smithi043 ceph-mon[27010]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 912 op/s; 24432/355749 objects misplaced (6.868%); 58 MiB/s, 932 objects/s recovering 2024-04-25T05:44:51.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:50 smithi043 ceph-mon[27010]: osdmap e334: 8 total, 8 up, 5 in 2024-04-25T05:44:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:50 smithi043 ceph-mon[37492]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 912 op/s; 24432/355749 objects misplaced (6.868%); 58 MiB/s, 932 objects/s recovering 2024-04-25T05:44:51.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:50 smithi043 ceph-mon[37492]: osdmap e334: 8 total, 8 up, 5 in 2024-04-25T05:44:51.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:51.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:50 smithi060 ceph-mon[30898]: pgmap v567: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 24 MiB/s wr, 912 op/s; 24432/355749 objects misplaced (6.868%); 58 MiB/s, 932 objects/s recovering 2024-04-25T05:44:51.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:50 smithi060 ceph-mon[30898]: osdmap e334: 8 total, 8 up, 5 in 2024-04-25T05:44:51.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:51.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:51.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:51.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:51.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:51.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:51.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:51.674 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:51.675 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:51.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:51.798 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:51.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:51.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:51.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:52.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:52.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:52.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:52.188 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:52.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.326 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:52.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:52.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:52 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:52.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:52 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:52.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:52 smithi060 ceph-mon[30898]: osdmap e335: 8 total, 8 up, 5 in 2024-04-25T05:44:52.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:52.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:44:52.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:52.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:52.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[27010]: osdmap e335: 8 total, 8 up, 5 in 2024-04-25T05:44:52.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:52.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:52.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:52 smithi043 ceph-mon[37492]: osdmap e335: 8 total, 8 up, 5 in 2024-04-25T05:44:52.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:52.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:52.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:52.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:52.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:52.877 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:52.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:52.882 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:44:52.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.014 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:53.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:53.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:53.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:53.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:53.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:53.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:53.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:53.517 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:53 smithi060 ceph-mon[30898]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 918 op/s; 24432/355749 objects misplaced (6.868%); 59 MiB/s, 938 objects/s recovering 2024-04-25T05:44:53.517 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:44:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:44:53.525 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:53.525 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:53.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:53.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:53 smithi043 ceph-mon[27010]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 918 op/s; 24432/355749 objects misplaced (6.868%); 59 MiB/s, 938 objects/s recovering 2024-04-25T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:53 smithi043 ceph-mon[37492]: pgmap v570: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 25 MiB/s wr, 918 op/s; 24432/355749 objects misplaced (6.868%); 59 MiB/s, 938 objects/s recovering 2024-04-25T05:44:53.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:53.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:53.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:53.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:53.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:53.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:53.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:54.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:54.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:54.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:44:54.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:54.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:54.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/270489365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:54 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.508 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:54.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:54.584 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:44:54.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.638 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:54.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/270489365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:44:54.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/270489365' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:54 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:44:54.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:54.766 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:54.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:54.893 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:44:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:54.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:54.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:55.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:55.033 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:55.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:55.174 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:55.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:55.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:55.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:55.439 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:55.481 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:55.481 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:55.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:55.571 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[27010]: pgmap v571: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 21 MiB/s wr, 1.66k op/s; 20737/338169 objects misplaced (6.132%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[27010]: osdmap e336: 8 total, 8 up, 5 in 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[27010]: osdmap e337: 8 total, 8 up, 5 in 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[37492]: pgmap v571: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 21 MiB/s wr, 1.66k op/s; 20737/338169 objects misplaced (6.132%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[37492]: osdmap e336: 8 total, 8 up, 5 in 2024-04-25T05:44:55.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:55 smithi043 ceph-mon[37492]: osdmap e337: 8 total, 8 up, 5 in 2024-04-25T05:44:55.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:55 smithi060 ceph-mon[30898]: pgmap v571: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.7 KiB/s rd, 21 MiB/s wr, 1.66k op/s; 20737/338169 objects misplaced (6.132%); 73 MiB/s, 1.17k objects/s recovering 2024-04-25T05:44:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:55 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:44:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:44:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:55 smithi060 ceph-mon[30898]: osdmap e336: 8 total, 8 up, 5 in 2024-04-25T05:44:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:55 smithi060 ceph-mon[30898]: osdmap e337: 8 total, 8 up, 5 in 2024-04-25T05:44:55.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:55.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:55.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:55.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:55.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:56.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:56.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:56.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:44:56.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:56.248 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:56.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:56.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:56.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:56.486 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:44:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:56.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:56.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:56.622 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:56 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:44:56.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:56 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:44:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:56 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:44:56.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:56 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:44:56.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:56 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:44:56.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:56 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:44:56.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:56.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:56.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:56.909 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:56.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:57.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:57.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:57.126 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:44:57.126 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:44:57.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:57.167 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:57.269 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:57.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:57.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:57.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:57.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:57.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:57.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:57 smithi043 ceph-mon[27010]: pgmap v574: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 857 B/s rd, 0 B/s wr, 981 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 419 objects/s recovering 2024-04-25T05:44:57.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:57 smithi043 ceph-mon[27010]: osdmap e338: 8 total, 8 up, 5 in 2024-04-25T05:44:57.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:57 smithi043 ceph-mon[37492]: pgmap v574: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 857 B/s rd, 0 B/s wr, 981 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 419 objects/s recovering 2024-04-25T05:44:57.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:57 smithi043 ceph-mon[37492]: osdmap e338: 8 total, 8 up, 5 in 2024-04-25T05:44:57.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:57.691 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:57 smithi060 ceph-mon[30898]: pgmap v574: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 857 B/s rd, 0 B/s wr, 981 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 419 objects/s recovering 2024-04-25T05:44:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:57 smithi060 ceph-mon[30898]: osdmap e338: 8 total, 8 up, 5 in 2024-04-25T05:44:57.793 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:57.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:44:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.827 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:57.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:57.960 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:57.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:58.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:44:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.100 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:58.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:44:58.169 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:44:58.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:44:58.231 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:44:58.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:58.362 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:58.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:58.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:44:58.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.636 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:58.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:58 smithi043 ceph-mon[27010]: 5.f deep-scrub starts 2024-04-25T05:44:58.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:58 smithi043 ceph-mon[27010]: 5.f deep-scrub ok 2024-04-25T05:44:58.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:58 smithi043 ceph-mon[37492]: 5.f deep-scrub starts 2024-04-25T05:44:58.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:58 smithi043 ceph-mon[37492]: 5.f deep-scrub ok 2024-04-25T05:44:58.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:58.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:58 smithi060 ceph-mon[30898]: 5.f deep-scrub starts 2024-04-25T05:44:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:58 smithi060 ceph-mon[30898]: 5.f deep-scrub ok 2024-04-25T05:44:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:58.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:58.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:44:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:58.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:58.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:44:59.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:59.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:59.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:44:59.170 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:44:59.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:59.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:59.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:44:59 smithi060 ceph-mon[30898]: pgmap v576: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 977 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 417 objects/s recovering 2024-04-25T05:44:59.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:44:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:44:59.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:44:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:44:59 smithi043 ceph-mon[37492]: pgmap v576: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 977 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 417 objects/s recovering 2024-04-25T05:44:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:44:59 smithi043 ceph-mon[27010]: pgmap v576: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 977 op/s; 20737/338169 objects misplaced (6.132%); 26 MiB/s, 417 objects/s recovering 2024-04-25T05:44:59.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:44:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.696 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:44:59.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:44:59.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:44:59.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:44:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:44:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:44:59.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:00.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:00.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:00.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:44:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:44:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:00.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:00.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:00.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:00.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:00.461 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:00.461 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:00.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.507 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:00.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:00.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:00.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:00.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:00.781 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:00.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:00.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:00.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:01.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:01.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:01.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:01.189 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[37492]: pgmap v577: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 17466/322173 objects misplaced (5.421%); 24 MiB/s, 387 objects/s recovering 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[37492]: osdmap e339: 8 total, 8 up, 5 in 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[27010]: pgmap v577: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 17466/322173 objects misplaced (5.421%); 24 MiB/s, 387 objects/s recovering 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[27010]: osdmap e339: 8 total, 8 up, 5 in 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:45:01.224 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:00 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:45:01.240 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:01.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:00 smithi060 ceph-mon[30898]: pgmap v577: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 17466/322173 objects misplaced (5.421%); 24 MiB/s, 387 objects/s recovering 2024-04-25T05:45:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:00 smithi060 ceph-mon[30898]: osdmap e339: 8 total, 8 up, 5 in 2024-04-25T05:45:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:00 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:45:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:00 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:45:01.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:01.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:45:01.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:01.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:01.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:01.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:01.657 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:01.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:01.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:01.748 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:45:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:01.758 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:45:01.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:01.879 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:01.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:01.979 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:01.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:02.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:02.022 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:02.063 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:02.088 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2268343585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:02.106 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:02.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2268343585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:02.107 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2268343585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:02.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:02.151 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:02.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:02.203 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:02.242 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:02.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123943 8.4 8:2708029d:::benchmark_data_smithi043_104001_object123942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 247.379665467, 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.128519394, 2024-04-25T05:45:02.275 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123943 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891965+0000", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.891967+0000", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5379999999999999e-06 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892016+0000", 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9607000000000001e-05 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892052+0000", 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5642000000000002e-05 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892061+0000", 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4720000000000001e-06 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892114+0000", 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3022999999999997e-05 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892283+0000", 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000168732 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.892847+0000", 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056390200000000002 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897475+0000", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046274669999999997 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.901536+0000", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040612139999999996 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917500+0000", 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015963893999999999 2024-04-25T05:45:02.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917511+0000", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0796e-05 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901955+0000", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98444451099999997 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901968+0000", 2024-04-25T05:45:02.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2508e-05 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020400+0000", 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.118432597 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020485+0000", 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4491000000000001e-05 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123953 8.4 8:27465e96:::benchmark_data_smithi043_104001_object123952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 247.37492817200001, 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1241087030000001, 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123953 2024-04-25T05:45:02.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896703+0000", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896704+0000", 2024-04-25T05:45:02.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3349999999999999e-06 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896735+0000", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1065000000000002e-05 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896745+0000", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0189e-05 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.896752+0000", 2024-04-25T05:45:02.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7800000000000003e-06 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897517+0000", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076539399999999997 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897672+0000", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015488200000000001 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.897862+0000", 2024-04-25T05:45:02.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018961499999999999 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.902561+0000", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046995270000000002 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917555+0000", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014993536999999999 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.917566+0000", 2024-04-25T05:45:02.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1559999999999999e-05 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.902011+0000", 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98444420700000002 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.902021+0000", 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.049e-05 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020751+0000", 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1187298779999999 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.020811+0000", 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0244000000000001e-05 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123959 8.d 8:b0698cdf:::benchmark_data_smithi043_104001_object123958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 247.343463568, 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.138867517, 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.290 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123959 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928167+0000", 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928169+0000", 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.406e-06 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928202+0000", 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3572000000000003e-05 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928214+0000", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2097e-05 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928220+0000", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8819999999999998e-06 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928280+0000", 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0406999999999999e-05 2024-04-25T05:45:02.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.928293+0000", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.27e-05 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901652+0000", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.973358995 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901666+0000", 2024-04-25T05:45:02.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.401e-05 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048106+0000", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1464394680000001 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048253+0000", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147081 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.048739+0000", 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000486613 2024-04-25T05:45:02.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057200+0000", 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0084604820000000001 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066420+0000", 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0092204339999999996 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066598+0000", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017789300000000001 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066859+0000", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026048400000000001 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066917+0000", 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8510000000000001e-05 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067035+0000", 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117483 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:123966 8.d 8:b64b38a9:::benchmark_data_smithi043_104001_object123965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 247.32210341199999, 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.136094049, 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.298 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 123966 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949527+0000", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949529+0000", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5680000000000001e-06 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949599+0000", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0146000000000002e-05 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949616+0000", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7413000000000001e-05 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949627+0000", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0267e-05 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949695+0000", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8750000000000004e-05 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.301 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:54.949706+0000", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0223e-05 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901696+0000", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95199062999999995 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:55.901704+0000", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2169999999999994e-06 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057238+0000", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.155533602 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.057382+0000", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014436399999999999 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.058010+0000", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00062746700000000004 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.066646+0000", 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086361749999999994 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067082+0000", 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043566699999999999 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067168+0000", 2024-04-25T05:45:02.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6076999999999996e-05 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085439+0000", 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.018271005 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085502+0000", 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2967999999999997e-05 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085621+0000", 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011951 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124002 8.d 8:b79b1d51:::benchmark_data_smithi043_104001_object124001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:45:02.306 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 246.27045145400001, 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0846060230000001, 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124002 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001179+0000", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001180+0000", 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6899999999999996e-07 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001301+0000", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120989 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001312+0000", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.063e-05 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001317+0000", 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6219999999999997e-06 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001365+0000", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7193e-05 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:56.001374+0000", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0880000000000001e-06 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.061525+0000", 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0601515500000001 2024-04-25T05:45:02.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.061653+0000", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012791500000000001 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.062074+0000", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00042062799999999998 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.067121+0000", 2024-04-25T05:45:02.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050470749999999998 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.073295+0000", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0061738749999999997 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.080544+0000", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072492839999999999 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085633+0000", 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050887900000000002 2024-04-25T05:45:02.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085681+0000", 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7899000000000003e-05 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.085785+0000", 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000104516 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.313 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124101 8.8 8:1d082edc:::benchmark_data_smithi043_104001_object124100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 245.16590652299999, 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.272319637, 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124101 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105724+0000", 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105726+0000", 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6500000000000001e-06 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105891+0000", 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000164845 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105903+0000", 2024-04-25T05:45:02.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2672e-05 2024-04-25T05:45:02.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105911+0000", 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9670000000000001e-06 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.105963+0000", 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1233999999999998e-05 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.106112+0000", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014978600000000001 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.106278+0000", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016582200000000001 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.109610+0000", 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033317619999999998 2024-04-25T05:45:02.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.109648+0000", 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8569000000000003e-05 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.377924+0000", 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26827566400000002 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.377996+0000", 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2193000000000005e-05 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.378044+0000", 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7472999999999998e-05 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.321 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127833 8.d 8:b40552da:::benchmark_data_smithi043_104001_object3709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 239.44738682400001, 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15866919099999999, 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127833 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.322 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824244+0000", 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824246+0000", 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7579999999999999e-06 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824252+0000", 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0270000000000001e-06 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824263+0000", 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1644999999999999e-05 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824271+0000", 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9889999999999992e-06 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828470+0000", 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041991429999999998 2024-04-25T05:45:02.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828597+0000", 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126799 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.828726+0000", 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012869300000000001 2024-04-25T05:45:02.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842094+0000", 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013367591999999999 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842428+0000", 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033413200000000001 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.845112+0000", 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002684559 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982745+0000", 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13763259899999999 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982785+0000", 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0163000000000002e-05 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.982913+0000", 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012809200000000001 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:127834 8.d 8:b2f67481:::benchmark_data_smithi043_104001_object3710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 239.447119359, 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.164348995, 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.330 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 127834 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824511+0000", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824512+0000", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4199999999999995e-07 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824520+0000", 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.644e-06 2024-04-25T05:45:02.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824529+0000", 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2590000000000006e-06 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.824537+0000", 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6249999999999998e-06 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831153+0000", 2024-04-25T05:45:02.334 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:02.334 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:02.334 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:02.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066163419999999999 2024-04-25T05:45:02.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831291+0000", 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013770899999999999 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6,7", 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.831427+0000", 2024-04-25T05:45:02.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013640500000000001 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842101+0000", 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010673741 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.842497+0000", 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039574500000000003 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.849394+0000", 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0068978620000000003 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988675+0000", 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13928105199999999 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988738+0000", 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2385999999999995e-05 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:02.988860+0000", 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000122483 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226334 8.8 8:1120fdcc:::benchmark_data_smithi043_104001_object102210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.408201263, 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.29533898800000002, 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.343 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226334 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863431+0000", 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4980000000000001e-06 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863437+0000", 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8420000000000001e-06 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863450+0000", 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3424999999999999e-05 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863459+0000", 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2129999999999997e-06 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863579+0000", 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119181 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863691+0000", 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011263400000000001 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863798+0000", 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010700300000000001 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.867039+0000", 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032403499999999999 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.867308+0000", 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026967000000000001 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.348 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158726+0000", 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.29141812900000003 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158768+0000", 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2042999999999997e-05 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226340 8.8 8:1cf7c54f:::benchmark_data_smithi043_104001_object102216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:45:02.349 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.40047969099999, 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.287807596, 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226340 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.350 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:45:02.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871152+0000", 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3260000000000001e-06 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871159+0000", 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9240000000000001e-06 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871168+0000", 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0510000000000002e-06 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871175+0000", 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3489999999999997e-06 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872018+0000", 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084366499999999995 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872171+0000", 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015301600000000001 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872311+0000", 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013923900000000001 2024-04-25T05:45:02.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.876118+0000", 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038069760000000001 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881212+0000", 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050944299999999996 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158899+0000", 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27768725900000002 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158959+0000", 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9361000000000003e-05 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226345 8.4 8:291622b3:::benchmark_data_smithi043_104001_object102221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.39364820399999, 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27591954099999999, 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226345 2024-04-25T05:45:02.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877983+0000", 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.13e-07 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877986+0000", 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4690000000000002e-06 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877994+0000", 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4699999999999996e-06 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878001+0000", 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6719999999999998e-06 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878398+0000", 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039738700000000002 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878515+0000", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117169 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878633+0000", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117488 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881439+0000", 2024-04-25T05:45:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002806092 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.885451+0000", 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040120099999999999 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.153884+0000", 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26843287300000002 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.153902+0000", 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8397999999999999e-05 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226346 8.8 8:1dfd0933:::benchmark_data_smithi043_104001_object102222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.39361232799999, 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.28112287200000002, 2024-04-25T05:45:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226346 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878019+0000", 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6600000000000002e-07 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878022+0000", 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0680000000000001e-06 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878028+0000", 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6960000000000002e-06 2024-04-25T05:45:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878033+0000", 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2190000000000003e-06 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878127+0000", 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4475000000000006e-05 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878237+0000", 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109503 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878344+0000", 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107265 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881455+0000", 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031110550000000002 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.882740+0000", 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001284997 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159099+0000", 2024-04-25T05:45:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27635901899999998 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159141+0000", 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2109e-05 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226334 8.8 8:1120fdcc:::benchmark_data_smithi043_104001_object102210:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.37781766699999, 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1173450709999999, 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226334 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893815+0000", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8640000000000001e-06 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893824+0000", 2024-04-25T05:45:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1400000000000006e-06 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893845+0000", 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0873000000000002e-05 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893856+0000", 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0793e-05 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897028+0000", 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003172663 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897034+0000", 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0299999999999999e-06 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159254+0000", 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.262219751 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159279+0000", 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4542e-05 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011075+0000", 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85179589600000005 2024-04-25T05:45:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011158+0000", 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3518999999999998e-05 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226340 8.18 8:1cf7c54f:::benchmark_data_smithi043_104001_object102216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.37774899799999, 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1499044039999999, 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226340 2024-04-25T05:45:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2399999999999999e-07 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893885+0000", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.83e-06 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893899+0000", 2024-04-25T05:45:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4236000000000001e-05 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893905+0000", 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.139e-06 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158402+0000", 2024-04-25T05:45:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26449650899999999 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158430+0000", 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8666999999999998e-05 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.039590+0000", 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88115992700000001 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.039724+0000", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013313499999999999 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.043720+0000", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039966180000000004 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.043786+0000", 2024-04-25T05:45:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5919000000000004e-05 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226345 8.14 8:291622b3:::benchmark_data_smithi043_104001_object102221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.37459572399999, 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1601359410000001, 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226345 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897036+0000", 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.393e-06 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897044+0000", 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5669999999999997e-06 2024-04-25T05:45:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897056+0000", 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1652e-05 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897064+0000", 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.156521+0000", 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.25945676899999998 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.156532+0000", 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1039999999999999e-05 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051043+0000", 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.894510994 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051153+0000", 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109852 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.057096+0000", 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059431110000000001 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.057171+0000", 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4857999999999994e-05 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226346 8.18 8:1dfd0933:::benchmark_data_smithi043_104001_object102222:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.374542424, 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.147144165, 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226346 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897089+0000", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3000000000000002e-07 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897091+0000", 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3410000000000001e-06 2024-04-25T05:45:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897095+0000", 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0110000000000002e-06 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897100+0000", 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7990000000000001e-06 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158558+0000", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26145807100000001 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158581+0000", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3422e-05 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.040060+0000", 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88147867000000002 2024-04-25T05:45:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.040209+0000", 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000148942 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.044181+0000", 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003972382 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.044232+0000", 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1196999999999998e-05 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226358 8.4 8:2507d7e8:::benchmark_data_smithi043_104001_object102234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.08880406500001, 2024-04-25T05:45:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87045504500000004, 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226358 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182828+0000", 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6029999999999999e-06 2024-04-25T05:45:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182834+0000", 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0700000000000003e-06 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183038+0000", 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020412100000000001 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183054+0000", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5245999999999999e-05 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183127+0000", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3009000000000005e-05 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183135+0000", 2024-04-25T05:45:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5830000000000007e-06 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020562+0000", 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.83742649899999999 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020696+0000", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013470499999999999 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.021014+0000", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031797900000000003 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051256+0000", 2024-04-25T05:45:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.030241160999999999 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051290+0000", 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4397000000000003e-05 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053198+0000", 2024-04-25T05:45:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019084589999999999 2024-04-25T05:45:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053241+0000", 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2534000000000003e-05 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053282+0000", 2024-04-25T05:45:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0679000000000001e-05 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228015 8.11 8:8aa3395b:::benchmark_data_smithi043_104001_object103891:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 140.29420929200001, 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.126170712, 2024-04-25T05:45:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228015 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977424+0000", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1619999999999998e-06 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977433+0000", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9469999999999992e-06 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977444+0000", 2024-04-25T05:45:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1599e-05 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977454+0000", 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7939999999999997e-06 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.985631+0000", 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0081774259999999998 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.985640+0000", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1710000000000008e-06 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981642+0000", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99600169299999997 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981656+0000", 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3356999999999999e-05 2024-04-25T05:45:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103494+0000", 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.121838624 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103592+0000", 2024-04-25T05:45:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7938999999999999e-05 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228038 8.11 8:8a1cbb27:::benchmark_data_smithi043_104001_object103914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 140.28062572499999, 2024-04-25T05:45:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1240071120000001, 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228038 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:45:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991006+0000", 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9700000000000005e-07 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991010+0000", 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4769999999999997e-06 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991023+0000", 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2221999999999999e-05 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991028+0000", 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1529999999999996e-06 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991086+0000", 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8495999999999997e-05 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991096+0000", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0159e-05 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981686+0000", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99058917800000001 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981695+0000", 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1989999999999995e-06 2024-04-25T05:45:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104063+0000", 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1223681910000001 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104174+0000", 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110653 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,6,7", 2024-04-25T05:45:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104398+0000", 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000224319 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.106906+0000", 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025081190000000001 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.107279+0000", 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000372755 2024-04-25T05:45:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.110458+0000", 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031791749999999998 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.110501+0000", 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2617000000000002e-05 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.114824+0000", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043234880000000003 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.114882+0000", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7948000000000002e-05 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.115012+0000", 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000130066 2024-04-25T05:45:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228061 8.19 8:9a04b146:::benchmark_data_smithi043_104001_object103937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 140.25480412600001, 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.087389859, 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:45:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228061 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016829+0000", 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9870000000000002e-06 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016838+0000", 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2299999999999997e-06 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016865+0000", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7203e-05 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016887+0000", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2500000000000001e-05 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.017005+0000", 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011742499999999999 2024-04-25T05:45:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.017012+0000", 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0500000000000003e-06 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981886+0000", 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96487387700000005 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981897+0000", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1289e-05 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.074722+0000", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.092824963 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.074853+0000", 2024-04-25T05:45:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000130925 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,4,6", 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.075106+0000", 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025335100000000002 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103590+0000", 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.028484112999999998 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103829+0000", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023812599999999999 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103869+0000", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0166000000000001e-05 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103969+0000", 2024-04-25T05:45:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010041500000000001 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104099+0000", 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012951 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104216+0000", 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117729 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:02.434 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:45:02.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:02.468 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:02.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:02.597 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:02.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:45:02.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:02.738 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:02.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:02.867 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:02.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:02.869 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2024-04-25T05:44:59.914571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7023434291,"num_objects":107391,"num_object_clones":0,"num_object_copies":322173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17466,"num_objects_unfound":0,"num_objects_dirty":107391,"num_whiteouts":0,"num_read":2056,"num_read_kb":1776,"num_write":136377,"num_write_kb":7763950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24287,"num_bytes_recovered":1559637794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53078,"ondisk_log_size":53078,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":22348884,"kb_used_data":19581596,"kb_used_omap":7,"kb_used_meta":2767224,"kb_avail":446376876,"statfs":{"total":479975178240,"available":457089921024,"internally_reserved":0,"allocated":20051554304,"data_stored":20049093902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2833637601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-349437952,"num_objects":-5332,"num_object_clones":0,"num_object_copies":-15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3271,"num_objects_unfound":0,"num_objects_dirty":-5332,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5332,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2328,"num_bytes_recovered":152567808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.650548"},"pg_stats":[{"pgid":"9.1","version":"338'8438","reported_seq":8531,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.628544+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:44:58.628544+0000","last_peered":"2024-04-25T05:44:58.628544+0000","last_clean":"2024-04-25T05:44:58.628544+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:44:58.628544+0000","last_undegraded":"2024-04-25T05:44:58.628544+0000","last_fullsized":"2024-04-25T05:44:58.628544+0000","mapping_epoch":296,"log_start":"321'5287","ondisk_log_start":"321'5287","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3151,"log_dups_size":0,"ondisk_log_size":3151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439746560,"num_objects":6710,"num_object_clones":0,"num_object_copies":20130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":300,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403890+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:44:56.403890+0000","last_peered":"2024-04-25T05:44:56.403890+0000","last_clean":"2024-04-25T05:44:56.403890+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:44:56.403890+0000","last_undegraded":"2024-04-25T05:44:56.403890+0000","last_fullsized":"2024-04-25T05:44:56.403890+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":590,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.404990+0000","last_change":"2024-04-25T05:44:28.173941+0000","last_active":"2024-04-25T05:44:56.404990+0000","last_peered":"2024-04-25T05:44:56.404990+0000","last_clean":"2024-04-25T05:44:56.404990+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:44:56.404990+0000","last_undegraded":"2024-04-25T05:44:56.404990+0000","last_fullsized":"2024-04-25T05:44:56.404990+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"325'12","reported_seq":385,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284039+0000","last_change":"2024-04-25T05:44:28.198744+0000","last_active":"2024-04-25T05:44:56.284039+0000","last_peered":"2024-04-25T05:44:56.284039+0000","last_clean":"2024-04-25T05:44:56.284039+0000","last_became_active":"2024-04-25T05:44:28.171810+0000","last_became_peered":"2024-04-25T05:44:28.171810+0000","last_unstale":"2024-04-25T05:44:56.284039+0000","last_undegraded":"2024-04-25T05:44:56.284039+0000","last_fullsized":"2024-04-25T05:44:56.284039+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":263,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405472+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:44:56.405472+0000","last_peered":"2024-04-25T05:44:56.405472+0000","last_clean":"2024-04-25T05:44:56.405472+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:44:56.405472+0000","last_undegraded":"2024-04-25T05:44:56.405472+0000","last_fullsized":"2024-04-25T05:44:56.405472+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":896,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285394+0000","last_change":"2024-04-25T05:43:19.210320+0000","last_active":"2024-04-25T05:44:56.285394+0000","last_peered":"2024-04-25T05:44:56.285394+0000","last_clean":"2024-04-25T05:44:56.285394+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:44:56.285394+0000","last_undegraded":"2024-04-25T05:44:56.285394+0000","last_fullsized":"2024-04-25T05:44:56.285394+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"338'8489","reported_seq":8586,"reported_epoch":338,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:44:58.716175+0000","last_change":"2024-04-25T05:44:29.211276+0000","last_active":"2024-04-25T05:44:58.716175+0000","last_peered":"2024-04-25T05:44:58.716175+0000","last_clean":"2024-04-25T05:44:27.095992+0000","last_became_active":"2024-04-25T05:44:29.189111+0000","last_became_peered":"2024-04-25T05:44:29.189111+0000","last_unstale":"2024-04-25T05:44:58.716175+0000","last_undegraded":"2024-04-25T05:44:58.716175+0000","last_fullsized":"2024-04-25T05:44:58.716175+0000","mapping_epoch":325,"log_start":"322'5587","ondisk_log_start":"322'5587","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2902,"log_dups_size":2115,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430505984,"num_objects":6569,"num_object_clones":0,"num_object_copies":19707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6569,"num_objects_unfound":0,"num_objects_dirty":6569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":481856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":6569}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":308,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286620+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:44:56.286620+0000","last_peered":"2024-04-25T05:44:56.286620+0000","last_clean":"2024-04-25T05:44:56.286620+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:44:56.286620+0000","last_undegraded":"2024-04-25T05:44:56.286620+0000","last_fullsized":"2024-04-25T05:44:56.286620+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.293115+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:44:56.293115+0000","last_peered":"2024-04-25T05:44:56.293115+0000","last_clean":"2024-04-25T05:44:56.293115+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:44:56.293115+0000","last_undegraded":"2024-04-25T05:44:56.293115+0000","last_fullsized":"2024-04-25T05:44:56.293115+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.541034+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":596,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286806+0000","last_change":"2024-04-25T05:44:28.168158+0000","last_active":"2024-04-25T05:44:56.286806+0000","last_peered":"2024-04-25T05:44:56.286806+0000","last_clean":"2024-04-25T05:44:56.286806+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:44:56.286806+0000","last_undegraded":"2024-04-25T05:44:56.286806+0000","last_fullsized":"2024-04-25T05:44:56.286806+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.404863+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:44:56.404863+0000","last_peered":"2024-04-25T05:44:56.404863+0000","last_clean":"2024-04-25T05:44:56.404863+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:44:56.404863+0000","last_undegraded":"2024-04-25T05:44:56.404863+0000","last_fullsized":"2024-04-25T05:44:56.404863+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.600058+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":496,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288266+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:44:56.288266+0000","last_peered":"2024-04-25T05:44:56.288266+0000","last_clean":"2024-04-25T05:44:56.288266+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:44:56.288266+0000","last_undegraded":"2024-04-25T05:44:56.288266+0000","last_fullsized":"2024-04-25T05:44:56.288266+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"338'8348","reported_seq":8441,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.937162+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:44:57.937162+0000","last_peered":"2024-04-25T05:44:57.937162+0000","last_clean":"2024-04-25T05:44:57.937162+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:44:57.937162+0000","last_undegraded":"2024-04-25T05:44:57.937162+0000","last_fullsized":"2024-04-25T05:44:57.937162+0000","mapping_epoch":296,"log_start":"320'5077","ondisk_log_start":"320'5077","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3271,"log_dups_size":0,"ondisk_log_size":3271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8348,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":609,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.295128+0000","last_change":"2024-04-25T05:42:53.501053+0000","last_active":"2024-04-25T05:44:56.295128+0000","last_peered":"2024-04-25T05:44:56.295128+0000","last_clean":"2024-04-25T05:44:56.295128+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:44:56.295128+0000","last_undegraded":"2024-04-25T05:44:56.295128+0000","last_fullsized":"2024-04-25T05:44:56.295128+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:42:53.500776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.500776+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":603,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287678+0000","last_change":"2024-04-25T05:44:46.503302+0000","last_active":"2024-04-25T05:44:56.287678+0000","last_peered":"2024-04-25T05:44:56.287678+0000","last_clean":"2024-04-25T05:44:56.287678+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:44:56.287678+0000","last_undegraded":"2024-04-25T05:44:56.287678+0000","last_fullsized":"2024-04-25T05:44:56.287678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:46.503173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:46.503173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:56.738128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":398,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.457920+0000","last_change":"2024-04-25T05:44:57.457953+0000","last_active":"2024-04-25T05:44:57.457920+0000","last_peered":"2024-04-25T05:44:57.457920+0000","last_clean":"2024-04-25T05:44:57.457920+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:44:57.457920+0000","last_undegraded":"2024-04-25T05:44:57.457920+0000","last_fullsized":"2024-04-25T05:44:57.457920+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:44:57.457842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:18.204723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":578,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403460+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:44:56.403460+0000","last_peered":"2024-04-25T05:44:56.403460+0000","last_clean":"2024-04-25T05:44:56.403460+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:44:56.403460+0000","last_undegraded":"2024-04-25T05:44:56.403460+0000","last_fullsized":"2024-04-25T05:44:56.403460+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"338'8650","reported_seq":8743,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.648922+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:44:58.648922+0000","last_peered":"2024-04-25T05:44:58.648922+0000","last_clean":"2024-04-25T05:44:58.648922+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:44:58.648922+0000","last_undegraded":"2024-04-25T05:44:58.648922+0000","last_fullsized":"2024-04-25T05:44:58.648922+0000","mapping_epoch":296,"log_start":"321'5481","ondisk_log_start":"321'5481","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3169,"log_dups_size":0,"ondisk_log_size":3169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443547648,"num_objects":6768,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8650,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":626,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289667+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:44:56.289667+0000","last_peered":"2024-04-25T05:44:56.289667+0000","last_clean":"2024-04-25T05:44:56.289667+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:44:56.289667+0000","last_undegraded":"2024-04-25T05:44:56.289667+0000","last_fullsized":"2024-04-25T05:44:56.289667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":581,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288757+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:44:56.288757+0000","last_peered":"2024-04-25T05:44:56.288757+0000","last_clean":"2024-04-25T05:44:56.288757+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:44:56.288757+0000","last_undegraded":"2024-04-25T05:44:56.288757+0000","last_fullsized":"2024-04-25T05:44:56.288757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.125277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"325'13","reported_seq":112,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288802+0000","last_change":"2024-04-25T05:44:28.188020+0000","last_active":"2024-04-25T05:44:56.288802+0000","last_peered":"2024-04-25T05:44:56.288802+0000","last_clean":"2024-04-25T05:44:56.288802+0000","last_became_active":"2024-04-25T05:44:28.178692+0000","last_became_peered":"2024-04-25T05:44:28.178692+0000","last_unstale":"2024-04-25T05:44:56.288802+0000","last_undegraded":"2024-04-25T05:44:56.288802+0000","last_fullsized":"2024-04-25T05:44:56.288802+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":442,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286424+0000","last_change":"2024-04-25T05:44:28.186072+0000","last_active":"2024-04-25T05:44:56.286424+0000","last_peered":"2024-04-25T05:44:56.286424+0000","last_clean":"2024-04-25T05:44:56.286424+0000","last_became_active":"2024-04-25T05:44:28.177729+0000","last_became_peered":"2024-04-25T05:44:28.177729+0000","last_unstale":"2024-04-25T05:44:56.286424+0000","last_undegraded":"2024-04-25T05:44:56.286424+0000","last_fullsized":"2024-04-25T05:44:56.286424+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":572,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.292925+0000","last_change":"2024-04-25T05:44:30.932605+0000","last_active":"2024-04-25T05:44:56.292925+0000","last_peered":"2024-04-25T05:44:56.292925+0000","last_clean":"2024-04-25T05:44:56.292925+0000","last_became_active":"2024-04-25T05:44:30.932221+0000","last_became_peered":"2024-04-25T05:44:30.932221+0000","last_unstale":"2024-04-25T05:44:56.292925+0000","last_undegraded":"2024-04-25T05:44:56.292925+0000","last_fullsized":"2024-04-25T05:44:56.292925+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.344182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"338'8477","reported_seq":8577,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.916685+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:44:57.916685+0000","last_peered":"2024-04-25T05:44:57.916685+0000","last_clean":"2024-04-25T05:44:57.916685+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:44:57.916685+0000","last_undegraded":"2024-04-25T05:44:57.916685+0000","last_fullsized":"2024-04-25T05:44:57.916685+0000","mapping_epoch":301,"log_start":"320'5153","ondisk_log_start":"320'5153","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3324,"log_dups_size":0,"ondisk_log_size":3324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441450520,"num_objects":6737,"num_object_clones":0,"num_object_copies":20211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6737,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8477,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":581,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287579+0000","last_change":"2024-04-25T05:43:58.091032+0000","last_active":"2024-04-25T05:44:56.287579+0000","last_peered":"2024-04-25T05:44:56.287579+0000","last_clean":"2024-04-25T05:44:56.287579+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:44:56.287579+0000","last_undegraded":"2024-04-25T05:44:56.287579+0000","last_fullsized":"2024-04-25T05:44:56.287579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:58.090855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:58.090855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.355724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":735,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288851+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:44:56.288851+0000","last_peered":"2024-04-25T05:44:56.288851+0000","last_clean":"2024-04-25T05:44:56.288851+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:44:56.288851+0000","last_undegraded":"2024-04-25T05:44:56.288851+0000","last_fullsized":"2024-04-25T05:44:56.288851+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.991166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":599,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405132+0000","last_change":"2024-04-25T05:44:28.180828+0000","last_active":"2024-04-25T05:44:56.405132+0000","last_peered":"2024-04-25T05:44:56.405132+0000","last_clean":"2024-04-25T05:44:56.405132+0000","last_became_active":"2024-04-25T05:44:28.178969+0000","last_became_peered":"2024-04-25T05:44:28.178969+0000","last_unstale":"2024-04-25T05:44:56.405132+0000","last_undegraded":"2024-04-25T05:44:56.405132+0000","last_fullsized":"2024-04-25T05:44:56.405132+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:15.041538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.041538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.837433+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":257,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.410316+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:44:56.410316+0000","last_peered":"2024-04-25T05:44:56.410316+0000","last_clean":"2024-04-25T05:44:56.410316+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:44:56.410316+0000","last_undegraded":"2024-04-25T05:44:56.410316+0000","last_fullsized":"2024-04-25T05:44:56.410316+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.232569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":801,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287763+0000","last_change":"2024-04-25T05:44:30.916141+0000","last_active":"2024-04-25T05:44:56.287763+0000","last_peered":"2024-04-25T05:44:56.287763+0000","last_clean":"2024-04-25T05:44:56.287763+0000","last_became_active":"2024-04-25T05:44:30.915718+0000","last_became_peered":"2024-04-25T05:44:30.915718+0000","last_unstale":"2024-04-25T05:44:56.287763+0000","last_undegraded":"2024-04-25T05:44:56.287763+0000","last_fullsized":"2024-04-25T05:44:56.287763+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.223066+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.958082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"338'8589","reported_seq":8682,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.664814+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:44:58.664814+0000","last_peered":"2024-04-25T05:44:58.664814+0000","last_clean":"2024-04-25T05:44:58.664814+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:44:58.664814+0000","last_undegraded":"2024-04-25T05:44:58.664814+0000","last_fullsized":"2024-04-25T05:44:58.664814+0000","mapping_epoch":296,"log_start":"321'5416","ondisk_log_start":"321'5416","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444268544,"num_objects":6779,"num_object_clones":0,"num_object_copies":20337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":705,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.404483+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:44:56.404483+0000","last_peered":"2024-04-25T05:44:56.404483+0000","last_clean":"2024-04-25T05:44:56.404483+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:44:56.404483+0000","last_undegraded":"2024-04-25T05:44:56.404483+0000","last_fullsized":"2024-04-25T05:44:56.404483+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":524,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294166+0000","last_change":"2024-04-25T05:44:28.134148+0000","last_active":"2024-04-25T05:44:56.294166+0000","last_peered":"2024-04-25T05:44:56.294166+0000","last_clean":"2024-04-25T05:44:56.294166+0000","last_became_active":"2024-04-25T05:44:28.124913+0000","last_became_peered":"2024-04-25T05:44:28.124913+0000","last_unstale":"2024-04-25T05:44:56.294166+0000","last_undegraded":"2024-04-25T05:44:56.294166+0000","last_fullsized":"2024-04-25T05:44:56.294166+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:43:24.628430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.628430+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":285,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.406130+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:44:56.406130+0000","last_peered":"2024-04-25T05:44:56.406130+0000","last_clean":"2024-04-25T05:44:56.406130+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:44:56.406130+0000","last_undegraded":"2024-04-25T05:44:56.406130+0000","last_fullsized":"2024-04-25T05:44:56.406130+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":213,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284682+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:44:56.284682+0000","last_peered":"2024-04-25T05:44:56.284682+0000","last_clean":"2024-04-25T05:44:56.284682+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:44:56.284682+0000","last_undegraded":"2024-04-25T05:44:56.284682+0000","last_fullsized":"2024-04-25T05:44:56.284682+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":761,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288839+0000","last_change":"2024-04-25T05:43:19.215636+0000","last_active":"2024-04-25T05:44:56.288839+0000","last_peered":"2024-04-25T05:44:56.288839+0000","last_clean":"2024-04-25T05:44:56.288839+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:44:56.288839+0000","last_undegraded":"2024-04-25T05:44:56.288839+0000","last_fullsized":"2024-04-25T05:44:56.288839+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"338'8490","reported_seq":8574,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.709623+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:44:58.709623+0000","last_peered":"2024-04-25T05:44:58.709623+0000","last_clean":"2024-04-25T05:44:58.709623+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:44:58.709623+0000","last_undegraded":"2024-04-25T05:44:58.709623+0000","last_fullsized":"2024-04-25T05:44:58.709623+0000","mapping_epoch":301,"log_start":"321'5538","ondisk_log_start":"321'5538","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2952,"log_dups_size":2015,"ondisk_log_size":2952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441188352,"num_objects":6732,"num_object_clones":0,"num_object_copies":20196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":296,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294351+0000","last_change":"2024-04-25T05:44:28.202828+0000","last_active":"2024-04-25T05:44:56.294351+0000","last_peered":"2024-04-25T05:44:56.294351+0000","last_clean":"2024-04-25T05:44:56.294351+0000","last_became_active":"2024-04-25T05:44:28.182043+0000","last_became_peered":"2024-04-25T05:44:28.182043+0000","last_unstale":"2024-04-25T05:44:56.294351+0000","last_undegraded":"2024-04-25T05:44:56.294351+0000","last_fullsized":"2024-04-25T05:44:56.294351+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.008065+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":282,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289927+0000","last_change":"2024-04-25T05:44:28.217831+0000","last_active":"2024-04-25T05:44:56.289927+0000","last_peered":"2024-04-25T05:44:56.289927+0000","last_clean":"2024-04-25T05:44:56.289927+0000","last_became_active":"2024-04-25T05:44:28.170458+0000","last_became_peered":"2024-04-25T05:44:28.170458+0000","last_unstale":"2024-04-25T05:44:56.289927+0000","last_undegraded":"2024-04-25T05:44:56.289927+0000","last_fullsized":"2024-04-25T05:44:56.289927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":467,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403524+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:44:56.403524+0000","last_peered":"2024-04-25T05:44:56.403524+0000","last_clean":"2024-04-25T05:44:56.403524+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:44:56.403524+0000","last_undegraded":"2024-04-25T05:44:56.403524+0000","last_fullsized":"2024-04-25T05:44:56.403524+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":199,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288440+0000","last_change":"2024-04-25T05:44:28.168971+0000","last_active":"2024-04-25T05:44:56.288440+0000","last_peered":"2024-04-25T05:44:56.288440+0000","last_clean":"2024-04-25T05:44:56.288440+0000","last_became_active":"2024-04-25T05:44:28.168333+0000","last_became_peered":"2024-04-25T05:44:28.168333+0000","last_unstale":"2024-04-25T05:44:56.288440+0000","last_undegraded":"2024-04-25T05:44:56.288440+0000","last_fullsized":"2024-04-25T05:44:56.288440+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":759,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.295086+0000","last_change":"2024-04-25T05:43:28.638040+0000","last_active":"2024-04-25T05:44:56.295086+0000","last_peered":"2024-04-25T05:44:56.295086+0000","last_clean":"2024-04-25T05:44:56.295086+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:44:56.295086+0000","last_undegraded":"2024-04-25T05:44:56.295086+0000","last_fullsized":"2024-04-25T05:44:56.295086+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:28.637882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:28.637882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"338'8340","reported_seq":8433,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.938487+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:44:57.938487+0000","last_peered":"2024-04-25T05:44:57.938487+0000","last_clean":"2024-04-25T05:44:57.938487+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:44:57.938487+0000","last_undegraded":"2024-04-25T05:44:57.938487+0000","last_fullsized":"2024-04-25T05:44:57.938487+0000","mapping_epoch":296,"log_start":"320'5030","ondisk_log_start":"320'5030","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437387264,"num_objects":6674,"num_object_clones":0,"num_object_copies":20022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.293200+0000","last_change":"2024-04-25T05:44:28.125054+0000","last_active":"2024-04-25T05:44:56.293200+0000","last_peered":"2024-04-25T05:44:56.293200+0000","last_clean":"2024-04-25T05:44:56.293200+0000","last_became_active":"2024-04-25T05:44:28.124674+0000","last_became_peered":"2024-04-25T05:44:28.124674+0000","last_unstale":"2024-04-25T05:44:56.293200+0000","last_undegraded":"2024-04-25T05:44:56.293200+0000","last_fullsized":"2024-04-25T05:44:56.293200+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:58.303045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285212+0000","last_change":"2024-04-25T05:44:28.198134+0000","last_active":"2024-04-25T05:44:56.285212+0000","last_peered":"2024-04-25T05:44:56.285212+0000","last_clean":"2024-04-25T05:44:56.285212+0000","last_became_active":"2024-04-25T05:44:28.168410+0000","last_became_peered":"2024-04-25T05:44:28.168410+0000","last_unstale":"2024-04-25T05:44:56.285212+0000","last_undegraded":"2024-04-25T05:44:56.285212+0000","last_fullsized":"2024-04-25T05:44:56.285212+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":607,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286584+0000","last_change":"2024-04-25T05:44:28.167276+0000","last_active":"2024-04-25T05:44:56.286584+0000","last_peered":"2024-04-25T05:44:56.286584+0000","last_clean":"2024-04-25T05:44:56.286584+0000","last_became_active":"2024-04-25T05:44:28.166968+0000","last_became_peered":"2024-04-25T05:44:28.166968+0000","last_unstale":"2024-04-25T05:44:56.286584+0000","last_undegraded":"2024-04-25T05:44:56.286584+0000","last_fullsized":"2024-04-25T05:44:56.286584+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":587,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285962+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:44:56.285962+0000","last_peered":"2024-04-25T05:44:56.285962+0000","last_clean":"2024-04-25T05:44:56.285962+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:44:56.285962+0000","last_undegraded":"2024-04-25T05:44:56.285962+0000","last_fullsized":"2024-04-25T05:44:56.285962+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":785,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.295002+0000","last_change":"2024-04-25T05:44:30.918290+0000","last_active":"2024-04-25T05:44:56.295002+0000","last_peered":"2024-04-25T05:44:56.295002+0000","last_clean":"2024-04-25T05:44:56.295002+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:44:56.295002+0000","last_undegraded":"2024-04-25T05:44:56.295002+0000","last_fullsized":"2024-04-25T05:44:56.295002+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:20.780175+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.780175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:42.506221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"338'8383","reported_seq":17767,"reported_epoch":338,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:57.913870+0000","last_change":"2024-04-25T05:44:29.210942+0000","last_active":"2024-04-25T05:44:57.913870+0000","last_peered":"2024-04-25T05:44:57.913870+0000","last_clean":"2024-04-25T05:44:27.036002+0000","last_became_active":"2024-04-25T05:44:29.183071+0000","last_became_peered":"2024-04-25T05:44:29.183071+0000","last_unstale":"2024-04-25T05:44:57.913870+0000","last_undegraded":"2024-04-25T05:44:57.913870+0000","last_fullsized":"2024-04-25T05:44:57.913870+0000","mapping_epoch":325,"log_start":"320'4811","ondisk_log_start":"320'4811","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433651712,"num_objects":6617,"num_object_clones":0,"num_object_copies":19851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2255,"num_objects_unfound":0,"num_objects_dirty":6617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8383,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4640,"num_bytes_recovered":304087040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":6617}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":455,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289896+0000","last_change":"2024-04-25T05:44:28.180356+0000","last_active":"2024-04-25T05:44:56.289896+0000","last_peered":"2024-04-25T05:44:56.289896+0000","last_clean":"2024-04-25T05:44:56.289896+0000","last_became_active":"2024-04-25T05:44:28.180059+0000","last_became_peered":"2024-04-25T05:44:28.180059+0000","last_unstale":"2024-04-25T05:44:56.289896+0000","last_undegraded":"2024-04-25T05:44:56.289896+0000","last_fullsized":"2024-04-25T05:44:56.289896+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.755552+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":129,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.406436+0000","last_change":"2024-04-25T05:44:28.234082+0000","last_active":"2024-04-25T05:44:56.406436+0000","last_peered":"2024-04-25T05:44:56.406436+0000","last_clean":"2024-04-25T05:44:56.406436+0000","last_became_active":"2024-04-25T05:44:28.177715+0000","last_became_peered":"2024-04-25T05:44:28.177715+0000","last_unstale":"2024-04-25T05:44:56.406436+0000","last_undegraded":"2024-04-25T05:44:56.406436+0000","last_fullsized":"2024-04-25T05:44:56.406436+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:44.249745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1281,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.283892+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:44:56.283892+0000","last_peered":"2024-04-25T05:44:56.283892+0000","last_clean":"2024-04-25T05:44:56.283892+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:44:56.283892+0000","last_undegraded":"2024-04-25T05:44:56.283892+0000","last_fullsized":"2024-04-25T05:44:56.283892+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.162472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":641,"num_read_kb":636,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287730+0000","last_change":"2024-04-25T05:44:28.172816+0000","last_active":"2024-04-25T05:44:56.287730+0000","last_peered":"2024-04-25T05:44:56.287730+0000","last_clean":"2024-04-25T05:44:56.287730+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:44:56.287730+0000","last_undegraded":"2024-04-25T05:44:56.287730+0000","last_fullsized":"2024-04-25T05:44:56.287730+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.421093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.421093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.272940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":575,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286315+0000","last_change":"2024-04-25T05:43:16.189190+0000","last_active":"2024-04-25T05:44:56.286315+0000","last_peered":"2024-04-25T05:44:56.286315+0000","last_clean":"2024-04-25T05:44:56.286315+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:44:56.286315+0000","last_undegraded":"2024-04-25T05:44:56.286315+0000","last_fullsized":"2024-04-25T05:44:56.286315+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":763,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286998+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:44:56.286998+0000","last_peered":"2024-04-25T05:44:56.286998+0000","last_clean":"2024-04-25T05:44:56.286998+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:44:56.286998+0000","last_undegraded":"2024-04-25T05:44:56.286998+0000","last_fullsized":"2024-04-25T05:44:56.286998+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"338'8481","reported_seq":15018,"reported_epoch":338,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:58.648864+0000","last_change":"2024-04-25T05:44:29.184637+0000","last_active":"2024-04-25T05:44:58.648864+0000","last_peered":"2024-04-25T05:44:58.648864+0000","last_clean":"2024-04-25T05:44:27.101546+0000","last_became_active":"2024-04-25T05:44:29.168753+0000","last_became_peered":"2024-04-25T05:44:29.168753+0000","last_unstale":"2024-04-25T05:44:58.648864+0000","last_undegraded":"2024-04-25T05:44:58.648864+0000","last_fullsized":"2024-04-25T05:44:58.648864+0000","mapping_epoch":325,"log_start":"321'5373","ondisk_log_start":"321'5373","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3108,"log_dups_size":0,"ondisk_log_size":3108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436404224,"num_objects":6659,"num_object_clones":0,"num_object_copies":19977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3671,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8481,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3216,"num_bytes_recovered":210763776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":6659}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":599,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284434+0000","last_change":"2024-04-25T05:44:28.178984+0000","last_active":"2024-04-25T05:44:56.284434+0000","last_peered":"2024-04-25T05:44:56.284434+0000","last_clean":"2024-04-25T05:44:56.284434+0000","last_became_active":"2024-04-25T05:44:28.178614+0000","last_became_peered":"2024-04-25T05:44:28.178614+0000","last_unstale":"2024-04-25T05:44:56.284434+0000","last_undegraded":"2024-04-25T05:44:56.284434+0000","last_fullsized":"2024-04-25T05:44:56.284434+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:23.945519+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":397,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294845+0000","last_change":"2024-04-25T05:43:22.635428+0000","last_active":"2024-04-25T05:44:56.294845+0000","last_peered":"2024-04-25T05:44:56.294845+0000","last_clean":"2024-04-25T05:44:56.294845+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:44:56.294845+0000","last_undegraded":"2024-04-25T05:44:56.294845+0000","last_fullsized":"2024-04-25T05:44:56.294845+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:43:22.635290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.635290+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":591,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.410026+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:44:56.410026+0000","last_peered":"2024-04-25T05:44:56.410026+0000","last_clean":"2024-04-25T05:44:56.410026+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:44:56.410026+0000","last_undegraded":"2024-04-25T05:44:56.410026+0000","last_fullsized":"2024-04-25T05:44:56.410026+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":898,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284034+0000","last_change":"2024-04-25T05:44:30.948007+0000","last_active":"2024-04-25T05:44:56.284034+0000","last_peered":"2024-04-25T05:44:56.284034+0000","last_clean":"2024-04-25T05:44:56.284034+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:44:56.284034+0000","last_undegraded":"2024-04-25T05:44:56.284034+0000","last_fullsized":"2024-04-25T05:44:56.284034+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":84,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.408711+0000","last_change":"2024-04-25T05:43:16.188274+0000","last_active":"2024-04-25T05:44:56.408711+0000","last_peered":"2024-04-25T05:44:56.408711+0000","last_clean":"2024-04-25T05:44:56.408711+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:44:56.408711+0000","last_undegraded":"2024-04-25T05:44:56.408711+0000","last_fullsized":"2024-04-25T05:44:56.408711+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"338'8613","reported_seq":8697,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.720735+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:44:58.720735+0000","last_peered":"2024-04-25T05:44:58.720735+0000","last_clean":"2024-04-25T05:44:58.720735+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:44:58.720735+0000","last_undegraded":"2024-04-25T05:44:58.720735+0000","last_fullsized":"2024-04-25T05:44:58.720735+0000","mapping_epoch":301,"log_start":"322'5670","ondisk_log_start":"322'5670","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2943,"log_dups_size":2015,"ondisk_log_size":2943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448331776,"num_objects":6841,"num_object_clones":0,"num_object_copies":20523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8613,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289452+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:44:56.289452+0000","last_peered":"2024-04-25T05:44:56.289452+0000","last_clean":"2024-04-25T05:44:56.289452+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:44:56.289452+0000","last_undegraded":"2024-04-25T05:44:56.289452+0000","last_fullsized":"2024-04-25T05:44:56.289452+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":708,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289516+0000","last_change":"2024-04-25T05:44:28.221414+0000","last_active":"2024-04-25T05:44:56.289516+0000","last_peered":"2024-04-25T05:44:56.289516+0000","last_clean":"2024-04-25T05:44:56.289516+0000","last_became_active":"2024-04-25T05:44:28.179973+0000","last_became_peered":"2024-04-25T05:44:28.179973+0000","last_unstale":"2024-04-25T05:44:56.289516+0000","last_undegraded":"2024-04-25T05:44:56.289516+0000","last_fullsized":"2024-04-25T05:44:56.289516+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":600,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288435+0000","last_change":"2024-04-25T05:44:28.186329+0000","last_active":"2024-04-25T05:44:56.288435+0000","last_peered":"2024-04-25T05:44:56.288435+0000","last_clean":"2024-04-25T05:44:56.288435+0000","last_became_active":"2024-04-25T05:44:28.184251+0000","last_became_peered":"2024-04-25T05:44:28.184251+0000","last_unstale":"2024-04-25T05:44:56.288435+0000","last_undegraded":"2024-04-25T05:44:56.288435+0000","last_fullsized":"2024-04-25T05:44:56.288435+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":619,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285738+0000","last_change":"2024-04-25T05:44:28.183966+0000","last_active":"2024-04-25T05:44:56.285738+0000","last_peered":"2024-04-25T05:44:56.285738+0000","last_clean":"2024-04-25T05:44:56.285738+0000","last_became_active":"2024-04-25T05:44:28.183528+0000","last_became_peered":"2024-04-25T05:44:28.183528+0000","last_unstale":"2024-04-25T05:44:56.285738+0000","last_undegraded":"2024-04-25T05:44:56.285738+0000","last_fullsized":"2024-04-25T05:44:56.285738+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.104924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":589,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284178+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:44:56.284178+0000","last_peered":"2024-04-25T05:44:56.284178+0000","last_clean":"2024-04-25T05:44:56.284178+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:44:56.284178+0000","last_undegraded":"2024-04-25T05:44:56.284178+0000","last_fullsized":"2024-04-25T05:44:56.284178+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":897,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285025+0000","last_change":"2024-04-25T05:44:30.947331+0000","last_active":"2024-04-25T05:44:56.285025+0000","last_peered":"2024-04-25T05:44:56.285025+0000","last_clean":"2024-04-25T05:44:56.285025+0000","last_became_active":"2024-04-25T05:44:30.946674+0000","last_became_peered":"2024-04-25T05:44:30.946674+0000","last_unstale":"2024-04-25T05:44:56.285025+0000","last_undegraded":"2024-04-25T05:44:56.285025+0000","last_fullsized":"2024-04-25T05:44:56.285025+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"338'8493","reported_seq":22727,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.935019+0000","last_change":"2024-04-25T05:44:47.251341+0000","last_active":"2024-04-25T05:44:57.935019+0000","last_peered":"2024-04-25T05:44:57.935019+0000","last_clean":"2024-04-25T05:44:57.935019+0000","last_became_active":"2024-04-25T05:44:47.245906+0000","last_became_peered":"2024-04-25T05:44:47.245906+0000","last_unstale":"2024-04-25T05:44:57.935019+0000","last_undegraded":"2024-04-25T05:44:57.935019+0000","last_fullsized":"2024-04-25T05:44:57.935019+0000","mapping_epoch":332,"log_start":"320'5208","ondisk_log_start":"320'5208","created":296,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8493,"num_write_kb":486464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7061,"num_bytes_recovered":462749696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":206,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286951+0000","last_change":"2024-04-25T05:44:28.188205+0000","last_active":"2024-04-25T05:44:56.286951+0000","last_peered":"2024-04-25T05:44:56.286951+0000","last_clean":"2024-04-25T05:44:56.286951+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:44:56.286951+0000","last_undegraded":"2024-04-25T05:44:56.286951+0000","last_fullsized":"2024-04-25T05:44:56.286951+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":828,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286738+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:44:56.286738+0000","last_peered":"2024-04-25T05:44:56.286738+0000","last_clean":"2024-04-25T05:44:56.286738+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:44:56.286738+0000","last_undegraded":"2024-04-25T05:44:56.286738+0000","last_fullsized":"2024-04-25T05:44:56.286738+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":376,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294176+0000","last_change":"2024-04-25T05:44:28.160664+0000","last_active":"2024-04-25T05:44:56.294176+0000","last_peered":"2024-04-25T05:44:56.294176+0000","last_clean":"2024-04-25T05:44:56.294176+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:44:56.294176+0000","last_undegraded":"2024-04-25T05:44:56.294176+0000","last_fullsized":"2024-04-25T05:44:56.294176+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.320979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":120,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289599+0000","last_change":"2024-04-25T05:44:28.205511+0000","last_active":"2024-04-25T05:44:56.289599+0000","last_peered":"2024-04-25T05:44:56.289599+0000","last_clean":"2024-04-25T05:44:56.289599+0000","last_became_active":"2024-04-25T05:44:28.176104+0000","last_became_peered":"2024-04-25T05:44:28.176104+0000","last_unstale":"2024-04-25T05:44:56.289599+0000","last_undegraded":"2024-04-25T05:44:56.289599+0000","last_fullsized":"2024-04-25T05:44:56.289599+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":561,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403715+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:44:56.403715+0000","last_peered":"2024-04-25T05:44:56.403715+0000","last_clean":"2024-04-25T05:44:56.403715+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:44:56.403715+0000","last_undegraded":"2024-04-25T05:44:56.403715+0000","last_fullsized":"2024-04-25T05:44:56.403715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":601,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287359+0000","last_change":"2024-04-25T05:44:30.918203+0000","last_active":"2024-04-25T05:44:56.287359+0000","last_peered":"2024-04-25T05:44:56.287359+0000","last_clean":"2024-04-25T05:44:56.287359+0000","last_became_active":"2024-04-25T05:44:30.917713+0000","last_became_peered":"2024-04-25T05:44:30.917713+0000","last_unstale":"2024-04-25T05:44:56.287359+0000","last_undegraded":"2024-04-25T05:44:56.287359+0000","last_fullsized":"2024-04-25T05:44:56.287359+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.424874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"338'8446","reported_seq":16153,"reported_epoch":338,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:58.727616+0000","last_change":"2024-04-25T05:44:29.205531+0000","last_active":"2024-04-25T05:44:58.727616+0000","last_peered":"2024-04-25T05:44:58.727616+0000","last_clean":"2024-04-25T05:44:27.108028+0000","last_became_active":"2024-04-25T05:44:29.173771+0000","last_became_peered":"2024-04-25T05:44:29.173771+0000","last_unstale":"2024-04-25T05:44:58.727616+0000","last_undegraded":"2024-04-25T05:44:58.727616+0000","last_fullsized":"2024-04-25T05:44:58.727616+0000","mapping_epoch":325,"log_start":"321'5491","ondisk_log_start":"321'5491","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2955,"log_dups_size":2015,"ondisk_log_size":2955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434241536,"num_objects":6626,"num_object_clones":0,"num_object_copies":19878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3037,"num_objects_unfound":0,"num_objects_dirty":6626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3807,"num_bytes_recovered":249495552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":6626}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":299,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286329+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:44:56.286329+0000","last_peered":"2024-04-25T05:44:56.286329+0000","last_clean":"2024-04-25T05:44:56.286329+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:44:56.286329+0000","last_undegraded":"2024-04-25T05:44:56.286329+0000","last_fullsized":"2024-04-25T05:44:56.286329+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":390,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403673+0000","last_change":"2024-04-25T05:43:16.191937+0000","last_active":"2024-04-25T05:44:56.403673+0000","last_peered":"2024-04-25T05:44:56.403673+0000","last_clean":"2024-04-25T05:44:56.403673+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:44:56.403673+0000","last_undegraded":"2024-04-25T05:44:56.403673+0000","last_fullsized":"2024-04-25T05:44:56.403673+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":229,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288776+0000","last_change":"2024-04-25T05:44:28.171402+0000","last_active":"2024-04-25T05:44:56.288776+0000","last_peered":"2024-04-25T05:44:56.288776+0000","last_clean":"2024-04-25T05:44:56.288776+0000","last_became_active":"2024-04-25T05:44:28.171004+0000","last_became_peered":"2024-04-25T05:44:28.171004+0000","last_unstale":"2024-04-25T05:44:56.288776+0000","last_undegraded":"2024-04-25T05:44:56.288776+0000","last_fullsized":"2024-04-25T05:44:56.288776+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.186275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.186275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:20.209466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285464+0000","last_change":"2024-04-25T05:44:28.169508+0000","last_active":"2024-04-25T05:44:56.285464+0000","last_peered":"2024-04-25T05:44:56.285464+0000","last_clean":"2024-04-25T05:44:56.285464+0000","last_became_active":"2024-04-25T05:44:28.169146+0000","last_became_peered":"2024-04-25T05:44:28.169146+0000","last_unstale":"2024-04-25T05:44:56.285464+0000","last_undegraded":"2024-04-25T05:44:56.285464+0000","last_fullsized":"2024-04-25T05:44:56.285464+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"338'8320","reported_seq":8403,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:57.900059+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:44:57.900059+0000","last_peered":"2024-04-25T05:44:57.900059+0000","last_clean":"2024-04-25T05:44:57.900059+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:44:57.900059+0000","last_undegraded":"2024-04-25T05:44:57.900059+0000","last_fullsized":"2024-04-25T05:44:57.900059+0000","mapping_epoch":301,"log_start":"320'5055","ondisk_log_start":"320'5055","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432013312,"num_objects":6592,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":350,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.293025+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:44:56.293025+0000","last_peered":"2024-04-25T05:44:56.293025+0000","last_clean":"2024-04-25T05:44:56.293025+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:44:56.293025+0000","last_undegraded":"2024-04-25T05:44:56.293025+0000","last_fullsized":"2024-04-25T05:44:56.293025+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":624,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284065+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:44:56.284065+0000","last_peered":"2024-04-25T05:44:56.284065+0000","last_clean":"2024-04-25T05:44:56.284065+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:44:56.284065+0000","last_undegraded":"2024-04-25T05:44:56.284065+0000","last_fullsized":"2024-04-25T05:44:56.284065+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":533,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284585+0000","last_change":"2024-04-25T05:44:28.180755+0000","last_active":"2024-04-25T05:44:56.284585+0000","last_peered":"2024-04-25T05:44:56.284585+0000","last_clean":"2024-04-25T05:44:56.284585+0000","last_became_active":"2024-04-25T05:44:28.180393+0000","last_became_peered":"2024-04-25T05:44:28.180393+0000","last_unstale":"2024-04-25T05:44:56.284585+0000","last_undegraded":"2024-04-25T05:44:56.284585+0000","last_fullsized":"2024-04-25T05:44:56.284585+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.534093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":583,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.283776+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:44:56.283776+0000","last_peered":"2024-04-25T05:44:56.283776+0000","last_clean":"2024-04-25T05:44:56.283776+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:44:56.283776+0000","last_undegraded":"2024-04-25T05:44:56.283776+0000","last_fullsized":"2024-04-25T05:44:56.283776+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.832260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"338'8555","reported_seq":8648,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:58.720547+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:44:58.720547+0000","last_peered":"2024-04-25T05:44:58.720547+0000","last_clean":"2024-04-25T05:44:58.720547+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:44:58.720547+0000","last_undegraded":"2024-04-25T05:44:58.720547+0000","last_fullsized":"2024-04-25T05:44:58.720547+0000","mapping_epoch":296,"log_start":"321'5586","ondisk_log_start":"321'5586","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2969,"log_dups_size":2015,"ondisk_log_size":2969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":20277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":534,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286936+0000","last_change":"2024-04-25T05:44:28.169954+0000","last_active":"2024-04-25T05:44:56.286936+0000","last_peered":"2024-04-25T05:44:56.286936+0000","last_clean":"2024-04-25T05:44:56.286936+0000","last_became_active":"2024-04-25T05:44:28.168753+0000","last_became_peered":"2024-04-25T05:44:28.168753+0000","last_unstale":"2024-04-25T05:44:56.286936+0000","last_undegraded":"2024-04-25T05:44:56.286936+0000","last_fullsized":"2024-04-25T05:44:56.286936+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":238,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285630+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:44:56.285630+0000","last_peered":"2024-04-25T05:44:56.285630+0000","last_clean":"2024-04-25T05:44:56.285630+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:44:56.285630+0000","last_undegraded":"2024-04-25T05:44:56.285630+0000","last_fullsized":"2024-04-25T05:44:56.285630+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.771369+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":588,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289103+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:44:56.289103+0000","last_peered":"2024-04-25T05:44:56.289103+0000","last_clean":"2024-04-25T05:44:56.289103+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:44:56.289103+0000","last_undegraded":"2024-04-25T05:44:56.289103+0000","last_fullsized":"2024-04-25T05:44:56.289103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":572,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.407492+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:44:56.407492+0000","last_peered":"2024-04-25T05:44:56.407492+0000","last_clean":"2024-04-25T05:44:56.407492+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:44:56.407492+0000","last_undegraded":"2024-04-25T05:44:56.407492+0000","last_fullsized":"2024-04-25T05:44:56.407492+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"338'8351","reported_seq":18598,"reported_epoch":338,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:44:57.940844+0000","last_change":"2024-04-25T05:44:29.255024+0000","last_active":"2024-04-25T05:44:57.940844+0000","last_peered":"2024-04-25T05:44:57.940844+0000","last_clean":"2024-04-25T05:44:27.090244+0000","last_became_active":"2024-04-25T05:44:29.209023+0000","last_became_peered":"2024-04-25T05:44:29.209023+0000","last_unstale":"2024-04-25T05:44:57.940844+0000","last_undegraded":"2024-04-25T05:44:57.940844+0000","last_fullsized":"2024-04-25T05:44:57.940844+0000","mapping_epoch":325,"log_start":"320'5093","ondisk_log_start":"320'5093","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3258,"log_dups_size":0,"ondisk_log_size":3258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1934,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5069,"num_bytes_recovered":332201984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":6709}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":745,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287900+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:44:56.287900+0000","last_peered":"2024-04-25T05:44:56.287900+0000","last_clean":"2024-04-25T05:44:56.287900+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:44:56.287900+0000","last_undegraded":"2024-04-25T05:44:56.287900+0000","last_fullsized":"2024-04-25T05:44:56.287900+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":190,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284197+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:44:56.284197+0000","last_peered":"2024-04-25T05:44:56.284197+0000","last_clean":"2024-04-25T05:44:56.284197+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:44:56.284197+0000","last_undegraded":"2024-04-25T05:44:56.284197+0000","last_fullsized":"2024-04-25T05:44:56.284197+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":636,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285317+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:44:56.285317+0000","last_peered":"2024-04-25T05:44:56.285317+0000","last_clean":"2024-04-25T05:44:56.285317+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:44:56.285317+0000","last_undegraded":"2024-04-25T05:44:56.285317+0000","last_fullsized":"2024-04-25T05:44:56.285317+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.858926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":463,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.292823+0000","last_change":"2024-04-25T05:44:28.174865+0000","last_active":"2024-04-25T05:44:56.292823+0000","last_peered":"2024-04-25T05:44:56.292823+0000","last_clean":"2024-04-25T05:44:56.292823+0000","last_became_active":"2024-04-25T05:44:28.174499+0000","last_became_peered":"2024-04-25T05:44:28.174499+0000","last_unstale":"2024-04-25T05:44:56.292823+0000","last_undegraded":"2024-04-25T05:44:56.292823+0000","last_fullsized":"2024-04-25T05:44:56.292823+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":606,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.409005+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:44:56.409005+0000","last_peered":"2024-04-25T05:44:56.409005+0000","last_clean":"2024-04-25T05:44:56.409005+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:44:56.409005+0000","last_undegraded":"2024-04-25T05:44:56.409005+0000","last_fullsized":"2024-04-25T05:44:56.409005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.253497+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294138+0000","last_change":"2024-04-25T05:44:28.191586+0000","last_active":"2024-04-25T05:44:56.294138+0000","last_peered":"2024-04-25T05:44:56.294138+0000","last_clean":"2024-04-25T05:44:56.294138+0000","last_became_active":"2024-04-25T05:44:28.191288+0000","last_became_peered":"2024-04-25T05:44:28.191288+0000","last_unstale":"2024-04-25T05:44:56.294138+0000","last_undegraded":"2024-04-25T05:44:56.294138+0000","last_fullsized":"2024-04-25T05:44:56.294138+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.539924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":92,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403549+0000","last_change":"2024-04-25T05:44:19.931291+0000","last_active":"2024-04-25T05:44:56.403549+0000","last_peered":"2024-04-25T05:44:56.403549+0000","last_clean":"2024-04-25T05:44:56.403549+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:44:56.403549+0000","last_undegraded":"2024-04-25T05:44:56.403549+0000","last_fullsized":"2024-04-25T05:44:56.403549+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.670336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":588,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.283717+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:44:56.283717+0000","last_peered":"2024-04-25T05:44:56.283717+0000","last_clean":"2024-04-25T05:44:56.283717+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:44:56.283717+0000","last_undegraded":"2024-04-25T05:44:56.283717+0000","last_fullsized":"2024-04-25T05:44:56.283717+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:06.898722+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":267,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403905+0000","last_change":"2024-04-25T05:44:28.203127+0000","last_active":"2024-04-25T05:44:56.403905+0000","last_peered":"2024-04-25T05:44:56.403905+0000","last_clean":"2024-04-25T05:44:56.403905+0000","last_became_active":"2024-04-25T05:44:28.184563+0000","last_became_peered":"2024-04-25T05:44:28.184563+0000","last_unstale":"2024-04-25T05:44:56.403905+0000","last_undegraded":"2024-04-25T05:44:56.403905+0000","last_fullsized":"2024-04-25T05:44:56.403905+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":606,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287720+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:44:56.287720+0000","last_peered":"2024-04-25T05:44:56.287720+0000","last_clean":"2024-04-25T05:44:56.287720+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:44:56.287720+0000","last_undegraded":"2024-04-25T05:44:56.287720+0000","last_fullsized":"2024-04-25T05:44:56.287720+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:44:59.461950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":564,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.408810+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:44:56.408810+0000","last_peered":"2024-04-25T05:44:56.408810+0000","last_clean":"2024-04-25T05:44:56.408810+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:44:56.408810+0000","last_undegraded":"2024-04-25T05:44:56.408810+0000","last_fullsized":"2024-04-25T05:44:56.408810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.522496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":230,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287645+0000","last_change":"2024-04-25T05:44:28.120205+0000","last_active":"2024-04-25T05:44:56.287645+0000","last_peered":"2024-04-25T05:44:56.287645+0000","last_clean":"2024-04-25T05:44:56.287645+0000","last_became_active":"2024-04-25T05:44:28.119749+0000","last_became_peered":"2024-04-25T05:44:28.119749+0000","last_unstale":"2024-04-25T05:44:56.287645+0000","last_undegraded":"2024-04-25T05:44:56.287645+0000","last_fullsized":"2024-04-25T05:44:56.287645+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.285907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":209,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284491+0000","last_change":"2024-04-25T05:43:16.229957+0000","last_active":"2024-04-25T05:44:56.284491+0000","last_peered":"2024-04-25T05:44:56.284491+0000","last_clean":"2024-04-25T05:44:56.284491+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:44:56.284491+0000","last_undegraded":"2024-04-25T05:44:56.284491+0000","last_fullsized":"2024-04-25T05:44:56.284491+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284810+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:44:56.284810+0000","last_peered":"2024-04-25T05:44:56.284810+0000","last_clean":"2024-04-25T05:44:56.284810+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:44:56.284810+0000","last_undegraded":"2024-04-25T05:44:56.284810+0000","last_fullsized":"2024-04-25T05:44:56.284810+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.919590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":714,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.404113+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:44:56.404113+0000","last_peered":"2024-04-25T05:44:56.404113+0000","last_clean":"2024-04-25T05:44:56.404113+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:44:56.404113+0000","last_undegraded":"2024-04-25T05:44:56.404113+0000","last_fullsized":"2024-04-25T05:44:56.404113+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.732091+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":44,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.283660+0000","last_change":"2024-04-25T05:44:28.167266+0000","last_active":"2024-04-25T05:44:56.283660+0000","last_peered":"2024-04-25T05:44:56.283660+0000","last_clean":"2024-04-25T05:44:56.283660+0000","last_became_active":"2024-04-25T05:44:28.166838+0000","last_became_peered":"2024-04-25T05:44:28.166838+0000","last_unstale":"2024-04-25T05:44:56.283660+0000","last_undegraded":"2024-04-25T05:44:56.283660+0000","last_fullsized":"2024-04-25T05:44:56.283660+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.460917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":622,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403566+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:44:56.403566+0000","last_peered":"2024-04-25T05:44:56.403566+0000","last_clean":"2024-04-25T05:44:56.403566+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:44:56.403566+0000","last_undegraded":"2024-04-25T05:44:56.403566+0000","last_fullsized":"2024-04-25T05:44:56.403566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":594,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284775+0000","last_change":"2024-04-25T05:44:13.015441+0000","last_active":"2024-04-25T05:44:56.284775+0000","last_peered":"2024-04-25T05:44:56.284775+0000","last_clean":"2024-04-25T05:44:56.284775+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:44:56.284775+0000","last_undegraded":"2024-04-25T05:44:56.284775+0000","last_fullsized":"2024-04-25T05:44:56.284775+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:27.074191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":143,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286753+0000","last_change":"2024-04-25T05:44:28.236616+0000","last_active":"2024-04-25T05:44:56.286753+0000","last_peered":"2024-04-25T05:44:56.286753+0000","last_clean":"2024-04-25T05:44:56.286753+0000","last_became_active":"2024-04-25T05:44:28.184915+0000","last_became_peered":"2024-04-25T05:44:28.184915+0000","last_unstale":"2024-04-25T05:44:56.286753+0000","last_undegraded":"2024-04-25T05:44:56.286753+0000","last_fullsized":"2024-04-25T05:44:56.286753+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288864+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:44:56.288864+0000","last_peered":"2024-04-25T05:44:56.288864+0000","last_clean":"2024-04-25T05:44:56.288864+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:44:56.288864+0000","last_undegraded":"2024-04-25T05:44:56.288864+0000","last_fullsized":"2024-04-25T05:44:56.288864+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":700,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289518+0000","last_change":"2024-04-25T05:44:19.254245+0000","last_active":"2024-04-25T05:44:56.289518+0000","last_peered":"2024-04-25T05:44:56.289518+0000","last_clean":"2024-04-25T05:44:56.289518+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:44:56.289518+0000","last_undegraded":"2024-04-25T05:44:56.289518+0000","last_fullsized":"2024-04-25T05:44:56.289518+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:19.254124+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.254124+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:41.554540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":260,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284154+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:44:56.284154+0000","last_peered":"2024-04-25T05:44:56.284154+0000","last_clean":"2024-04-25T05:44:56.284154+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:44:56.284154+0000","last_undegraded":"2024-04-25T05:44:56.284154+0000","last_fullsized":"2024-04-25T05:44:56.284154+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":581,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405369+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:44:56.405369+0000","last_peered":"2024-04-25T05:44:56.405369+0000","last_clean":"2024-04-25T05:44:56.405369+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:44:56.405369+0000","last_undegraded":"2024-04-25T05:44:56.405369+0000","last_fullsized":"2024-04-25T05:44:56.405369+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":592,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288327+0000","last_change":"2024-04-25T05:44:28.186508+0000","last_active":"2024-04-25T05:44:56.288327+0000","last_peered":"2024-04-25T05:44:56.288327+0000","last_clean":"2024-04-25T05:44:56.288327+0000","last_became_active":"2024-04-25T05:44:28.184914+0000","last_became_peered":"2024-04-25T05:44:28.184914+0000","last_unstale":"2024-04-25T05:44:56.288327+0000","last_undegraded":"2024-04-25T05:44:56.288327+0000","last_fullsized":"2024-04-25T05:44:56.288327+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":118,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287011+0000","last_change":"2024-04-25T05:44:28.208196+0000","last_active":"2024-04-25T05:44:56.287011+0000","last_peered":"2024-04-25T05:44:56.287011+0000","last_clean":"2024-04-25T05:44:56.287011+0000","last_became_active":"2024-04-25T05:44:28.160392+0000","last_became_peered":"2024-04-25T05:44:28.160392+0000","last_unstale":"2024-04-25T05:44:56.287011+0000","last_undegraded":"2024-04-25T05:44:56.287011+0000","last_fullsized":"2024-04-25T05:44:56.287011+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":588,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.283841+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:44:56.283841+0000","last_peered":"2024-04-25T05:44:56.283841+0000","last_clean":"2024-04-25T05:44:56.283841+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:44:56.283841+0000","last_undegraded":"2024-04-25T05:44:56.283841+0000","last_fullsized":"2024-04-25T05:44:56.283841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":582,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405075+0000","last_change":"2024-04-25T05:44:28.174271+0000","last_active":"2024-04-25T05:44:56.405075+0000","last_peered":"2024-04-25T05:44:56.405075+0000","last_clean":"2024-04-25T05:44:56.405075+0000","last_became_active":"2024-04-25T05:44:28.172179+0000","last_became_peered":"2024-04-25T05:44:28.172179+0000","last_unstale":"2024-04-25T05:44:56.405075+0000","last_undegraded":"2024-04-25T05:44:56.405075+0000","last_fullsized":"2024-04-25T05:44:56.405075+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.204906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":193,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286700+0000","last_change":"2024-04-25T05:44:28.179459+0000","last_active":"2024-04-25T05:44:56.286700+0000","last_peered":"2024-04-25T05:44:56.286700+0000","last_clean":"2024-04-25T05:44:56.286700+0000","last_became_active":"2024-04-25T05:44:28.176134+0000","last_became_peered":"2024-04-25T05:44:28.176134+0000","last_unstale":"2024-04-25T05:44:56.286700+0000","last_undegraded":"2024-04-25T05:44:56.286700+0000","last_fullsized":"2024-04-25T05:44:56.286700+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":659,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288724+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:44:56.288724+0000","last_peered":"2024-04-25T05:44:56.288724+0000","last_clean":"2024-04-25T05:44:56.288724+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:44:56.288724+0000","last_undegraded":"2024-04-25T05:44:56.288724+0000","last_fullsized":"2024-04-25T05:44:56.288724+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.001968+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":92,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289018+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:44:56.289018+0000","last_peered":"2024-04-25T05:44:56.289018+0000","last_clean":"2024-04-25T05:44:56.289018+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:44:56.289018+0000","last_undegraded":"2024-04-25T05:44:56.289018+0000","last_fullsized":"2024-04-25T05:44:56.289018+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":598,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284378+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:44:56.284378+0000","last_peered":"2024-04-25T05:44:56.284378+0000","last_clean":"2024-04-25T05:44:56.284378+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:44:56.284378+0000","last_undegraded":"2024-04-25T05:44:56.284378+0000","last_fullsized":"2024-04-25T05:44:56.284378+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":590,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.407324+0000","last_change":"2024-04-25T05:44:28.179442+0000","last_active":"2024-04-25T05:44:56.407324+0000","last_peered":"2024-04-25T05:44:56.407324+0000","last_clean":"2024-04-25T05:44:56.407324+0000","last_became_active":"2024-04-25T05:44:28.179103+0000","last_became_peered":"2024-04-25T05:44:28.179103+0000","last_unstale":"2024-04-25T05:44:56.407324+0000","last_undegraded":"2024-04-25T05:44:56.407324+0000","last_fullsized":"2024-04-25T05:44:56.407324+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":604,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288654+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:44:56.288654+0000","last_peered":"2024-04-25T05:44:56.288654+0000","last_clean":"2024-04-25T05:44:56.288654+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:44:56.288654+0000","last_undegraded":"2024-04-25T05:44:56.288654+0000","last_fullsized":"2024-04-25T05:44:56.288654+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":185,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405030+0000","last_change":"2024-04-25T05:44:28.188135+0000","last_active":"2024-04-25T05:44:56.405030+0000","last_peered":"2024-04-25T05:44:56.405030+0000","last_clean":"2024-04-25T05:44:56.405030+0000","last_became_active":"2024-04-25T05:44:28.187806+0000","last_became_peered":"2024-04-25T05:44:28.187806+0000","last_unstale":"2024-04-25T05:44:56.405030+0000","last_undegraded":"2024-04-25T05:44:56.405030+0000","last_fullsized":"2024-04-25T05:44:56.405030+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":296,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294406+0000","last_change":"2024-04-25T05:43:29.610487+0000","last_active":"2024-04-25T05:44:56.294406+0000","last_peered":"2024-04-25T05:44:56.294406+0000","last_clean":"2024-04-25T05:44:56.294406+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:44:56.294406+0000","last_undegraded":"2024-04-25T05:44:56.294406+0000","last_fullsized":"2024-04-25T05:44:56.294406+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.610411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.610411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":587,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285054+0000","last_change":"2024-04-25T05:44:19.960606+0000","last_active":"2024-04-25T05:44:56.285054+0000","last_peered":"2024-04-25T05:44:56.285054+0000","last_clean":"2024-04-25T05:44:56.285054+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:44:56.285054+0000","last_undegraded":"2024-04-25T05:44:56.285054+0000","last_fullsized":"2024-04-25T05:44:56.285054+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:24.771833+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":376,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.295200+0000","last_change":"2024-04-25T05:44:28.212080+0000","last_active":"2024-04-25T05:44:56.295200+0000","last_peered":"2024-04-25T05:44:56.295200+0000","last_clean":"2024-04-25T05:44:56.295200+0000","last_became_active":"2024-04-25T05:44:28.181996+0000","last_became_peered":"2024-04-25T05:44:28.181996+0000","last_unstale":"2024-04-25T05:44:56.295200+0000","last_undegraded":"2024-04-25T05:44:56.295200+0000","last_fullsized":"2024-04-25T05:44:56.295200+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.299319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":598,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288489+0000","last_change":"2024-04-25T05:44:28.172975+0000","last_active":"2024-04-25T05:44:56.288489+0000","last_peered":"2024-04-25T05:44:56.288489+0000","last_clean":"2024-04-25T05:44:56.288489+0000","last_became_active":"2024-04-25T05:44:28.171311+0000","last_became_peered":"2024-04-25T05:44:28.171311+0000","last_unstale":"2024-04-25T05:44:56.288489+0000","last_undegraded":"2024-04-25T05:44:56.288489+0000","last_fullsized":"2024-04-25T05:44:56.288489+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:37.398601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289201+0000","last_change":"2024-04-25T05:44:28.179599+0000","last_active":"2024-04-25T05:44:56.289201+0000","last_peered":"2024-04-25T05:44:56.289201+0000","last_clean":"2024-04-25T05:44:56.289201+0000","last_became_active":"2024-04-25T05:44:28.178710+0000","last_became_peered":"2024-04-25T05:44:28.178710+0000","last_unstale":"2024-04-25T05:44:56.289201+0000","last_undegraded":"2024-04-25T05:44:56.289201+0000","last_fullsized":"2024-04-25T05:44:56.289201+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:32.286921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":574,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286527+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:44:56.286527+0000","last_peered":"2024-04-25T05:44:56.286527+0000","last_clean":"2024-04-25T05:44:56.286527+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:44:56.286527+0000","last_undegraded":"2024-04-25T05:44:56.286527+0000","last_fullsized":"2024-04-25T05:44:56.286527+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":273,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289354+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:44:56.289354+0000","last_peered":"2024-04-25T05:44:56.289354+0000","last_clean":"2024-04-25T05:44:56.289354+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:44:56.289354+0000","last_undegraded":"2024-04-25T05:44:56.289354+0000","last_fullsized":"2024-04-25T05:44:56.289354+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":589,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285738+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:44:56.285738+0000","last_peered":"2024-04-25T05:44:56.285738+0000","last_clean":"2024-04-25T05:44:56.285738+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:44:56.285738+0000","last_undegraded":"2024-04-25T05:44:56.285738+0000","last_fullsized":"2024-04-25T05:44:56.285738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:40.500720+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":373,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.295059+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:44:56.295059+0000","last_peered":"2024-04-25T05:44:56.295059+0000","last_clean":"2024-04-25T05:44:56.295059+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:44:56.295059+0000","last_undegraded":"2024-04-25T05:44:56.295059+0000","last_fullsized":"2024-04-25T05:44:56.295059+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.185065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":597,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284317+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:44:56.284317+0000","last_peered":"2024-04-25T05:44:56.284317+0000","last_clean":"2024-04-25T05:44:56.284317+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:44:56.284317+0000","last_undegraded":"2024-04-25T05:44:56.284317+0000","last_fullsized":"2024-04-25T05:44:56.284317+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":612,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403711+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:44:56.403711+0000","last_peered":"2024-04-25T05:44:56.403711+0000","last_clean":"2024-04-25T05:44:56.403711+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:44:56.403711+0000","last_undegraded":"2024-04-25T05:44:56.403711+0000","last_fullsized":"2024-04-25T05:44:56.403711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":600,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288576+0000","last_change":"2024-04-25T05:44:28.167895+0000","last_active":"2024-04-25T05:44:56.288576+0000","last_peered":"2024-04-25T05:44:56.288576+0000","last_clean":"2024-04-25T05:44:56.288576+0000","last_became_active":"2024-04-25T05:44:28.167563+0000","last_became_peered":"2024-04-25T05:44:28.167563+0000","last_unstale":"2024-04-25T05:44:56.288576+0000","last_undegraded":"2024-04-25T05:44:56.288576+0000","last_fullsized":"2024-04-25T05:44:56.288576+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.192432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.192432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":257,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403783+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:44:56.403783+0000","last_peered":"2024-04-25T05:44:56.403783+0000","last_clean":"2024-04-25T05:44:56.403783+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:44:56.403783+0000","last_undegraded":"2024-04-25T05:44:56.403783+0000","last_fullsized":"2024-04-25T05:44:56.403783+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":579,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.408910+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:44:56.408910+0000","last_peered":"2024-04-25T05:44:56.408910+0000","last_clean":"2024-04-25T05:44:56.408910+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:44:56.408910+0000","last_undegraded":"2024-04-25T05:44:56.408910+0000","last_fullsized":"2024-04-25T05:44:56.408910+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":479,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286475+0000","last_change":"2024-04-25T05:44:28.237495+0000","last_active":"2024-04-25T05:44:56.286475+0000","last_peered":"2024-04-25T05:44:56.286475+0000","last_clean":"2024-04-25T05:44:56.286475+0000","last_became_active":"2024-04-25T05:44:28.182958+0000","last_became_peered":"2024-04-25T05:44:28.182958+0000","last_unstale":"2024-04-25T05:44:56.286475+0000","last_undegraded":"2024-04-25T05:44:56.286475+0000","last_fullsized":"2024-04-25T05:44:56.286475+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":76,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285118+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:44:56.285118+0000","last_peered":"2024-04-25T05:44:56.285118+0000","last_clean":"2024-04-25T05:44:56.285118+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:44:56.285118+0000","last_undegraded":"2024-04-25T05:44:56.285118+0000","last_fullsized":"2024-04-25T05:44:56.285118+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:21.442272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":598,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288405+0000","last_change":"2024-04-25T05:44:28.189092+0000","last_active":"2024-04-25T05:44:56.288405+0000","last_peered":"2024-04-25T05:44:56.288405+0000","last_clean":"2024-04-25T05:44:56.288405+0000","last_became_active":"2024-04-25T05:44:28.185597+0000","last_became_peered":"2024-04-25T05:44:28.185597+0000","last_unstale":"2024-04-25T05:44:56.288405+0000","last_undegraded":"2024-04-25T05:44:56.288405+0000","last_fullsized":"2024-04-25T05:44:56.288405+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.532823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":595,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.290184+0000","last_change":"2024-04-25T05:44:00.169219+0000","last_active":"2024-04-25T05:44:56.290184+0000","last_peered":"2024-04-25T05:44:56.290184+0000","last_clean":"2024-04-25T05:44:56.290184+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:44:56.290184+0000","last_undegraded":"2024-04-25T05:44:56.290184+0000","last_fullsized":"2024-04-25T05:44:56.290184+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.169057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.169057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:03.657106+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.404891+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:44:56.404891+0000","last_peered":"2024-04-25T05:44:56.404891+0000","last_clean":"2024-04-25T05:44:56.404891+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:44:56.404891+0000","last_undegraded":"2024-04-25T05:44:56.404891+0000","last_fullsized":"2024-04-25T05:44:56.404891+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":92,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287440+0000","last_change":"2024-04-25T05:43:17.283718+0000","last_active":"2024-04-25T05:44:56.287440+0000","last_peered":"2024-04-25T05:44:56.287440+0000","last_clean":"2024-04-25T05:44:56.287440+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:44:56.287440+0000","last_undegraded":"2024-04-25T05:44:56.287440+0000","last_fullsized":"2024-04-25T05:44:56.287440+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.283618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.283618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":572,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.284913+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:44:56.284913+0000","last_peered":"2024-04-25T05:44:56.284913+0000","last_clean":"2024-04-25T05:44:56.284913+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:44:56.284913+0000","last_undegraded":"2024-04-25T05:44:56.284913+0000","last_fullsized":"2024-04-25T05:44:56.284913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":90,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286046+0000","last_change":"2024-04-25T05:44:28.201019+0000","last_active":"2024-04-25T05:44:56.286046+0000","last_peered":"2024-04-25T05:44:56.286046+0000","last_clean":"2024-04-25T05:44:56.286046+0000","last_became_active":"2024-04-25T05:44:28.200623+0000","last_became_peered":"2024-04-25T05:44:28.200623+0000","last_unstale":"2024-04-25T05:44:56.286046+0000","last_undegraded":"2024-04-25T05:44:56.286046+0000","last_fullsized":"2024-04-25T05:44:56.286046+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":359,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.410173+0000","last_change":"2024-04-25T05:44:28.198166+0000","last_active":"2024-04-25T05:44:56.410173+0000","last_peered":"2024-04-25T05:44:56.410173+0000","last_clean":"2024-04-25T05:44:56.410173+0000","last_became_active":"2024-04-25T05:44:28.179156+0000","last_became_peered":"2024-04-25T05:44:28.179156+0000","last_unstale":"2024-04-25T05:44:56.410173+0000","last_undegraded":"2024-04-25T05:44:56.410173+0000","last_fullsized":"2024-04-25T05:44:56.410173+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":398,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294884+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:44:56.294884+0000","last_peered":"2024-04-25T05:44:56.294884+0000","last_clean":"2024-04-25T05:44:56.294884+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:44:56.294884+0000","last_undegraded":"2024-04-25T05:44:56.294884+0000","last_fullsized":"2024-04-25T05:44:56.294884+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.453574+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":396,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285023+0000","last_change":"2024-04-25T05:44:28.126200+0000","last_active":"2024-04-25T05:44:56.285023+0000","last_peered":"2024-04-25T05:44:56.285023+0000","last_clean":"2024-04-25T05:44:56.285023+0000","last_became_active":"2024-04-25T05:44:28.125552+0000","last_became_peered":"2024-04-25T05:44:28.125552+0000","last_unstale":"2024-04-25T05:44:56.285023+0000","last_undegraded":"2024-04-25T05:44:56.285023+0000","last_fullsized":"2024-04-25T05:44:56.285023+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":200,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.287044+0000","last_change":"2024-04-25T05:44:28.169494+0000","last_active":"2024-04-25T05:44:56.287044+0000","last_peered":"2024-04-25T05:44:56.287044+0000","last_clean":"2024-04-25T05:44:56.287044+0000","last_became_active":"2024-04-25T05:44:28.168461+0000","last_became_peered":"2024-04-25T05:44:28.168461+0000","last_unstale":"2024-04-25T05:44:56.287044+0000","last_undegraded":"2024-04-25T05:44:56.287044+0000","last_fullsized":"2024-04-25T05:44:56.287044+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":809,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.288951+0000","last_change":"2024-04-25T05:44:28.225524+0000","last_active":"2024-04-25T05:44:56.288951+0000","last_peered":"2024-04-25T05:44:56.288951+0000","last_clean":"2024-04-25T05:44:56.288951+0000","last_became_active":"2024-04-25T05:44:28.182800+0000","last_became_peered":"2024-04-25T05:44:28.182800+0000","last_unstale":"2024-04-25T05:44:56.288951+0000","last_undegraded":"2024-04-25T05:44:56.288951+0000","last_fullsized":"2024-04-25T05:44:56.288951+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":599,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.403376+0000","last_change":"2024-04-25T05:44:16.995982+0000","last_active":"2024-04-25T05:44:56.403376+0000","last_peered":"2024-04-25T05:44:56.403376+0000","last_clean":"2024-04-25T05:44:56.403376+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:44:56.403376+0000","last_undegraded":"2024-04-25T05:44:56.403376+0000","last_fullsized":"2024-04-25T05:44:56.403376+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:38.671823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":547,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294411+0000","last_change":"2024-04-25T05:43:18.685398+0000","last_active":"2024-04-25T05:44:56.294411+0000","last_peered":"2024-04-25T05:44:56.294411+0000","last_clean":"2024-04-25T05:44:56.294411+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:44:56.294411+0000","last_undegraded":"2024-04-25T05:44:56.294411+0000","last_fullsized":"2024-04-25T05:44:56.294411+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.685299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.685299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":572,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.405162+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:44:56.405162+0000","last_peered":"2024-04-25T05:44:56.405162+0000","last_clean":"2024-04-25T05:44:56.405162+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:44:56.405162+0000","last_undegraded":"2024-04-25T05:44:56.405162+0000","last_fullsized":"2024-04-25T05:44:56.405162+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":662,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.286927+0000","last_change":"2024-04-25T05:44:02.120412+0000","last_active":"2024-04-25T05:44:56.286927+0000","last_peered":"2024-04-25T05:44:56.286927+0000","last_clean":"2024-04-25T05:44:56.286927+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:44:56.286927+0000","last_undegraded":"2024-04-25T05:44:56.286927+0000","last_fullsized":"2024-04-25T05:44:56.286927+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:44:02.120258+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.120258+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.452057+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":199,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.294869+0000","last_change":"2024-04-25T05:44:28.182297+0000","last_active":"2024-04-25T05:44:56.294869+0000","last_peered":"2024-04-25T05:44:56.294869+0000","last_clean":"2024-04-25T05:44:56.294869+0000","last_became_active":"2024-04-25T05:44:28.181951+0000","last_became_peered":"2024-04-25T05:44:28.181951+0000","last_unstale":"2024-04-25T05:44:56.294869+0000","last_undegraded":"2024-04-25T05:44:56.294869+0000","last_fullsized":"2024-04-25T05:44:56.294869+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:57.544063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.544063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":586,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.285843+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:44:56.285843+0000","last_peered":"2024-04-25T05:44:56.285843+0000","last_clean":"2024-04-25T05:44:56.285843+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:44:56.285843+0000","last_undegraded":"2024-04-25T05:44:56.285843+0000","last_fullsized":"2024-04-25T05:44:56.285843+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.182548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":86,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-04-25T05:44:56.289290+0000","last_change":"2024-04-25T05:43:16.198408+0000","last_active":"2024-04-25T05:44:56.289290+0000","last_peered":"2024-04-25T05:44:56.289290+0000","last_clean":"2024-04-25T05:44:56.289290+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:44:56.289290+0000","last_undegraded":"2024-04-25T05:44:56.289290+0000","last_fullsized":"2024-04-25T05:44:56.289290+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7022837784,"num_objects":107161,"num_object_clones":0,"num_object_copies":321483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17466,"num_objects_unfound":0,"num_objects_dirty":107161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135463,"num_write_kb":7763905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23793,"num_bytes_recovered":1559298048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22578475008,"data_stored":22578462792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50607,"ondisk_log_size":50607,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":641,"num_read_kb":636,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":8366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":14104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070601,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5535508,"kb_used_data":4994824,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":88209644,"statfs":{"total":95995035648,"available":90326675456,"internally_reserved":0,"allocated":5114699776,"data_stored":5114218521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364941,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2613064,"kb_used_data":2148732,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91132088,"statfs":{"total":95995035648,"available":93319258112,"internally_reserved":0,"allocated":2200301568,"data_stored":2199832335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475462086},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6002040,"kb_used_data":5357228,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":87743112,"statfs":{"total":95995035648,"available":89848946688,"internally_reserved":0,"allocated":5485801472,"data_stored":5485282657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738540,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5620520,"kb_used_data":4903516,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":88124632,"statfs":{"total":95995035648,"available":90239623168,"internally_reserved":0,"allocated":5021200384,"data_stored":5020698747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":734198215},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2577752,"kb_used_data":2177296,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91167400,"statfs":{"total":95995035648,"available":93355417600,"internally_reserved":0,"allocated":2229551104,"data_stored":2229061642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2643394560,"data_stored":2643394560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5439492096,"data_stored":5439488024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5005049856,"data_stored":5005049856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2217152512,"data_stored":2217148440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2187722752,"data_stored":2187722752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5085663232,"data_stored":5085659160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:45:02.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:02.999 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: pgmap v579: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 764 op/s; 17466/322173 objects misplaced (5.421%); 21 MiB/s, 333 objects/s recovering 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:02.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:02 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:03.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:03.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: pgmap v579: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 764 op/s; 17466/322173 objects misplaced (5.421%); 21 MiB/s, 333 objects/s recovering 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: from='client.15783 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: pgmap v579: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 764 op/s; 17466/322173 objects misplaced (5.421%); 21 MiB/s, 333 objects/s recovering 2024-04-25T05:45:03.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:45:03.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:45:03.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:03.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:02 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:03.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:03.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:03.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:03.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:03.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:03.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:03.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:03.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:45:03.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:03.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:03.754 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:03.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:03.816 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:03.918 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:03.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:03.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:04.002 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:04.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:04.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:04.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:04.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:04.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:03 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:04.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:03 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:04.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:03 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:04.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:04.366 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:04.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.505 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:04.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:04.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:04.619 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:45:04.619 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:45:04.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.619 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:142858 9.b 9:d3e81999:::benchmark_data_smithi043_129820_object21544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:45:04.607656+0000", 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.010873404999999999, 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010897753, 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 142858 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607656+0000", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607656+0000", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607658+0000", 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.139e-06 2024-04-25T05:45:04.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607663+0000", 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4160000000000003e-06 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607678+0000", 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4712000000000001e-05 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:04.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607686+0000", 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1189999999999997e-06 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607755+0000", 2024-04-25T05:45:04.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8820999999999997e-05 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607944+0000", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018895699999999999 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,3", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.608099+0000", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015552099999999999 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.611487+0000", 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033874669999999999 2024-04-25T05:45:04.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.617731+0000", 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062444620000000001 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.618217+0000", 2024-04-25T05:45:04.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048548200000000003 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:142860 9.b 9:d8a8c635:::benchmark_data_smithi043_129820_object21546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:45:04.607967+0000", 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.010562847, 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010649149, 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:45:04.626 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 142860 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607967+0000", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607967+0000", 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:45:04.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607968+0000", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5799999999999998e-07 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607980+0000", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2437e-05 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607990+0000", 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3309999999999993e-06 2024-04-25T05:45:04.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.607997+0000", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7440000000000007e-06 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.610134+0000", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021363380000000002 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.610265+0000", 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013165299999999999 2024-04-25T05:45:04.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,3", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.610369+0000", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103935 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.611666+0000", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012970760000000001 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:45:04.618268+0000", 2024-04-25T05:45:04.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006601717 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 2 2024-04-25T05:45:04.631 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:45:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:04.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:04.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:04.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:04.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:04.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:04.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:04.934 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:45:05.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:05.037 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:05.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: pgmap v580: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14057/304824 objects misplaced (4.612%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: osdmap e340: 8 total, 8 up, 5 in 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1758376627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:05.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:04 smithi060 ceph-mon[30898]: osdmap e341: 8 total, 8 up, 5 in 2024-04-25T05:45:05.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:05.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: pgmap v580: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14057/304824 objects misplaced (4.612%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: osdmap e340: 8 total, 8 up, 5 in 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1758376627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.379 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[27010]: osdmap e341: 8 total, 8 up, 5 in 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: pgmap v580: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14057/304824 objects misplaced (4.612%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: osdmap e340: 8 total, 8 up, 5 in 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1758376627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:05.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:04 smithi043 ceph-mon[37492]: osdmap e341: 8 total, 8 up, 5 in 2024-04-25T05:45:05.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:05.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:05.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.609 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:05.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:05.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:45:05.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:45:05.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:05.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:05.904 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:05.950 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:05.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:06.005 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:06.051 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:06.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:06.158 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:06.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.235 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:06.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:06.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:06.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:06.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:06.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:06.629 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:06.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:06.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:06.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:06.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:06.906 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:06.931 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:06.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:06.951 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:06.955 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:45:06.955 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:45:06.955 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:45:06.955 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:45:06.955 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:45:06.991 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:07.032 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:07.033 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:07.060 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:07.074 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:07.120 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:07.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:07.165 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:07.166 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:07 smithi043 ceph-mon[27010]: pgmap v583: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 14057/304824 objects misplaced (4.612%); 27 MiB/s, 425 objects/s recovering 2024-04-25T05:45:07.166 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2827520353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:07.167 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:07 smithi043 ceph-mon[37492]: pgmap v583: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 14057/304824 objects misplaced (4.612%); 27 MiB/s, 425 objects/s recovering 2024-04-25T05:45:07.167 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2827520353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:07.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:07.208 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:07.213 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:07.214 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:07 smithi060 ceph-mon[30898]: pgmap v583: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 14057/304824 objects misplaced (4.612%); 27 MiB/s, 425 objects/s recovering 2024-04-25T05:45:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2827520353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:07.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:07.325 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:07.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:07.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:07.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:07.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:07.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:45:07.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:45:07.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:07.877 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:45:07.877 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:45:07.877 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T05:45:07.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:45:07.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:07.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:07.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:08.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:08.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:08.137 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:08.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:08.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.306 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:08.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:08.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:08.410 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:08.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:08.494 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:08.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:08.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:08.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:08.858 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:08.859 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:08.870 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:08.903 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:08.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:09.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:09.032 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:09.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:09.162 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:09.182 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:09 smithi043 ceph-mon[27010]: pgmap v584: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 725 op/s; 14057/304824 objects misplaced (4.612%); 20 MiB/s, 320 objects/s recovering 2024-04-25T05:45:09.182 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:09 smithi043 ceph-mon[37492]: pgmap v584: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 725 op/s; 14057/304824 objects misplaced (4.612%); 20 MiB/s, 320 objects/s recovering 2024-04-25T05:45:09.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:09 smithi060 ceph-mon[30898]: pgmap v584: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 725 op/s; 14057/304824 objects misplaced (4.612%); 20 MiB/s, 320 objects/s recovering 2024-04-25T05:45:09.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.290 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:09.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:09.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:09.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:09.549 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:09.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:09.677 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:09.949 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 342, 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:45:09.002735+0000", 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:45:09.951 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:44:27.099440+0000", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5801984, 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "noscrub", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:45:09.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 9, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.954 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "174", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "171", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.955 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.1600000858306885, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.1600000858306885, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.3600000143051147, 2024-04-25T05:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.3600000143051147, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6600000858306885, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6600000858306885, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.5, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.3300000429153442, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.3300000429153442, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.4800000190734863, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.4800000190734863, 2024-04-25T05:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.559999942779541, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.559999942779541, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.7300000190734863, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.7300000190734863, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.7200000286102295, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.7200000286102295, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.4099999666213989, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.4099999666213989, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.983 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.984 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.985 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.986 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.7300000190734863, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.7300000190734863, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.7200000286102295, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.7200000286102295, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_2", 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:43:08.943402+0000", 2024-04-25T05:45:09.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:45:09.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "300", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:45:09.989 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:45:09.990 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:45:09.991 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.9800000190734863, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.9800000190734863, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.62999999523162842, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.8799999952316284, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:45:09.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 304, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:09.993 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:45:09.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 325, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:45:09.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.000 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 273, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:45:10.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:45:10.003 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.004 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 325, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:45:10.007 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0.610748291015625, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 327, 2024-04-25T05:45:10.008 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:45:10.009 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 296, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.012 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.013 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:45:10.014 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.015 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 327, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:45:10.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.018 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:45:10.019 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 341, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.020 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:45:10.022 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:30.326470+0000", 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:45:03.235047+0000", 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:29.214785+0000", 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:44.224899+0000", 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:31.357330+0000", 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:50.281423+0000", 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:14.831099+0000", 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:45:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:44:56.133753+0000", 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:45:10.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 7 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:45:10.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 7, 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 3, 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.030 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "9.0", 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "9.6", 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:45:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "9.8", 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "9.b", 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:45:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 0 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:45:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:10.035 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:45:10.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:10.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:45:10.137 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:10.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:10.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:10.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:10.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:10.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:45:10.329 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: osdmap e342: 8 total, 8 up, 5 in 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2273040766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: osdmap e343: 8 total, 8 up, 5 in 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4145960230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: osdmap e342: 8 total, 8 up, 5 in 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2273040766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: osdmap e343: 8 total, 8 up, 5 in 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4145960230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:10.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:10.423 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: osdmap e342: 8 total, 8 up, 5 in 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2273040766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: osdmap e343: 8 total, 8 up, 5 in 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.451 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4145960230' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.452 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:10.452 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:10.452 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:10.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:10.547 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:10.649 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:10.672 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:10.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:10.799 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:10.886 INFO:tasks.thrashosds.thrasher:j is {'epoch': 342, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:45:09.002735+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:44:27.099440+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '171', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1600000858306885, 'score_stable': 2.1600000858306885, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.3600000143051147, 'raw_score_stable': 1.3600000143051147, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.7300000190734863, 'score_stable': 2.7300000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.7200000286102295, 'raw_score_stable': 1.7200000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.409999966621399, 'raw_score_stable': 1.409999966621399, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.7300000190734863, 'score_stable': 2.7300000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.7200000286102295, 'raw_score_stable': 1.7200000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-25T05:43:08.943402+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '300', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9800000190734863, 'score_stable': 2.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 304, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 273, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.610748291015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 327, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 296, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 327, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:44:30.326470+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:45:03.235047+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:44:29.214785+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:44:44.224899+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:44:31.357330+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:44:50.281423+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:44:14.831099+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:44:56.133753+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '6.3', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '6.9', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.0', 'osds': [1, 6, 0]}, {'pgid': '9.6', 'osds': [1, 6, 0]}, {'pgid': '9.8', 'osds': [3, 7, 0]}, {'pgid': '9.b', 'osds': [6, 1, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:45:10.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:45:10.891 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:10.891 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:10.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:45:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:10.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:10.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:11.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:11.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: pgmap v586: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi060 to 3249M 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: Adjusting osd_memory_target on smithi043 to 3249M 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:45:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:45:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:11.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[27010]: osdmap e344: 8 total, 8 up, 5 in 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: pgmap v586: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi060 to 3249M 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: Adjusting osd_memory_target on smithi043 to 3249M 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:11.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:11.146 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:11 smithi043 ceph-mon[37492]: osdmap e344: 8 total, 8 up, 5 in 2024-04-25T05:45:11.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:11.236 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:11.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:11.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:11.462 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: pgmap v586: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:11.462 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi060 to 3249M 2024-04-25T05:45:11.462 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.462 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.462 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: Adjusting osd_memory_target on smithi043 to 3249M 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:11.463 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-mon[30898]: osdmap e344: 8 total, 8 up, 5 in 2024-04-25T05:45:11.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:11.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:11.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:11.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:11.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:11.765 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:11.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:45:11.820 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:11.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:11.890 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:11.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:11.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:11.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:45:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=cleanup t=2024-04-25T05:45:11.77056258Z level=info msg="Completed cleanup jobs" duration=2.505376ms 2024-04-25T05:45:12.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.064 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:12.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:12.154 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:12.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:12.200 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:12.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:12.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:12.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:12.468 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:12.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.597 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:12.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:12.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:12.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:12.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:12.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:12.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:12.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:13.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:13.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:13 smithi043 ceph-mon[27010]: pgmap v589: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:13.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:13 smithi043 ceph-mon[37492]: pgmap v589: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:13.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:13.154 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:13.200 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:13.246 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:13.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:13.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:13.290 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:13.319 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:13 smithi060 ceph-mon[30898]: pgmap v589: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 10517/285711 objects misplaced (3.681%); 29 MiB/s, 457 objects/s recovering 2024-04-25T05:45:13.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:13.384 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:13.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:13.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:13.425 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:13.466 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:13.479 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:13.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.542 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:13.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:13.554 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:45:13.557 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:45:13.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:13.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:13.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:45:13.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:13.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:13.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:13.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:14.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2757448801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:14.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:14.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:14.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:14.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:14.233 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:14.298 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2757448801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:14.299 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:14.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:14.364 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:14.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2757448801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:14.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:14.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:14.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:14.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:14.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:14.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:14.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:14.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:14.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:14.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:15.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:15.050 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:15.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:15.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:15.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.320 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:15.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:15.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[27010]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:15.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[27010]: pgmap v590: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 6535/272664 objects misplaced (2.397%); 67 MiB/s, 1.07k objects/s recovering 2024-04-25T05:45:15.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[27010]: osdmap e345: 8 total, 8 up, 5 in 2024-04-25T05:45:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[37492]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[37492]: pgmap v590: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 6535/272664 objects misplaced (2.397%); 67 MiB/s, 1.07k objects/s recovering 2024-04-25T05:45:15.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:15 smithi043 ceph-mon[37492]: osdmap e345: 8 total, 8 up, 5 in 2024-04-25T05:45:15.403 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:15 smithi060 ceph-mon[30898]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:15.403 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:15 smithi060 ceph-mon[30898]: pgmap v590: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 6535/272664 objects misplaced (2.397%); 67 MiB/s, 1.07k objects/s recovering 2024-04-25T05:45:15.403 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:15 smithi060 ceph-mon[30898]: osdmap e345: 8 total, 8 up, 5 in 2024-04-25T05:45:15.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:15.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:15.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.580 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:15.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:15.711 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:15.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:15.817 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-04-25T05:45:11.893816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6226909747,"num_objects":95237,"num_object_clones":0,"num_object_copies":285711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10517,"num_objects_unfound":0,"num_objects_dirty":95237,"num_whiteouts":0,"num_read":2066,"num_read_kb":1786,"num_write":148531,"num_write_kb":7763950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29588,"num_bytes_recovered":1907044130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53032,"ondisk_log_size":53032,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":20505208,"kb_used_data":17669840,"kb_used_omap":7,"kb_used_meta":2835256,"kb_avail":448220552,"statfs":{"total":479975178240,"available":458977845248,"internally_reserved":0,"allocated":18093916160,"data_stored":18091377498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":2903302369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,12,5,6,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-417529856,"num_objects":-6371,"num_object_clones":0,"num_object_copies":-19113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3540,"num_objects_unfound":0,"num_objects_dirty":-6371,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6371,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2748,"num_bytes_recovered":180092928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002076"},"pg_stats":[{"pgid":"9.1","version":"341'9226","reported_seq":9325,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.663365+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:45:08.663365+0000","last_peered":"2024-04-25T05:45:08.663365+0000","last_clean":"2024-04-25T05:45:08.663365+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:45:08.663365+0000","last_undegraded":"2024-04-25T05:45:08.663365+0000","last_fullsized":"2024-04-25T05:45:08.663365+0000","mapping_epoch":296,"log_start":"322'6087","ondisk_log_start":"322'6087","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388104192,"num_objects":5922,"num_object_clones":0,"num_object_copies":17766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":306,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.939111+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:45:04.939111+0000","last_peered":"2024-04-25T05:45:04.939111+0000","last_clean":"2024-04-25T05:45:04.939111+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:45:04.939111+0000","last_undegraded":"2024-04-25T05:45:04.939111+0000","last_fullsized":"2024-04-25T05:45:04.939111+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":596,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926124+0000","last_change":"2024-04-25T05:44:28.173941+0000","last_active":"2024-04-25T05:45:04.926124+0000","last_peered":"2024-04-25T05:45:04.926124+0000","last_clean":"2024-04-25T05:45:04.926124+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:04.926124+0000","last_undegraded":"2024-04-25T05:45:04.926124+0000","last_fullsized":"2024-04-25T05:45:04.926124+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"325'12","reported_seq":393,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929527+0000","last_change":"2024-04-25T05:44:28.198744+0000","last_active":"2024-04-25T05:45:04.929527+0000","last_peered":"2024-04-25T05:45:04.929527+0000","last_clean":"2024-04-25T05:45:04.929527+0000","last_became_active":"2024-04-25T05:44:28.171810+0000","last_became_peered":"2024-04-25T05:44:28.171810+0000","last_unstale":"2024-04-25T05:45:04.929527+0000","last_undegraded":"2024-04-25T05:45:04.929527+0000","last_fullsized":"2024-04-25T05:45:04.929527+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":269,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.931564+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:45:04.931564+0000","last_peered":"2024-04-25T05:45:04.931564+0000","last_clean":"2024-04-25T05:45:04.931564+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:45:04.931564+0000","last_undegraded":"2024-04-25T05:45:04.931564+0000","last_fullsized":"2024-04-25T05:45:04.931564+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":903,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927014+0000","last_change":"2024-04-25T05:43:19.210320+0000","last_active":"2024-04-25T05:45:04.927014+0000","last_peered":"2024-04-25T05:45:04.927014+0000","last_clean":"2024-04-25T05:45:04.927014+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:45:04.927014+0000","last_undegraded":"2024-04-25T05:45:04.927014+0000","last_fullsized":"2024-04-25T05:45:04.927014+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"341'9229","reported_seq":9337,"reported_epoch":341,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:45:08.736166+0000","last_change":"2024-04-25T05:44:59.224944+0000","last_active":"2024-04-25T05:45:08.736166+0000","last_peered":"2024-04-25T05:45:08.736166+0000","last_clean":"2024-04-25T05:44:27.095992+0000","last_became_active":"2024-04-25T05:44:29.189111+0000","last_became_peered":"2024-04-25T05:44:29.189111+0000","last_unstale":"2024-04-25T05:45:08.736166+0000","last_undegraded":"2024-04-25T05:45:08.736166+0000","last_fullsized":"2024-04-25T05:45:08.736166+0000","mapping_epoch":325,"log_start":"323'6287","ondisk_log_start":"323'6287","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":2815,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5829,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9229,"num_write_kb":481856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":5829}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":314,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923453+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:45:04.923453+0000","last_peered":"2024-04-25T05:45:04.923453+0000","last_clean":"2024-04-25T05:45:04.923453+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:45:04.923453+0000","last_undegraded":"2024-04-25T05:45:04.923453+0000","last_fullsized":"2024-04-25T05:45:04.923453+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":310,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924788+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:45:04.924788+0000","last_peered":"2024-04-25T05:45:04.924788+0000","last_clean":"2024-04-25T05:45:04.924788+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:45:04.924788+0000","last_undegraded":"2024-04-25T05:45:04.924788+0000","last_fullsized":"2024-04-25T05:45:04.924788+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.922243+0000","last_change":"2024-04-25T05:44:28.168158+0000","last_active":"2024-04-25T05:45:04.922243+0000","last_peered":"2024-04-25T05:45:04.922243+0000","last_clean":"2024-04-25T05:45:04.922243+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:45:04.922243+0000","last_undegraded":"2024-04-25T05:45:04.922243+0000","last_fullsized":"2024-04-25T05:45:04.922243+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:44.537120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:44.537120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":221,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.935958+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:45:04.935958+0000","last_peered":"2024-04-25T05:45:04.935958+0000","last_clean":"2024-04-25T05:45:04.935958+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:45:04.935958+0000","last_undegraded":"2024-04-25T05:45:04.935958+0000","last_fullsized":"2024-04-25T05:45:04.935958+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.600058+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":502,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924630+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:45:04.924630+0000","last_peered":"2024-04-25T05:45:04.924630+0000","last_clean":"2024-04-25T05:45:04.924630+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:45:04.924630+0000","last_undegraded":"2024-04-25T05:45:04.924630+0000","last_fullsized":"2024-04-25T05:45:04.924630+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"341'9077","reported_seq":9176,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:07.916356+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:45:07.916356+0000","last_peered":"2024-04-25T05:45:07.916356+0000","last_clean":"2024-04-25T05:45:07.916356+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:45:07.916356+0000","last_undegraded":"2024-04-25T05:45:07.916356+0000","last_fullsized":"2024-04-25T05:45:07.916356+0000","mapping_epoch":296,"log_start":"322'5777","ondisk_log_start":"322'5777","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390004736,"num_objects":5951,"num_object_clones":0,"num_object_copies":17853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":615,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927893+0000","last_change":"2024-04-25T05:42:53.501053+0000","last_active":"2024-04-25T05:45:04.927893+0000","last_peered":"2024-04-25T05:45:04.927893+0000","last_clean":"2024-04-25T05:45:04.927893+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:45:04.927893+0000","last_undegraded":"2024-04-25T05:45:04.927893+0000","last_fullsized":"2024-04-25T05:45:04.927893+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-04-25T05:42:53.500776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.500776+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":609,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.918688+0000","last_change":"2024-04-25T05:44:46.503302+0000","last_active":"2024-04-25T05:45:04.918688+0000","last_peered":"2024-04-25T05:45:04.918688+0000","last_clean":"2024-04-25T05:45:04.918688+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:45:04.918688+0000","last_undegraded":"2024-04-25T05:45:04.918688+0000","last_fullsized":"2024-04-25T05:45:04.918688+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:46.503173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:46.503173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:56.738128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":404,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.921420+0000","last_change":"2024-04-25T05:44:57.457953+0000","last_active":"2024-04-25T05:45:04.921420+0000","last_peered":"2024-04-25T05:45:04.921420+0000","last_clean":"2024-04-25T05:45:04.921420+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:45:04.921420+0000","last_undegraded":"2024-04-25T05:45:04.921420+0000","last_fullsized":"2024-04-25T05:45:04.921420+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:44:57.457842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:18.204723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":584,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.939627+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:45:04.939627+0000","last_peered":"2024-04-25T05:45:04.939627+0000","last_clean":"2024-04-25T05:45:04.939627+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:45:04.939627+0000","last_undegraded":"2024-04-25T05:45:04.939627+0000","last_fullsized":"2024-04-25T05:45:04.939627+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"341'9407","reported_seq":9506,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.683922+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:45:08.683922+0000","last_peered":"2024-04-25T05:45:08.683922+0000","last_clean":"2024-04-25T05:45:08.683922+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:45:08.683922+0000","last_undegraded":"2024-04-25T05:45:08.683922+0000","last_fullsized":"2024-04-25T05:45:08.683922+0000","mapping_epoch":296,"log_start":"322'6281","ondisk_log_start":"322'6281","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3126,"log_dups_size":0,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393936896,"num_objects":6011,"num_object_clones":0,"num_object_copies":18033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":632,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923042+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:45:04.923042+0000","last_peered":"2024-04-25T05:45:04.923042+0000","last_clean":"2024-04-25T05:45:04.923042+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:45:04.923042+0000","last_undegraded":"2024-04-25T05:45:04.923042+0000","last_fullsized":"2024-04-25T05:45:04.923042+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":587,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925109+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:45:04.925109+0000","last_peered":"2024-04-25T05:45:04.925109+0000","last_clean":"2024-04-25T05:45:04.925109+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:45:04.925109+0000","last_undegraded":"2024-04-25T05:45:04.925109+0000","last_fullsized":"2024-04-25T05:45:04.925109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.125277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"325'13","reported_seq":120,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932127+0000","last_change":"2024-04-25T05:44:28.188020+0000","last_active":"2024-04-25T05:45:04.932127+0000","last_peered":"2024-04-25T05:45:04.932127+0000","last_clean":"2024-04-25T05:45:04.932127+0000","last_became_active":"2024-04-25T05:44:28.178692+0000","last_became_peered":"2024-04-25T05:44:28.178692+0000","last_unstale":"2024-04-25T05:45:04.932127+0000","last_undegraded":"2024-04-25T05:45:04.932127+0000","last_fullsized":"2024-04-25T05:45:04.932127+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'11","last_scrub_stamp":"2024-04-25T05:41:38.552614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:38.552614+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":448,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923256+0000","last_change":"2024-04-25T05:44:28.186072+0000","last_active":"2024-04-25T05:45:04.923256+0000","last_peered":"2024-04-25T05:45:04.923256+0000","last_clean":"2024-04-25T05:45:04.923256+0000","last_became_active":"2024-04-25T05:44:28.177729+0000","last_became_peered":"2024-04-25T05:44:28.177729+0000","last_unstale":"2024-04-25T05:45:04.923256+0000","last_undegraded":"2024-04-25T05:45:04.923256+0000","last_fullsized":"2024-04-25T05:45:04.923256+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:50.460004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:50.460004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":578,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924536+0000","last_change":"2024-04-25T05:44:30.932605+0000","last_active":"2024-04-25T05:45:04.924536+0000","last_peered":"2024-04-25T05:45:04.924536+0000","last_clean":"2024-04-25T05:45:04.924536+0000","last_became_active":"2024-04-25T05:44:30.932221+0000","last_became_peered":"2024-04-25T05:44:30.932221+0000","last_unstale":"2024-04-25T05:45:04.924536+0000","last_undegraded":"2024-04-25T05:45:04.924536+0000","last_fullsized":"2024-04-25T05:45:04.924536+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.344182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"341'9217","reported_seq":9323,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:07.916474+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:45:07.916474+0000","last_peered":"2024-04-25T05:45:07.916474+0000","last_clean":"2024-04-25T05:45:07.916474+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:45:07.916474+0000","last_undegraded":"2024-04-25T05:45:07.916474+0000","last_fullsized":"2024-04-25T05:45:07.916474+0000","mapping_epoch":301,"log_start":"322'5953","ondisk_log_start":"322'5953","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392953880,"num_objects":5997,"num_object_clones":0,"num_object_copies":17991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5997,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9217,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":587,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.918590+0000","last_change":"2024-04-25T05:43:58.091032+0000","last_active":"2024-04-25T05:45:04.918590+0000","last_peered":"2024-04-25T05:45:04.918590+0000","last_clean":"2024-04-25T05:45:04.918590+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:45:04.918590+0000","last_undegraded":"2024-04-25T05:45:04.918590+0000","last_fullsized":"2024-04-25T05:45:04.918590+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:58.090855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:58.090855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.355724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":741,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925191+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:45:04.925191+0000","last_peered":"2024-04-25T05:45:04.925191+0000","last_clean":"2024-04-25T05:45:04.925191+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:45:04.925191+0000","last_undegraded":"2024-04-25T05:45:04.925191+0000","last_fullsized":"2024-04-25T05:45:04.925191+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.991166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.936290+0000","last_change":"2024-04-25T05:44:28.180828+0000","last_active":"2024-04-25T05:45:04.936290+0000","last_peered":"2024-04-25T05:45:04.936290+0000","last_clean":"2024-04-25T05:45:04.936290+0000","last_became_active":"2024-04-25T05:44:28.178969+0000","last_became_peered":"2024-04-25T05:44:28.178969+0000","last_unstale":"2024-04-25T05:45:04.936290+0000","last_undegraded":"2024-04-25T05:45:04.936290+0000","last_fullsized":"2024-04-25T05:45:04.936290+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:15.041538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.041538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.837433+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":263,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933789+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:45:04.933789+0000","last_peered":"2024-04-25T05:45:04.933789+0000","last_clean":"2024-04-25T05:45:04.933789+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:45:04.933789+0000","last_undegraded":"2024-04-25T05:45:04.933789+0000","last_fullsized":"2024-04-25T05:45:04.933789+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:12.232569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":807,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.922232+0000","last_change":"2024-04-25T05:44:30.916141+0000","last_active":"2024-04-25T05:45:04.922232+0000","last_peered":"2024-04-25T05:45:04.922232+0000","last_clean":"2024-04-25T05:45:04.922232+0000","last_became_active":"2024-04-25T05:44:30.915718+0000","last_became_peered":"2024-04-25T05:44:30.915718+0000","last_unstale":"2024-04-25T05:45:04.922232+0000","last_undegraded":"2024-04-25T05:45:04.922232+0000","last_fullsized":"2024-04-25T05:45:04.922232+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.223066+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.958082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"341'9393","reported_seq":9492,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.656820+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:45:08.656820+0000","last_peered":"2024-04-25T05:45:08.656820+0000","last_clean":"2024-04-25T05:45:08.656820+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:45:08.656820+0000","last_undegraded":"2024-04-25T05:45:08.656820+0000","last_fullsized":"2024-04-25T05:45:08.656820+0000","mapping_epoch":296,"log_start":"322'6216","ondisk_log_start":"322'6216","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391577600,"num_objects":5975,"num_object_clones":0,"num_object_copies":17925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9393,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":713,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925344+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:45:04.925344+0000","last_peered":"2024-04-25T05:45:04.925344+0000","last_clean":"2024-04-25T05:45:04.925344+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:45:04.925344+0000","last_undegraded":"2024-04-25T05:45:04.925344+0000","last_fullsized":"2024-04-25T05:45:04.925344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":530,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924754+0000","last_change":"2024-04-25T05:44:28.134148+0000","last_active":"2024-04-25T05:45:04.924754+0000","last_peered":"2024-04-25T05:45:04.924754+0000","last_clean":"2024-04-25T05:45:04.924754+0000","last_became_active":"2024-04-25T05:44:28.124913+0000","last_became_peered":"2024-04-25T05:44:28.124913+0000","last_unstale":"2024-04-25T05:45:04.924754+0000","last_undegraded":"2024-04-25T05:45:04.924754+0000","last_fullsized":"2024-04-25T05:45:04.924754+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'20","last_scrub_stamp":"2024-04-25T05:43:24.628430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.628430+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":291,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.931814+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:45:04.931814+0000","last_peered":"2024-04-25T05:45:04.931814+0000","last_clean":"2024-04-25T05:45:04.931814+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:45:04.931814+0000","last_undegraded":"2024-04-25T05:45:04.931814+0000","last_fullsized":"2024-04-25T05:45:04.931814+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":220,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929111+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:45:04.929111+0000","last_peered":"2024-04-25T05:45:04.929111+0000","last_clean":"2024-04-25T05:45:04.929111+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:45:04.929111+0000","last_undegraded":"2024-04-25T05:45:04.929111+0000","last_fullsized":"2024-04-25T05:45:04.929111+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":767,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933676+0000","last_change":"2024-04-25T05:43:19.215636+0000","last_active":"2024-04-25T05:45:04.933676+0000","last_peered":"2024-04-25T05:45:04.933676+0000","last_clean":"2024-04-25T05:45:04.933676+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:45:04.933676+0000","last_undegraded":"2024-04-25T05:45:04.933676+0000","last_fullsized":"2024-04-25T05:45:04.933676+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"341'9243","reported_seq":9333,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.733003+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:45:08.733003+0000","last_peered":"2024-04-25T05:45:08.733003+0000","last_clean":"2024-04-25T05:45:08.733003+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:45:08.733003+0000","last_undegraded":"2024-04-25T05:45:08.733003+0000","last_fullsized":"2024-04-25T05:45:08.733003+0000","mapping_epoch":301,"log_start":"323'6338","ondisk_log_start":"323'6338","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2905,"log_dups_size":2815,"ondisk_log_size":2905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391839744,"num_objects":5979,"num_object_clones":0,"num_object_copies":17937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9243,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":304,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925074+0000","last_change":"2024-04-25T05:44:28.202828+0000","last_active":"2024-04-25T05:45:04.925074+0000","last_peered":"2024-04-25T05:45:04.925074+0000","last_clean":"2024-04-25T05:45:04.925074+0000","last_became_active":"2024-04-25T05:44:28.182043+0000","last_became_peered":"2024-04-25T05:44:28.182043+0000","last_unstale":"2024-04-25T05:45:04.925074+0000","last_undegraded":"2024-04-25T05:45:04.925074+0000","last_fullsized":"2024-04-25T05:45:04.925074+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.008065+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":289,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.945121+0000","last_change":"2024-04-25T05:44:28.217831+0000","last_active":"2024-04-25T05:45:04.945121+0000","last_peered":"2024-04-25T05:45:04.945121+0000","last_clean":"2024-04-25T05:45:04.945121+0000","last_became_active":"2024-04-25T05:44:28.170458+0000","last_became_peered":"2024-04-25T05:44:28.170458+0000","last_unstale":"2024-04-25T05:45:04.945121+0000","last_undegraded":"2024-04-25T05:45:04.945121+0000","last_fullsized":"2024-04-25T05:45:04.945121+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":473,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932280+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:45:04.932280+0000","last_peered":"2024-04-25T05:45:04.932280+0000","last_clean":"2024-04-25T05:45:04.932280+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:45:04.932280+0000","last_undegraded":"2024-04-25T05:45:04.932280+0000","last_fullsized":"2024-04-25T05:45:04.932280+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":205,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924784+0000","last_change":"2024-04-25T05:44:28.168971+0000","last_active":"2024-04-25T05:45:04.924784+0000","last_peered":"2024-04-25T05:45:04.924784+0000","last_clean":"2024-04-25T05:45:04.924784+0000","last_became_active":"2024-04-25T05:44:28.168333+0000","last_became_peered":"2024-04-25T05:44:28.168333+0000","last_unstale":"2024-04-25T05:45:04.924784+0000","last_undegraded":"2024-04-25T05:45:04.924784+0000","last_fullsized":"2024-04-25T05:45:04.924784+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:14.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:14.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":765,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926550+0000","last_change":"2024-04-25T05:43:28.638040+0000","last_active":"2024-04-25T05:45:04.926550+0000","last_peered":"2024-04-25T05:45:04.926550+0000","last_clean":"2024-04-25T05:45:04.926550+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:45:04.926550+0000","last_undegraded":"2024-04-25T05:45:04.926550+0000","last_fullsized":"2024-04-25T05:45:04.926550+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:28.637882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:28.637882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"341'9113","reported_seq":9212,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:07.919818+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:45:07.919818+0000","last_peered":"2024-04-25T05:45:07.919818+0000","last_clean":"2024-04-25T05:45:07.919818+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:45:07.919818+0000","last_undegraded":"2024-04-25T05:45:07.919818+0000","last_fullsized":"2024-04-25T05:45:07.919818+0000","mapping_epoch":296,"log_start":"322'5830","ondisk_log_start":"322'5830","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3283,"log_dups_size":0,"ondisk_log_size":3283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386727936,"num_objects":5901,"num_object_clones":0,"num_object_copies":17703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9113,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924890+0000","last_change":"2024-04-25T05:44:28.125054+0000","last_active":"2024-04-25T05:45:04.924890+0000","last_peered":"2024-04-25T05:45:04.924890+0000","last_clean":"2024-04-25T05:45:04.924890+0000","last_became_active":"2024-04-25T05:44:28.124674+0000","last_became_peered":"2024-04-25T05:44:28.124674+0000","last_unstale":"2024-04-25T05:45:04.924890+0000","last_undegraded":"2024-04-25T05:45:04.924890+0000","last_fullsized":"2024-04-25T05:45:04.924890+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":317,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929439+0000","last_change":"2024-04-25T05:44:28.198134+0000","last_active":"2024-04-25T05:45:04.929439+0000","last_peered":"2024-04-25T05:45:04.929439+0000","last_clean":"2024-04-25T05:45:04.929439+0000","last_became_active":"2024-04-25T05:44:28.168410+0000","last_became_peered":"2024-04-25T05:44:28.168410+0000","last_unstale":"2024-04-25T05:45:04.929439+0000","last_undegraded":"2024-04-25T05:45:04.929439+0000","last_fullsized":"2024-04-25T05:45:04.929439+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":613,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920952+0000","last_change":"2024-04-25T05:44:28.167276+0000","last_active":"2024-04-25T05:45:04.920952+0000","last_peered":"2024-04-25T05:45:04.920952+0000","last_clean":"2024-04-25T05:45:04.920952+0000","last_became_active":"2024-04-25T05:44:28.166968+0000","last_became_peered":"2024-04-25T05:44:28.166968+0000","last_unstale":"2024-04-25T05:45:04.920952+0000","last_undegraded":"2024-04-25T05:45:04.920952+0000","last_fullsized":"2024-04-25T05:45:04.920952+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":593,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920834+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:45:04.920834+0000","last_peered":"2024-04-25T05:45:04.920834+0000","last_clean":"2024-04-25T05:45:04.920834+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:45:04.920834+0000","last_undegraded":"2024-04-25T05:45:04.920834+0000","last_fullsized":"2024-04-25T05:45:04.920834+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":791,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927668+0000","last_change":"2024-04-25T05:44:30.918290+0000","last_active":"2024-04-25T05:45:04.927668+0000","last_peered":"2024-04-25T05:45:04.927668+0000","last_clean":"2024-04-25T05:45:04.927668+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:45:04.927668+0000","last_undegraded":"2024-04-25T05:45:04.927668+0000","last_fullsized":"2024-04-25T05:45:04.927668+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:20.780175+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.780175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:42.506221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"341'9098","reported_seq":21450,"reported_epoch":341,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:45:07.922701+0000","last_change":"2024-04-25T05:44:29.210942+0000","last_active":"2024-04-25T05:45:07.922701+0000","last_peered":"2024-04-25T05:45:07.922701+0000","last_clean":"2024-04-25T05:44:27.036002+0000","last_became_active":"2024-04-25T05:44:29.183071+0000","last_became_peered":"2024-04-25T05:44:29.183071+0000","last_unstale":"2024-04-25T05:45:07.922701+0000","last_undegraded":"2024-04-25T05:45:07.922701+0000","last_fullsized":"2024-04-25T05:45:07.922701+0000","mapping_epoch":325,"log_start":"321'5511","ondisk_log_start":"321'5511","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":17706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":627,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9098,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6121,"num_bytes_recovered":401145856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":5902}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":461,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926971+0000","last_change":"2024-04-25T05:44:28.180356+0000","last_active":"2024-04-25T05:45:04.926971+0000","last_peered":"2024-04-25T05:45:04.926971+0000","last_clean":"2024-04-25T05:45:04.926971+0000","last_became_active":"2024-04-25T05:44:28.180059+0000","last_became_peered":"2024-04-25T05:44:28.180059+0000","last_unstale":"2024-04-25T05:45:04.926971+0000","last_undegraded":"2024-04-25T05:45:04.926971+0000","last_fullsized":"2024-04-25T05:45:04.926971+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.755552+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":135,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932151+0000","last_change":"2024-04-25T05:44:28.234082+0000","last_active":"2024-04-25T05:45:04.932151+0000","last_peered":"2024-04-25T05:45:04.932151+0000","last_clean":"2024-04-25T05:45:04.932151+0000","last_became_active":"2024-04-25T05:44:28.177715+0000","last_became_peered":"2024-04-25T05:44:28.177715+0000","last_unstale":"2024-04-25T05:45:04.932151+0000","last_undegraded":"2024-04-25T05:45:04.932151+0000","last_fullsized":"2024-04-25T05:45:04.932151+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:44.249745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1297,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:06.244312+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:45:06.244312+0000","last_peered":"2024-04-25T05:45:06.244312+0000","last_clean":"2024-04-25T05:45:06.244312+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:45:06.244312+0000","last_undegraded":"2024-04-25T05:45:06.244312+0000","last_fullsized":"2024-04-25T05:45:06.244312+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:17.162472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":651,"num_read_kb":646,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.918785+0000","last_change":"2024-04-25T05:44:28.172816+0000","last_active":"2024-04-25T05:45:04.918785+0000","last_peered":"2024-04-25T05:45:04.918785+0000","last_clean":"2024-04-25T05:45:04.918785+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:04.918785+0000","last_undegraded":"2024-04-25T05:45:04.918785+0000","last_fullsized":"2024-04-25T05:45:04.918785+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.421093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.421093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:18.272940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":581,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920744+0000","last_change":"2024-04-25T05:43:16.189190+0000","last_active":"2024-04-25T05:45:04.920744+0000","last_peered":"2024-04-25T05:45:04.920744+0000","last_clean":"2024-04-25T05:45:04.920744+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:45:04.920744+0000","last_undegraded":"2024-04-25T05:45:04.920744+0000","last_fullsized":"2024-04-25T05:45:04.920744+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":769,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924495+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:45:04.924495+0000","last_peered":"2024-04-25T05:45:04.924495+0000","last_clean":"2024-04-25T05:45:04.924495+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:45:04.924495+0000","last_undegraded":"2024-04-25T05:45:04.924495+0000","last_fullsized":"2024-04-25T05:45:04.924495+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"341'9235","reported_seq":17746,"reported_epoch":341,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:45:08.682365+0000","last_change":"2024-04-25T05:44:29.184637+0000","last_active":"2024-04-25T05:45:08.682365+0000","last_peered":"2024-04-25T05:45:08.682365+0000","last_clean":"2024-04-25T05:44:27.101546+0000","last_became_active":"2024-04-25T05:44:29.168753+0000","last_became_peered":"2024-04-25T05:44:29.168753+0000","last_unstale":"2024-04-25T05:45:08.682365+0000","last_undegraded":"2024-04-25T05:45:08.682365+0000","last_fullsized":"2024-04-25T05:45:08.682365+0000","mapping_epoch":325,"log_start":"322'6073","ondisk_log_start":"322'6073","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386990080,"num_objects":5905,"num_object_clones":0,"num_object_copies":17715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2327,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9235,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4200,"num_bytes_recovered":275251200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":5905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928102+0000","last_change":"2024-04-25T05:44:28.178984+0000","last_active":"2024-04-25T05:45:04.928102+0000","last_peered":"2024-04-25T05:45:04.928102+0000","last_clean":"2024-04-25T05:45:04.928102+0000","last_became_active":"2024-04-25T05:44:28.178614+0000","last_became_peered":"2024-04-25T05:44:28.178614+0000","last_unstale":"2024-04-25T05:45:04.928102+0000","last_undegraded":"2024-04-25T05:45:04.928102+0000","last_fullsized":"2024-04-25T05:45:04.928102+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:23.945519+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":405,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925932+0000","last_change":"2024-04-25T05:43:22.635428+0000","last_active":"2024-04-25T05:45:04.925932+0000","last_peered":"2024-04-25T05:45:04.925932+0000","last_clean":"2024-04-25T05:45:04.925932+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:45:04.925932+0000","last_undegraded":"2024-04-25T05:45:04.925932+0000","last_fullsized":"2024-04-25T05:45:04.925932+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:43:22.635290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.635290+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":597,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933469+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:45:04.933469+0000","last_peered":"2024-04-25T05:45:04.933469+0000","last_clean":"2024-04-25T05:45:04.933469+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:45:04.933469+0000","last_undegraded":"2024-04-25T05:45:04.933469+0000","last_fullsized":"2024-04-25T05:45:04.933469+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":904,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925486+0000","last_change":"2024-04-25T05:44:30.948007+0000","last_active":"2024-04-25T05:45:04.925486+0000","last_peered":"2024-04-25T05:45:04.925486+0000","last_clean":"2024-04-25T05:45:04.925486+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:45:04.925486+0000","last_undegraded":"2024-04-25T05:45:04.925486+0000","last_fullsized":"2024-04-25T05:45:04.925486+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":90,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.936489+0000","last_change":"2024-04-25T05:43:16.188274+0000","last_active":"2024-04-25T05:45:04.936489+0000","last_peered":"2024-04-25T05:45:04.936489+0000","last_clean":"2024-04-25T05:45:04.936489+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:45:04.936489+0000","last_undegraded":"2024-04-25T05:45:04.936489+0000","last_fullsized":"2024-04-25T05:45:04.936489+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"341'9420","reported_seq":9510,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.742208+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:45:08.742208+0000","last_peered":"2024-04-25T05:45:08.742208+0000","last_clean":"2024-04-25T05:45:08.742208+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:45:08.742208+0000","last_undegraded":"2024-04-25T05:45:08.742208+0000","last_fullsized":"2024-04-25T05:45:08.742208+0000","mapping_epoch":301,"log_start":"323'6470","ondisk_log_start":"323'6470","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":2815,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395444224,"num_objects":6034,"num_object_clones":0,"num_object_copies":18102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9420,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":100,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929347+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:45:04.929347+0000","last_peered":"2024-04-25T05:45:04.929347+0000","last_clean":"2024-04-25T05:45:04.929347+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:45:04.929347+0000","last_undegraded":"2024-04-25T05:45:04.929347+0000","last_fullsized":"2024-04-25T05:45:04.929347+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":714,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.922941+0000","last_change":"2024-04-25T05:44:28.221414+0000","last_active":"2024-04-25T05:45:04.922941+0000","last_peered":"2024-04-25T05:45:04.922941+0000","last_clean":"2024-04-25T05:45:04.922941+0000","last_became_active":"2024-04-25T05:44:28.179973+0000","last_became_peered":"2024-04-25T05:44:28.179973+0000","last_unstale":"2024-04-25T05:45:04.922941+0000","last_undegraded":"2024-04-25T05:45:04.922941+0000","last_fullsized":"2024-04-25T05:45:04.922941+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":607,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932777+0000","last_change":"2024-04-25T05:44:28.186329+0000","last_active":"2024-04-25T05:45:04.932777+0000","last_peered":"2024-04-25T05:45:04.932777+0000","last_clean":"2024-04-25T05:45:04.932777+0000","last_became_active":"2024-04-25T05:44:28.184251+0000","last_became_peered":"2024-04-25T05:44:28.184251+0000","last_unstale":"2024-04-25T05:45:04.932777+0000","last_undegraded":"2024-04-25T05:45:04.932777+0000","last_fullsized":"2024-04-25T05:45:04.932777+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":625,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924425+0000","last_change":"2024-04-25T05:44:28.183966+0000","last_active":"2024-04-25T05:45:04.924425+0000","last_peered":"2024-04-25T05:45:04.924425+0000","last_clean":"2024-04-25T05:45:04.924425+0000","last_became_active":"2024-04-25T05:44:28.183528+0000","last_became_peered":"2024-04-25T05:44:28.183528+0000","last_unstale":"2024-04-25T05:45:04.924425+0000","last_undegraded":"2024-04-25T05:45:04.924425+0000","last_fullsized":"2024-04-25T05:45:04.924425+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.104924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":595,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929885+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:45:04.929885+0000","last_peered":"2024-04-25T05:45:04.929885+0000","last_clean":"2024-04-25T05:45:04.929885+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:45:04.929885+0000","last_undegraded":"2024-04-25T05:45:04.929885+0000","last_fullsized":"2024-04-25T05:45:04.929885+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":903,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928537+0000","last_change":"2024-04-25T05:44:30.947331+0000","last_active":"2024-04-25T05:45:04.928537+0000","last_peered":"2024-04-25T05:45:04.928537+0000","last_clean":"2024-04-25T05:45:04.928537+0000","last_became_active":"2024-04-25T05:44:30.946674+0000","last_became_peered":"2024-04-25T05:44:30.946674+0000","last_unstale":"2024-04-25T05:45:04.928537+0000","last_undegraded":"2024-04-25T05:45:04.928537+0000","last_fullsized":"2024-04-25T05:45:04.928537+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:41:44.999751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:41:44.999751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"341'9298","reported_seq":23540,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:07.905695+0000","last_change":"2024-04-25T05:44:47.251341+0000","last_active":"2024-04-25T05:45:07.905695+0000","last_peered":"2024-04-25T05:45:07.905695+0000","last_clean":"2024-04-25T05:45:07.905695+0000","last_became_active":"2024-04-25T05:44:47.245906+0000","last_became_peered":"2024-04-25T05:44:47.245906+0000","last_unstale":"2024-04-25T05:45:07.905695+0000","last_undegraded":"2024-04-25T05:45:07.905695+0000","last_fullsized":"2024-04-25T05:45:07.905695+0000","mapping_epoch":332,"log_start":"322'6008","ondisk_log_start":"322'6008","created":296,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3290,"log_dups_size":0,"ondisk_log_size":3290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386924544,"num_objects":5904,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":486464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7061,"num_bytes_recovered":462749696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":212,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923790+0000","last_change":"2024-04-25T05:44:28.188205+0000","last_active":"2024-04-25T05:45:04.923790+0000","last_peered":"2024-04-25T05:45:04.923790+0000","last_clean":"2024-04-25T05:45:04.923790+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:45:04.923790+0000","last_undegraded":"2024-04-25T05:45:04.923790+0000","last_fullsized":"2024-04-25T05:45:04.923790+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:18.598937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:18.598937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":836,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.921114+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:45:04.921114+0000","last_peered":"2024-04-25T05:45:04.921114+0000","last_clean":"2024-04-25T05:45:04.921114+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:45:04.921114+0000","last_undegraded":"2024-04-25T05:45:04.921114+0000","last_fullsized":"2024-04-25T05:45:04.921114+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":382,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924889+0000","last_change":"2024-04-25T05:44:28.160664+0000","last_active":"2024-04-25T05:45:04.924889+0000","last_peered":"2024-04-25T05:45:04.924889+0000","last_clean":"2024-04-25T05:45:04.924889+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:45:04.924889+0000","last_undegraded":"2024-04-25T05:45:04.924889+0000","last_fullsized":"2024-04-25T05:45:04.924889+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.320979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":126,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.944394+0000","last_change":"2024-04-25T05:44:28.205511+0000","last_active":"2024-04-25T05:45:04.944394+0000","last_peered":"2024-04-25T05:45:04.944394+0000","last_clean":"2024-04-25T05:45:04.944394+0000","last_became_active":"2024-04-25T05:44:28.176104+0000","last_became_peered":"2024-04-25T05:44:28.176104+0000","last_unstale":"2024-04-25T05:45:04.944394+0000","last_undegraded":"2024-04-25T05:45:04.944394+0000","last_fullsized":"2024-04-25T05:45:04.944394+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":567,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932686+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:45:04.932686+0000","last_peered":"2024-04-25T05:45:04.932686+0000","last_clean":"2024-04-25T05:45:04.932686+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:45:04.932686+0000","last_undegraded":"2024-04-25T05:45:04.932686+0000","last_fullsized":"2024-04-25T05:45:04.932686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":607,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.918383+0000","last_change":"2024-04-25T05:44:30.918203+0000","last_active":"2024-04-25T05:45:04.918383+0000","last_peered":"2024-04-25T05:45:04.918383+0000","last_clean":"2024-04-25T05:45:04.918383+0000","last_became_active":"2024-04-25T05:44:30.917713+0000","last_became_peered":"2024-04-25T05:44:30.917713+0000","last_unstale":"2024-04-25T05:45:04.918383+0000","last_undegraded":"2024-04-25T05:45:04.918383+0000","last_fullsized":"2024-04-25T05:45:04.918383+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.424874+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"341'9184","reported_seq":19224,"reported_epoch":341,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:45:08.750919+0000","last_change":"2024-04-25T05:44:29.205531+0000","last_active":"2024-04-25T05:45:08.750919+0000","last_peered":"2024-04-25T05:45:08.750919+0000","last_clean":"2024-04-25T05:44:27.108028+0000","last_became_active":"2024-04-25T05:44:29.173771+0000","last_became_peered":"2024-04-25T05:44:29.173771+0000","last_unstale":"2024-04-25T05:45:08.750919+0000","last_undegraded":"2024-04-25T05:45:08.750919+0000","last_fullsized":"2024-04-25T05:45:08.750919+0000","mapping_epoch":325,"log_start":"323'6291","ondisk_log_start":"323'6291","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2893,"log_dups_size":2815,"ondisk_log_size":2893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385875968,"num_objects":5888,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":5888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":5888}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":305,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923988+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:45:04.923988+0000","last_peered":"2024-04-25T05:45:04.923988+0000","last_clean":"2024-04-25T05:45:04.923988+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:45:04.923988+0000","last_undegraded":"2024-04-25T05:45:04.923988+0000","last_fullsized":"2024-04-25T05:45:04.923988+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":396,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.938927+0000","last_change":"2024-04-25T05:43:16.191937+0000","last_active":"2024-04-25T05:45:04.938927+0000","last_peered":"2024-04-25T05:45:04.938927+0000","last_clean":"2024-04-25T05:45:04.938927+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:45:04.938927+0000","last_undegraded":"2024-04-25T05:45:04.938927+0000","last_fullsized":"2024-04-25T05:45:04.938927+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:41:28.560585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":235,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.919712+0000","last_change":"2024-04-25T05:44:28.171402+0000","last_active":"2024-04-25T05:45:04.919712+0000","last_peered":"2024-04-25T05:45:04.919712+0000","last_clean":"2024-04-25T05:45:04.919712+0000","last_became_active":"2024-04-25T05:44:28.171004+0000","last_became_peered":"2024-04-25T05:44:28.171004+0000","last_unstale":"2024-04-25T05:45:04.919712+0000","last_undegraded":"2024-04-25T05:45:04.919712+0000","last_fullsized":"2024-04-25T05:45:04.919712+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.186275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.186275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:20.209466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927120+0000","last_change":"2024-04-25T05:44:28.169508+0000","last_active":"2024-04-25T05:45:04.927120+0000","last_peered":"2024-04-25T05:45:04.927120+0000","last_clean":"2024-04-25T05:45:04.927120+0000","last_became_active":"2024-04-25T05:44:28.169146+0000","last_became_peered":"2024-04-25T05:44:28.169146+0000","last_unstale":"2024-04-25T05:45:04.927120+0000","last_undegraded":"2024-04-25T05:45:04.927120+0000","last_fullsized":"2024-04-25T05:45:04.927120+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"341'9065","reported_seq":9155,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:07.895741+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:45:07.895741+0000","last_peered":"2024-04-25T05:45:07.895741+0000","last_clean":"2024-04-25T05:45:07.895741+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:45:07.895741+0000","last_undegraded":"2024-04-25T05:45:07.895741+0000","last_fullsized":"2024-04-25T05:45:07.895741+0000","mapping_epoch":301,"log_start":"322'5755","ondisk_log_start":"322'5755","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383188992,"num_objects":5847,"num_object_clones":0,"num_object_copies":17541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9065,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":356,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924677+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:45:04.924677+0000","last_peered":"2024-04-25T05:45:04.924677+0000","last_clean":"2024-04-25T05:45:04.924677+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:45:04.924677+0000","last_undegraded":"2024-04-25T05:45:04.924677+0000","last_fullsized":"2024-04-25T05:45:04.924677+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":630,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929678+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:45:04.929678+0000","last_peered":"2024-04-25T05:45:04.929678+0000","last_clean":"2024-04-25T05:45:04.929678+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:45:04.929678+0000","last_undegraded":"2024-04-25T05:45:04.929678+0000","last_fullsized":"2024-04-25T05:45:04.929678+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":539,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928913+0000","last_change":"2024-04-25T05:44:28.180755+0000","last_active":"2024-04-25T05:45:04.928913+0000","last_peered":"2024-04-25T05:45:04.928913+0000","last_clean":"2024-04-25T05:45:04.928913+0000","last_became_active":"2024-04-25T05:44:28.180393+0000","last_became_peered":"2024-04-25T05:44:28.180393+0000","last_unstale":"2024-04-25T05:45:04.928913+0000","last_undegraded":"2024-04-25T05:45:04.928913+0000","last_fullsized":"2024-04-25T05:45:04.928913+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.534093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":589,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925235+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:45:04.925235+0000","last_peered":"2024-04-25T05:45:04.925235+0000","last_clean":"2024-04-25T05:45:04.925235+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:45:04.925235+0000","last_undegraded":"2024-04-25T05:45:04.925235+0000","last_fullsized":"2024-04-25T05:45:04.925235+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.832260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"341'9295","reported_seq":9394,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:08.736622+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:45:08.736622+0000","last_peered":"2024-04-25T05:45:08.736622+0000","last_clean":"2024-04-25T05:45:08.736622+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:45:08.736622+0000","last_undegraded":"2024-04-25T05:45:08.736622+0000","last_fullsized":"2024-04-25T05:45:08.736622+0000","mapping_epoch":296,"log_start":"323'6386","ondisk_log_start":"323'6386","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2909,"log_dups_size":2815,"ondisk_log_size":2909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394461184,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9295,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":540,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926610+0000","last_change":"2024-04-25T05:44:28.169954+0000","last_active":"2024-04-25T05:45:04.926610+0000","last_peered":"2024-04-25T05:45:04.926610+0000","last_clean":"2024-04-25T05:45:04.926610+0000","last_became_active":"2024-04-25T05:44:28.168753+0000","last_became_peered":"2024-04-25T05:44:28.168753+0000","last_unstale":"2024-04-25T05:45:04.926610+0000","last_undegraded":"2024-04-25T05:45:04.926610+0000","last_fullsized":"2024-04-25T05:45:04.926610+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":244,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920585+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:45:04.920585+0000","last_peered":"2024-04-25T05:45:04.920585+0000","last_clean":"2024-04-25T05:45:04.920585+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:45:04.920585+0000","last_undegraded":"2024-04-25T05:45:04.920585+0000","last_fullsized":"2024-04-25T05:45:04.920585+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.771369+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":594,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928963+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:45:04.928963+0000","last_peered":"2024-04-25T05:45:04.928963+0000","last_clean":"2024-04-25T05:45:04.928963+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:45:04.928963+0000","last_undegraded":"2024-04-25T05:45:04.928963+0000","last_fullsized":"2024-04-25T05:45:04.928963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":578,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926417+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:45:04.926417+0000","last_peered":"2024-04-25T05:45:04.926417+0000","last_clean":"2024-04-25T05:45:04.926417+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:45:04.926417+0000","last_undegraded":"2024-04-25T05:45:04.926417+0000","last_fullsized":"2024-04-25T05:45:04.926417+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"341'9117","reported_seq":22715,"reported_epoch":341,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:45:07.937408+0000","last_change":"2024-04-25T05:44:29.255024+0000","last_active":"2024-04-25T05:45:07.937408+0000","last_peered":"2024-04-25T05:45:07.937408+0000","last_clean":"2024-04-25T05:44:27.090244+0000","last_became_active":"2024-04-25T05:44:29.209023+0000","last_became_peered":"2024-04-25T05:44:29.209023+0000","last_unstale":"2024-04-25T05:45:07.937408+0000","last_undegraded":"2024-04-25T05:45:07.937408+0000","last_fullsized":"2024-04-25T05:45:07.937408+0000","mapping_epoch":325,"log_start":"322'5793","ondisk_log_start":"322'5793","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3324,"log_dups_size":0,"ondisk_log_size":3324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389480448,"num_objects":5943,"num_object_clones":0,"num_object_copies":17829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":141,"num_objects_unfound":0,"num_objects_dirty":5943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9117,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6741,"num_bytes_recovered":441778176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":5943}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":753,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927637+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:45:04.927637+0000","last_peered":"2024-04-25T05:45:04.927637+0000","last_clean":"2024-04-25T05:45:04.927637+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:45:04.927637+0000","last_undegraded":"2024-04-25T05:45:04.927637+0000","last_fullsized":"2024-04-25T05:45:04.927637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":196,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925620+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:45:04.925620+0000","last_peered":"2024-04-25T05:45:04.925620+0000","last_clean":"2024-04-25T05:45:04.925620+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:45:04.925620+0000","last_undegraded":"2024-04-25T05:45:04.925620+0000","last_fullsized":"2024-04-25T05:45:04.925620+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":642,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.931177+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:45:04.931177+0000","last_peered":"2024-04-25T05:45:04.931177+0000","last_clean":"2024-04-25T05:45:04.931177+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:45:04.931177+0000","last_undegraded":"2024-04-25T05:45:04.931177+0000","last_fullsized":"2024-04-25T05:45:04.931177+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.858926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":469,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924423+0000","last_change":"2024-04-25T05:44:28.174865+0000","last_active":"2024-04-25T05:45:04.924423+0000","last_peered":"2024-04-25T05:45:04.924423+0000","last_clean":"2024-04-25T05:45:04.924423+0000","last_became_active":"2024-04-25T05:44:28.174499+0000","last_became_peered":"2024-04-25T05:44:28.174499+0000","last_unstale":"2024-04-25T05:45:04.924423+0000","last_undegraded":"2024-04-25T05:45:04.924423+0000","last_fullsized":"2024-04-25T05:45:04.924423+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":612,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.938907+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:45:04.938907+0000","last_peered":"2024-04-25T05:45:04.938907+0000","last_clean":"2024-04-25T05:45:04.938907+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:45:04.938907+0000","last_undegraded":"2024-04-25T05:45:04.938907+0000","last_fullsized":"2024-04-25T05:45:04.938907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.253497+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928024+0000","last_change":"2024-04-25T05:44:28.191586+0000","last_active":"2024-04-25T05:45:04.928024+0000","last_peered":"2024-04-25T05:45:04.928024+0000","last_clean":"2024-04-25T05:45:04.928024+0000","last_became_active":"2024-04-25T05:44:28.191288+0000","last_became_peered":"2024-04-25T05:44:28.191288+0000","last_unstale":"2024-04-25T05:45:04.928024+0000","last_undegraded":"2024-04-25T05:45:04.928024+0000","last_fullsized":"2024-04-25T05:45:04.928024+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.539924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":98,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932501+0000","last_change":"2024-04-25T05:44:19.931291+0000","last_active":"2024-04-25T05:45:04.932501+0000","last_peered":"2024-04-25T05:45:04.932501+0000","last_clean":"2024-04-25T05:45:04.932501+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:45:04.932501+0000","last_undegraded":"2024-04-25T05:45:04.932501+0000","last_fullsized":"2024-04-25T05:45:04.932501+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.670336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":594,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929205+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:45:04.929205+0000","last_peered":"2024-04-25T05:45:04.929205+0000","last_clean":"2024-04-25T05:45:04.929205+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:45:04.929205+0000","last_undegraded":"2024-04-25T05:45:04.929205+0000","last_fullsized":"2024-04-25T05:45:04.929205+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:06.898722+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":273,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932582+0000","last_change":"2024-04-25T05:44:28.203127+0000","last_active":"2024-04-25T05:45:04.932582+0000","last_peered":"2024-04-25T05:45:04.932582+0000","last_clean":"2024-04-25T05:45:04.932582+0000","last_became_active":"2024-04-25T05:44:28.184563+0000","last_became_peered":"2024-04-25T05:44:28.184563+0000","last_unstale":"2024-04-25T05:45:04.932582+0000","last_undegraded":"2024-04-25T05:45:04.932582+0000","last_fullsized":"2024-04-25T05:45:04.932582+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":612,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927400+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:45:04.927400+0000","last_peered":"2024-04-25T05:45:04.927400+0000","last_clean":"2024-04-25T05:45:04.927400+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:45:04.927400+0000","last_undegraded":"2024-04-25T05:45:04.927400+0000","last_fullsized":"2024-04-25T05:45:04.927400+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":570,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.936591+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:45:04.936591+0000","last_peered":"2024-04-25T05:45:04.936591+0000","last_clean":"2024-04-25T05:45:04.936591+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:45:04.936591+0000","last_undegraded":"2024-04-25T05:45:04.936591+0000","last_fullsized":"2024-04-25T05:45:04.936591+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.522496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":236,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.921210+0000","last_change":"2024-04-25T05:44:28.120205+0000","last_active":"2024-04-25T05:45:04.921210+0000","last_peered":"2024-04-25T05:45:04.921210+0000","last_clean":"2024-04-25T05:45:04.921210+0000","last_became_active":"2024-04-25T05:44:28.119749+0000","last_became_peered":"2024-04-25T05:44:28.119749+0000","last_unstale":"2024-04-25T05:45:04.921210+0000","last_undegraded":"2024-04-25T05:45:04.921210+0000","last_fullsized":"2024-04-25T05:45:04.921210+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.285907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":215,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929783+0000","last_change":"2024-04-25T05:43:16.229957+0000","last_active":"2024-04-25T05:45:04.929783+0000","last_peered":"2024-04-25T05:45:04.929783+0000","last_clean":"2024-04-25T05:45:04.929783+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:45:04.929783+0000","last_undegraded":"2024-04-25T05:45:04.929783+0000","last_fullsized":"2024-04-25T05:45:04.929783+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925450+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:45:04.925450+0000","last_peered":"2024-04-25T05:45:04.925450+0000","last_clean":"2024-04-25T05:45:04.925450+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:45:04.925450+0000","last_undegraded":"2024-04-25T05:45:04.925450+0000","last_fullsized":"2024-04-25T05:45:04.925450+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:15.919590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":722,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.939844+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:45:04.939844+0000","last_peered":"2024-04-25T05:45:04.939844+0000","last_clean":"2024-04-25T05:45:04.939844+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:45:04.939844+0000","last_undegraded":"2024-04-25T05:45:04.939844+0000","last_fullsized":"2024-04-25T05:45:04.939844+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.732091+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928874+0000","last_change":"2024-04-25T05:44:28.167266+0000","last_active":"2024-04-25T05:45:04.928874+0000","last_peered":"2024-04-25T05:45:04.928874+0000","last_clean":"2024-04-25T05:45:04.928874+0000","last_became_active":"2024-04-25T05:44:28.166838+0000","last_became_peered":"2024-04-25T05:44:28.166838+0000","last_unstale":"2024-04-25T05:45:04.928874+0000","last_undegraded":"2024-04-25T05:45:04.928874+0000","last_fullsized":"2024-04-25T05:45:04.928874+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:22.460917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":628,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.938821+0000","last_change":"2024-04-25T05:42:10.358229+0000","last_active":"2024-04-25T05:45:04.938821+0000","last_peered":"2024-04-25T05:45:04.938821+0000","last_clean":"2024-04-25T05:45:04.938821+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:45:04.938821+0000","last_undegraded":"2024-04-25T05:45:04.938821+0000","last_fullsized":"2024-04-25T05:45:04.938821+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:10.357898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:10.357898+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":600,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929184+0000","last_change":"2024-04-25T05:44:13.015441+0000","last_active":"2024-04-25T05:45:04.929184+0000","last_peered":"2024-04-25T05:45:04.929184+0000","last_clean":"2024-04-25T05:45:04.929184+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:45:04.929184+0000","last_undegraded":"2024-04-25T05:45:04.929184+0000","last_fullsized":"2024-04-25T05:45:04.929184+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:27.074191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":151,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923559+0000","last_change":"2024-04-25T05:44:28.236616+0000","last_active":"2024-04-25T05:45:04.923559+0000","last_peered":"2024-04-25T05:45:04.923559+0000","last_clean":"2024-04-25T05:45:04.923559+0000","last_became_active":"2024-04-25T05:44:28.184915+0000","last_became_peered":"2024-04-25T05:44:28.184915+0000","last_unstale":"2024-04-25T05:45:04.923559+0000","last_undegraded":"2024-04-25T05:45:04.923559+0000","last_fullsized":"2024-04-25T05:45:04.923559+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928848+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:45:04.928848+0000","last_peered":"2024-04-25T05:45:04.928848+0000","last_clean":"2024-04-25T05:45:04.928848+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:45:04.928848+0000","last_undegraded":"2024-04-25T05:45:04.928848+0000","last_fullsized":"2024-04-25T05:45:04.928848+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":706,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.919432+0000","last_change":"2024-04-25T05:44:19.254245+0000","last_active":"2024-04-25T05:45:04.919432+0000","last_peered":"2024-04-25T05:45:04.919432+0000","last_clean":"2024-04-25T05:45:04.919432+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:45:04.919432+0000","last_undegraded":"2024-04-25T05:45:04.919432+0000","last_fullsized":"2024-04-25T05:45:04.919432+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:19.254124+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.254124+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:41.554540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":266,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927809+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:45:04.927809+0000","last_peered":"2024-04-25T05:45:04.927809+0000","last_clean":"2024-04-25T05:45:04.927809+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:45:04.927809+0000","last_undegraded":"2024-04-25T05:45:04.927809+0000","last_fullsized":"2024-04-25T05:45:04.927809+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.945186+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:45:04.945186+0000","last_peered":"2024-04-25T05:45:04.945186+0000","last_clean":"2024-04-25T05:45:04.945186+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:45:04.945186+0000","last_undegraded":"2024-04-25T05:45:04.945186+0000","last_fullsized":"2024-04-25T05:45:04.945186+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":598,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932640+0000","last_change":"2024-04-25T05:44:28.186508+0000","last_active":"2024-04-25T05:45:04.932640+0000","last_peered":"2024-04-25T05:45:04.932640+0000","last_clean":"2024-04-25T05:45:04.932640+0000","last_became_active":"2024-04-25T05:44:28.184914+0000","last_became_peered":"2024-04-25T05:44:28.184914+0000","last_unstale":"2024-04-25T05:45:04.932640+0000","last_undegraded":"2024-04-25T05:45:04.932640+0000","last_fullsized":"2024-04-25T05:45:04.932640+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":124,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.935506+0000","last_change":"2024-04-25T05:44:28.208196+0000","last_active":"2024-04-25T05:45:04.935506+0000","last_peered":"2024-04-25T05:45:04.935506+0000","last_clean":"2024-04-25T05:45:04.935506+0000","last_became_active":"2024-04-25T05:44:28.160392+0000","last_became_peered":"2024-04-25T05:44:28.160392+0000","last_unstale":"2024-04-25T05:45:04.935506+0000","last_undegraded":"2024-04-25T05:45:04.935506+0000","last_fullsized":"2024-04-25T05:45:04.935506+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":594,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929304+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:45:04.929304+0000","last_peered":"2024-04-25T05:45:04.929304+0000","last_clean":"2024-04-25T05:45:04.929304+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:45:04.929304+0000","last_undegraded":"2024-04-25T05:45:04.929304+0000","last_fullsized":"2024-04-25T05:45:04.929304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":588,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926230+0000","last_change":"2024-04-25T05:44:28.174271+0000","last_active":"2024-04-25T05:45:04.926230+0000","last_peered":"2024-04-25T05:45:04.926230+0000","last_clean":"2024-04-25T05:45:04.926230+0000","last_became_active":"2024-04-25T05:44:28.172179+0000","last_became_peered":"2024-04-25T05:44:28.172179+0000","last_unstale":"2024-04-25T05:45:04.926230+0000","last_undegraded":"2024-04-25T05:45:04.926230+0000","last_fullsized":"2024-04-25T05:45:04.926230+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:29.204906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":199,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924800+0000","last_change":"2024-04-25T05:44:28.179459+0000","last_active":"2024-04-25T05:45:04.924800+0000","last_peered":"2024-04-25T05:45:04.924800+0000","last_clean":"2024-04-25T05:45:04.924800+0000","last_became_active":"2024-04-25T05:44:28.176134+0000","last_became_peered":"2024-04-25T05:44:28.176134+0000","last_unstale":"2024-04-25T05:45:04.924800+0000","last_undegraded":"2024-04-25T05:45:04.924800+0000","last_fullsized":"2024-04-25T05:45:04.924800+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":665,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933425+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:45:04.933425+0000","last_peered":"2024-04-25T05:45:04.933425+0000","last_clean":"2024-04-25T05:45:04.933425+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:45:04.933425+0000","last_undegraded":"2024-04-25T05:45:04.933425+0000","last_fullsized":"2024-04-25T05:45:04.933425+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.001968+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929008+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:45:04.929008+0000","last_peered":"2024-04-25T05:45:04.929008+0000","last_clean":"2024-04-25T05:45:04.929008+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:45:04.929008+0000","last_undegraded":"2024-04-25T05:45:04.929008+0000","last_fullsized":"2024-04-25T05:45:04.929008+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":604,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925763+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:45:04.925763+0000","last_peered":"2024-04-25T05:45:04.925763+0000","last_clean":"2024-04-25T05:45:04.925763+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:45:04.925763+0000","last_undegraded":"2024-04-25T05:45:04.925763+0000","last_fullsized":"2024-04-25T05:45:04.925763+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":596,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926517+0000","last_change":"2024-04-25T05:44:28.179442+0000","last_active":"2024-04-25T05:45:04.926517+0000","last_peered":"2024-04-25T05:45:04.926517+0000","last_clean":"2024-04-25T05:45:04.926517+0000","last_became_active":"2024-04-25T05:44:28.179103+0000","last_became_peered":"2024-04-25T05:44:28.179103+0000","last_unstale":"2024-04-25T05:45:04.926517+0000","last_undegraded":"2024-04-25T05:45:04.926517+0000","last_fullsized":"2024-04-25T05:45:04.926517+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":610,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.925026+0000","last_change":"2024-04-25T05:40:55.608005+0000","last_active":"2024-04-25T05:45:04.925026+0000","last_peered":"2024-04-25T05:45:04.925026+0000","last_clean":"2024-04-25T05:45:04.925026+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:45:04.925026+0000","last_undegraded":"2024-04-25T05:45:04.925026+0000","last_fullsized":"2024-04-25T05:45:04.925026+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T05:40:55.607852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:40:55.607852+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":191,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.936200+0000","last_change":"2024-04-25T05:44:28.188135+0000","last_active":"2024-04-25T05:45:04.936200+0000","last_peered":"2024-04-25T05:45:04.936200+0000","last_clean":"2024-04-25T05:45:04.936200+0000","last_became_active":"2024-04-25T05:44:28.187806+0000","last_became_peered":"2024-04-25T05:44:28.187806+0000","last_unstale":"2024-04-25T05:45:04.936200+0000","last_undegraded":"2024-04-25T05:45:04.936200+0000","last_fullsized":"2024-04-25T05:45:04.936200+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":302,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924985+0000","last_change":"2024-04-25T05:43:29.610487+0000","last_active":"2024-04-25T05:45:04.924985+0000","last_peered":"2024-04-25T05:45:04.924985+0000","last_clean":"2024-04-25T05:45:04.924985+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:45:04.924985+0000","last_undegraded":"2024-04-25T05:45:04.924985+0000","last_fullsized":"2024-04-25T05:45:04.924985+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.610411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.610411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":593,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928646+0000","last_change":"2024-04-25T05:44:19.960606+0000","last_active":"2024-04-25T05:45:04.928646+0000","last_peered":"2024-04-25T05:45:04.928646+0000","last_clean":"2024-04-25T05:45:04.928646+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:45:04.928646+0000","last_undegraded":"2024-04-25T05:45:04.928646+0000","last_fullsized":"2024-04-25T05:45:04.928646+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:24.771833+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":382,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926710+0000","last_change":"2024-04-25T05:44:28.212080+0000","last_active":"2024-04-25T05:45:04.926710+0000","last_peered":"2024-04-25T05:45:04.926710+0000","last_clean":"2024-04-25T05:45:04.926710+0000","last_became_active":"2024-04-25T05:44:28.181996+0000","last_became_peered":"2024-04-25T05:44:28.181996+0000","last_unstale":"2024-04-25T05:45:04.926710+0000","last_undegraded":"2024-04-25T05:45:04.926710+0000","last_fullsized":"2024-04-25T05:45:04.926710+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.299319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":604,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.924883+0000","last_change":"2024-04-25T05:44:28.172975+0000","last_active":"2024-04-25T05:45:04.924883+0000","last_peered":"2024-04-25T05:45:04.924883+0000","last_clean":"2024-04-25T05:45:04.924883+0000","last_became_active":"2024-04-25T05:44:28.171311+0000","last_became_peered":"2024-04-25T05:44:28.171311+0000","last_unstale":"2024-04-25T05:45:04.924883+0000","last_undegraded":"2024-04-25T05:45:04.924883+0000","last_fullsized":"2024-04-25T05:45:04.924883+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:37.398601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929073+0000","last_change":"2024-04-25T05:44:28.179599+0000","last_active":"2024-04-25T05:45:04.929073+0000","last_peered":"2024-04-25T05:45:04.929073+0000","last_clean":"2024-04-25T05:45:04.929073+0000","last_became_active":"2024-04-25T05:44:28.178710+0000","last_became_peered":"2024-04-25T05:44:28.178710+0000","last_unstale":"2024-04-25T05:45:04.929073+0000","last_undegraded":"2024-04-25T05:45:04.929073+0000","last_fullsized":"2024-04-25T05:45:04.929073+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:32.286921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":580,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923357+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:45:04.923357+0000","last_peered":"2024-04-25T05:45:04.923357+0000","last_clean":"2024-04-25T05:45:04.923357+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:45:04.923357+0000","last_undegraded":"2024-04-25T05:45:04.923357+0000","last_fullsized":"2024-04-25T05:45:04.923357+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":279,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929233+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:45:04.929233+0000","last_peered":"2024-04-25T05:45:04.929233+0000","last_clean":"2024-04-25T05:45:04.929233+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:45:04.929233+0000","last_undegraded":"2024-04-25T05:45:04.929233+0000","last_fullsized":"2024-04-25T05:45:04.929233+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":595,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920692+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:45:04.920692+0000","last_peered":"2024-04-25T05:45:04.920692+0000","last_clean":"2024-04-25T05:45:04.920692+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:45:04.920692+0000","last_undegraded":"2024-04-25T05:45:04.920692+0000","last_fullsized":"2024-04-25T05:45:04.920692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:40.500720+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":381,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926447+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:45:04.926447+0000","last_peered":"2024-04-25T05:45:04.926447+0000","last_clean":"2024-04-25T05:45:04.926447+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:45:04.926447+0000","last_undegraded":"2024-04-25T05:45:04.926447+0000","last_fullsized":"2024-04-25T05:45:04.926447+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":603,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.927972+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:45:04.927972+0000","last_peered":"2024-04-25T05:45:04.927972+0000","last_clean":"2024-04-25T05:45:04.927972+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:45:04.927972+0000","last_undegraded":"2024-04-25T05:45:04.927972+0000","last_fullsized":"2024-04-25T05:45:04.927972+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":618,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932442+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:45:04.932442+0000","last_peered":"2024-04-25T05:45:04.932442+0000","last_clean":"2024-04-25T05:45:04.932442+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:45:04.932442+0000","last_undegraded":"2024-04-25T05:45:04.932442+0000","last_fullsized":"2024-04-25T05:45:04.932442+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":614,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.919585+0000","last_change":"2024-04-25T05:45:02.437804+0000","last_active":"2024-04-25T05:45:04.919585+0000","last_peered":"2024-04-25T05:45:04.919585+0000","last_clean":"2024-04-25T05:45:04.919585+0000","last_became_active":"2024-04-25T05:44:28.167563+0000","last_became_peered":"2024-04-25T05:44:28.167563+0000","last_unstale":"2024-04-25T05:45:04.919585+0000","last_undegraded":"2024-04-25T05:45:04.919585+0000","last_fullsized":"2024-04-25T05:45:04.919585+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:02.437727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:02.437727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:27.778001+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":263,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.939017+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:45:04.939017+0000","last_peered":"2024-04-25T05:45:04.939017+0000","last_clean":"2024-04-25T05:45:04.939017+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:45:04.939017+0000","last_undegraded":"2024-04-25T05:45:04.939017+0000","last_fullsized":"2024-04-25T05:45:04.939017+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":585,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.938783+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:45:04.938783+0000","last_peered":"2024-04-25T05:45:04.938783+0000","last_clean":"2024-04-25T05:45:04.938783+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:45:04.938783+0000","last_undegraded":"2024-04-25T05:45:04.938783+0000","last_fullsized":"2024-04-25T05:45:04.938783+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":485,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920838+0000","last_change":"2024-04-25T05:44:28.237495+0000","last_active":"2024-04-25T05:45:04.920838+0000","last_peered":"2024-04-25T05:45:04.920838+0000","last_clean":"2024-04-25T05:45:04.920838+0000","last_became_active":"2024-04-25T05:44:28.182958+0000","last_became_peered":"2024-04-25T05:44:28.182958+0000","last_unstale":"2024-04-25T05:45:04.920838+0000","last_undegraded":"2024-04-25T05:45:04.920838+0000","last_fullsized":"2024-04-25T05:45:04.920838+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":82,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929358+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:45:04.929358+0000","last_peered":"2024-04-25T05:45:04.929358+0000","last_clean":"2024-04-25T05:45:04.929358+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:45:04.929358+0000","last_undegraded":"2024-04-25T05:45:04.929358+0000","last_fullsized":"2024-04-25T05:45:04.929358+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:21.442272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":604,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.923706+0000","last_change":"2024-04-25T05:44:28.189092+0000","last_active":"2024-04-25T05:45:04.923706+0000","last_peered":"2024-04-25T05:45:04.923706+0000","last_clean":"2024-04-25T05:45:04.923706+0000","last_became_active":"2024-04-25T05:44:28.185597+0000","last_became_peered":"2024-04-25T05:44:28.185597+0000","last_unstale":"2024-04-25T05:45:04.923706+0000","last_undegraded":"2024-04-25T05:45:04.923706+0000","last_fullsized":"2024-04-25T05:45:04.923706+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:19.532823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":601,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.921688+0000","last_change":"2024-04-25T05:44:00.169219+0000","last_active":"2024-04-25T05:45:04.921688+0000","last_peered":"2024-04-25T05:45:04.921688+0000","last_clean":"2024-04-25T05:45:04.921688+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:45:04.921688+0000","last_undegraded":"2024-04-25T05:45:04.921688+0000","last_fullsized":"2024-04-25T05:45:04.921688+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.169057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.169057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":328,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.940400+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:45:04.940400+0000","last_peered":"2024-04-25T05:45:04.940400+0000","last_clean":"2024-04-25T05:45:04.940400+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:45:04.940400+0000","last_undegraded":"2024-04-25T05:45:04.940400+0000","last_fullsized":"2024-04-25T05:45:04.940400+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":106,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920927+0000","last_change":"2024-04-25T05:45:00.435729+0000","last_active":"2024-04-25T05:45:04.920927+0000","last_peered":"2024-04-25T05:45:04.920927+0000","last_clean":"2024-04-25T05:45:04.920927+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:45:04.920927+0000","last_undegraded":"2024-04-25T05:45:04.920927+0000","last_fullsized":"2024-04-25T05:45:04.920927+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:00.435460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:45:00.435460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:25.355468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":578,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928402+0000","last_change":"2024-04-25T05:41:39.996022+0000","last_active":"2024-04-25T05:45:04.928402+0000","last_peered":"2024-04-25T05:45:04.928402+0000","last_clean":"2024-04-25T05:45:04.928402+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:45:04.928402+0000","last_undegraded":"2024-04-25T05:45:04.928402+0000","last_fullsized":"2024-04-25T05:45:04.928402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:39.995740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:39.995740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":96,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920970+0000","last_change":"2024-04-25T05:44:28.201019+0000","last_active":"2024-04-25T05:45:04.920970+0000","last_peered":"2024-04-25T05:45:04.920970+0000","last_clean":"2024-04-25T05:45:04.920970+0000","last_became_active":"2024-04-25T05:44:28.200623+0000","last_became_peered":"2024-04-25T05:44:28.200623+0000","last_unstale":"2024-04-25T05:45:04.920970+0000","last_undegraded":"2024-04-25T05:45:04.920970+0000","last_fullsized":"2024-04-25T05:45:04.920970+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":365,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933627+0000","last_change":"2024-04-25T05:44:28.198166+0000","last_active":"2024-04-25T05:45:04.933627+0000","last_peered":"2024-04-25T05:45:04.933627+0000","last_clean":"2024-04-25T05:45:04.933627+0000","last_became_active":"2024-04-25T05:44:28.179156+0000","last_became_peered":"2024-04-25T05:44:28.179156+0000","last_unstale":"2024-04-25T05:45:04.933627+0000","last_undegraded":"2024-04-25T05:45:04.933627+0000","last_fullsized":"2024-04-25T05:45:04.933627+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":404,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926050+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:45:04.926050+0000","last_peered":"2024-04-25T05:45:04.926050+0000","last_clean":"2024-04-25T05:45:04.926050+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:45:04.926050+0000","last_undegraded":"2024-04-25T05:45:04.926050+0000","last_fullsized":"2024-04-25T05:45:04.926050+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:26.453574+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":402,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.929325+0000","last_change":"2024-04-25T05:44:28.126200+0000","last_active":"2024-04-25T05:45:04.929325+0000","last_peered":"2024-04-25T05:45:04.929325+0000","last_clean":"2024-04-25T05:45:04.929325+0000","last_became_active":"2024-04-25T05:44:28.125552+0000","last_became_peered":"2024-04-25T05:44:28.125552+0000","last_unstale":"2024-04-25T05:45:04.929325+0000","last_undegraded":"2024-04-25T05:45:04.929325+0000","last_fullsized":"2024-04-25T05:45:04.929325+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:15.371929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:40:15.371929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":206,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926734+0000","last_change":"2024-04-25T05:44:28.169494+0000","last_active":"2024-04-25T05:45:04.926734+0000","last_peered":"2024-04-25T05:45:04.926734+0000","last_clean":"2024-04-25T05:45:04.926734+0000","last_became_active":"2024-04-25T05:44:28.168461+0000","last_became_peered":"2024-04-25T05:44:28.168461+0000","last_unstale":"2024-04-25T05:45:04.926734+0000","last_undegraded":"2024-04-25T05:45:04.926734+0000","last_fullsized":"2024-04-25T05:45:04.926734+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":815,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.933692+0000","last_change":"2024-04-25T05:44:28.225524+0000","last_active":"2024-04-25T05:45:04.933692+0000","last_peered":"2024-04-25T05:45:04.933692+0000","last_clean":"2024-04-25T05:45:04.933692+0000","last_became_active":"2024-04-25T05:44:28.182800+0000","last_became_peered":"2024-04-25T05:44:28.182800+0000","last_unstale":"2024-04-25T05:45:04.933692+0000","last_undegraded":"2024-04-25T05:45:04.933692+0000","last_fullsized":"2024-04-25T05:45:04.933692+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":605,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.932345+0000","last_change":"2024-04-25T05:44:16.995982+0000","last_active":"2024-04-25T05:45:04.932345+0000","last_peered":"2024-04-25T05:45:04.932345+0000","last_clean":"2024-04-25T05:45:04.932345+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:45:04.932345+0000","last_undegraded":"2024-04-25T05:45:04.932345+0000","last_fullsized":"2024-04-25T05:45:04.932345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:38.671823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":554,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.928315+0000","last_change":"2024-04-25T05:43:18.685398+0000","last_active":"2024-04-25T05:45:04.928315+0000","last_peered":"2024-04-25T05:45:04.928315+0000","last_clean":"2024-04-25T05:45:04.928315+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:45:04.928315+0000","last_undegraded":"2024-04-25T05:45:04.928315+0000","last_fullsized":"2024-04-25T05:45:04.928315+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.685299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.685299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":578,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926323+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:45:04.926323+0000","last_peered":"2024-04-25T05:45:04.926323+0000","last_clean":"2024-04-25T05:45:04.926323+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:45:04.926323+0000","last_undegraded":"2024-04-25T05:45:04.926323+0000","last_fullsized":"2024-04-25T05:45:04.926323+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":668,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.918411+0000","last_change":"2024-04-25T05:44:02.120412+0000","last_active":"2024-04-25T05:45:04.918411+0000","last_peered":"2024-04-25T05:45:04.918411+0000","last_clean":"2024-04-25T05:45:04.918411+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:45:04.918411+0000","last_undegraded":"2024-04-25T05:45:04.918411+0000","last_fullsized":"2024-04-25T05:45:04.918411+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:44:02.120258+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.120258+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:10.452057+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":205,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.926234+0000","last_change":"2024-04-25T05:44:28.182297+0000","last_active":"2024-04-25T05:45:04.926234+0000","last_peered":"2024-04-25T05:45:04.926234+0000","last_clean":"2024-04-25T05:45:04.926234+0000","last_became_active":"2024-04-25T05:44:28.181951+0000","last_became_peered":"2024-04-25T05:44:28.181951+0000","last_unstale":"2024-04-25T05:45:04.926234+0000","last_undegraded":"2024-04-25T05:45:04.926234+0000","last_fullsized":"2024-04-25T05:45:04.926234+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:57.544063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.544063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":592,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.920748+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:45:04.920748+0000","last_peered":"2024-04-25T05:45:04.920748+0000","last_clean":"2024-04-25T05:45:04.920748+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:45:04.920748+0000","last_undegraded":"2024-04-25T05:45:04.920748+0000","last_fullsized":"2024-04-25T05:45:04.920748+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:28.182548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-04-25T05:45:04.935908+0000","last_change":"2024-04-25T05:43:16.198408+0000","last_active":"2024-04-25T05:45:04.935908+0000","last_peered":"2024-04-25T05:45:04.935908+0000","last_clean":"2024-04-25T05:45:04.935908+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:45:04.935908+0000","last_undegraded":"2024-04-25T05:45:04.935908+0000","last_fullsized":"2024-04-25T05:45:04.935908+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6226313240,"num_objects":95007,"num_object_clones":0,"num_object_copies":285021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10517,"num_objects_unfound":0,"num_objects_dirty":95007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147617,"num_write_kb":7763905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29094,"num_bytes_recovered":1906704384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20384854016,"data_stored":20384841800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50561,"ondisk_log_size":50561,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":651,"num_read_kb":646,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":8366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":14104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070603,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5073460,"kb_used_data":4500844,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":88671692,"statfs":{"total":95995035648,"available":90799812608,"internally_reserved":0,"allocated":4608864256,"data_stored":4608367349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":586349002},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364943,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2370280,"kb_used_data":1910112,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91374872,"statfs":{"total":95995035648,"available":93567868928,"internally_reserved":0,"allocated":1955954688,"data_stored":1955469803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575016,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5428696,"kb_used_data":4775504,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":88316456,"statfs":{"total":95995035648,"available":90436050944,"internally_reserved":0,"allocated":4890116096,"data_stored":4889581629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5292168,"kb_used_data":4543232,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":88452984,"statfs":{"total":95995035648,"available":90575855616,"internally_reserved":0,"allocated":4652269568,"data_stored":4651752279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":766900679},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986328,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2340604,"kb_used_data":1940148,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91404548,"statfs":{"total":95995035648,"available":93598257152,"internally_reserved":0,"allocated":1986711552,"data_stored":1986206438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2381381632,"data_stored":2381381632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4885516288,"data_stored":4885512216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4627824640,"data_stored":4627824640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1970802688,"data_stored":1970798616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1940848640,"data_stored":1940848640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4578480128,"data_stored":4578476056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:45:15.824 INFO:tasks.thrashosds.thrasher:Setting 5.1c to [6, 0, 5] 2024-04-25T05:45:15.824 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1c', '6', '0', '5'] 2024-04-25T05:45:15.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 5.1c 6 0 5 2024-04-25T05:45:15.834 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:15.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:15.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:15.966 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:15.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:16.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:16.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:16.342 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:45:16.342 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:45:16.342 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:45:16.342 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:45:16.342 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:45:16.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:16.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:16.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:16.499 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:16.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:16.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:16.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:45:16.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:16.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:16.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:16.870 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:16.899 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:16.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:16.994 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:16.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:17.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:17.120 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:17.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:17.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:17.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:17 smithi043 ceph-mon[27010]: pgmap v592: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:17.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:17 smithi043 ceph-mon[27010]: osdmap e346: 8 total, 8 up, 5 in 2024-04-25T05:45:17.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:17 smithi043 ceph-mon[37492]: pgmap v592: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:17.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:17 smithi043 ceph-mon[37492]: osdmap e346: 8 total, 8 up, 5 in 2024-04-25T05:45:17.277 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:17.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:17.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:17.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:17.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:17 smithi060 ceph-mon[30898]: pgmap v592: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:17 smithi060 ceph-mon[30898]: osdmap e346: 8 total, 8 up, 5 in 2024-04-25T05:45:17.554 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:17.597 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:17.598 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:17.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:17.635 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:17.670 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:17.699 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:17.720 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:17.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:17.740 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:17.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:17.771 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:17.774 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:17.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:17.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:17.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:18.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:45:18.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:18.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:18.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:18.264 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:18.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: osdmap e347: 8 total, 8 up, 5 in 2024-04-25T05:45:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1910589874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1674834856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: osdmap e347: 8 total, 8 up, 5 in 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1910589874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1674834856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:18.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: osdmap e347: 8 total, 8 up, 5 in 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1910589874' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1674834856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:18.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:18.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:18.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:18.668 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:18.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:18.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:18.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:18.937 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:18.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:19.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:19.070 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:19.094 INFO:teuthology.orchestra.run.smithi043.stderr:set 5.1c pg_upmap mapping to [6,0,5] 2024-04-25T05:45:19.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:19.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:19.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:19.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: pgmap v595: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: osdmap e348: 8 total, 8 up, 5 in 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/146947654' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[27010]: 9.8 scrub starts 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: pgmap v595: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: osdmap e348: 8 total, 8 up, 5 in 2024-04-25T05:45:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/146947654' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-mon[37492]: 9.8 scrub starts 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: pgmap v595: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 725 op/s; 6535/272664 objects misplaced (2.397%); 38 MiB/s, 609 objects/s recovering 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: osdmap e348: 8 total, 8 up, 5 in 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/146947654' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]: dispatch 2024-04-25T05:45:19.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:19 smithi060 ceph-mon[30898]: 9.8 scrub starts 2024-04-25T05:45:19.437 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:19.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:45:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:19.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:19.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:19.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.756 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:19.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:19.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:19.870 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:45:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:19.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:19.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:20.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:20.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:20.170 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:20.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:20.302 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:20.329 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]': finished 2024-04-25T05:45:20.329 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:20 smithi060 ceph-mon[30898]: osdmap e349: 8 total, 8 up, 5 in 2024-04-25T05:45:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]': finished 2024-04-25T05:45:20.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:20 smithi043 ceph-mon[27010]: osdmap e349: 8 total, 8 up, 5 in 2024-04-25T05:45:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [6, 0, 5]}]': finished 2024-04-25T05:45:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:20 smithi043 ceph-mon[37492]: osdmap e349: 8 total, 8 up, 5 in 2024-04-25T05:45:20.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:20.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:20.530 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:20.530 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:20.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:20.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:20.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:20.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:20.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.841 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:20.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:20.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:20.971 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:20.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:21.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.105 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:21.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:21.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:45:21.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:21.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:21.370 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: pgmap v598: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 5353/259998 objects misplaced (2.059%); 15 MiB/s, 233 objects/s recovering 2024-04-25T05:45:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:45:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:45:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 9.6 scrub starts 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 9.8 scrub ok 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: pgmap v598: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 5353/259998 objects misplaced (2.059%); 15 MiB/s, 233 objects/s recovering 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 9.6 scrub starts 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 9.8 scrub ok 2024-04-25T05:45:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:45:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:21 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:45:21.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: pgmap v598: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 5353/259998 objects misplaced (2.059%); 15 MiB/s, 233 objects/s recovering 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 9.6 scrub starts 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 9.8 scrub ok 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:45:21.440 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:21 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:45:21.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:21.483 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:45:21.501 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:21.603 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:21.665 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:21.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:21.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:21.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:21.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:21.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:22.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.049 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:22.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:22.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:22.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:22.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:22.443 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:22.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:45:22.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:22.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:22.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:22.705 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:22.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:22.836 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:22.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:22.969 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:22.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:23.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:23.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:23.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:23.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:23.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:23.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: pgmap v599: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 877 B/s rd, 0 B/s wr, 724 op/s; 5353/259998 objects misplaced (2.059%); 10 MiB/s, 160 objects/s recovering 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 9.6 scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: pgmap v599: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 877 B/s rd, 0 B/s wr, 724 op/s; 5353/259998 objects misplaced (2.059%); 10 MiB/s, 160 objects/s recovering 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 9.6 scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:45:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:45:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:45:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:45:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:23 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:45:23.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:23.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: pgmap v599: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 877 B/s rd, 0 B/s wr, 724 op/s; 5353/259998 objects misplaced (2.059%); 10 MiB/s, 160 objects/s recovering 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 9.6 scrub ok 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:45:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:23 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:45:23.502 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:23.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:23.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:23.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:23.764 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:23.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:23.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:23.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:23.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:24.021 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:24.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:24.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:24.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:24.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:24.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:24.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:24.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:24.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:24.544 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:24.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:24.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:24.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:24.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:24.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:24.945 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:24.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:25.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:25.083 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:25.084 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:45:25.084 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:45:25.084 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-04-25T05:45:25.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:45:25.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:25.212 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:25.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:25.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[37492]: pgmap v600: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 4843/237717 objects misplaced (2.037%); 8.5 MiB/s, 136 objects/s recovering 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[27010]: pgmap v600: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 4843/237717 objects misplaced (2.037%); 8.5 MiB/s, 136 objects/s recovering 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:45:25.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:25 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:45:25.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:25.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:25.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:25 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:45:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:25 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:45:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:25 smithi060 ceph-mon[30898]: pgmap v600: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 4843/237717 objects misplaced (2.037%); 8.5 MiB/s, 136 objects/s recovering 2024-04-25T05:45:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:25 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:45:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:25 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:45:25.586 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:25.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:25.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:25.746 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:25.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:25.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:25.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:25.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:26.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:26.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:26.143 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:45:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:26.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:26.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:26.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3537204530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3537204530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:26.437 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3537204530' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:26.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:26.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:26.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:26.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:26.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:26.838 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:26.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:26.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:26.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:27.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:27.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:27.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:27.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.226 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:27.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:27.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:27.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: pgmap v601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 4843/237717 objects misplaced (2.037%); 7.3 MiB/s, 116 objects/s recovering 2024-04-25T05:45:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:45:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: osdmap e350: 8 total, 8 up, 5 in 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: pgmap v601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 4843/237717 objects misplaced (2.037%); 7.3 MiB/s, 116 objects/s recovering 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: osdmap e350: 8 total, 8 up, 5 in 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:45:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:27 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:45:27.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: pgmap v601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 4843/237717 objects misplaced (2.037%); 7.3 MiB/s, 116 objects/s recovering 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: osdmap e350: 8 total, 8 up, 5 in 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:45:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:27 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:45:27.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:27.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:27.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:27.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:27.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:27.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:27.903 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:27.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:28.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:28.033 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:28.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:28.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:28.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:28.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:45:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:45:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:45:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:45:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:45:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:45:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:45:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:28 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:45:28.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:28.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:28.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:28 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:45:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:28 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:45:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:28 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:45:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:28 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:45:28.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:28.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:28.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:28.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:28.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:28.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:28.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:28.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:29.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:29.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:29.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:29.214 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:29.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.348 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:29.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:29.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:29 smithi060 ceph-mon[30898]: pgmap v603: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 581 B/s rd, 0 B/s wr, 843 op/s; 4843/237717 objects misplaced (2.037%) 2024-04-25T05:45:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:29 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:45:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:29 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:45:29.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.608 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:29.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[27010]: pgmap v603: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 581 B/s rd, 0 B/s wr, 843 op/s; 4843/237717 objects misplaced (2.037%) 2024-04-25T05:45:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:45:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:45:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[37492]: pgmap v603: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 581 B/s rd, 0 B/s wr, 843 op/s; 4843/237717 objects misplaced (2.037%) 2024-04-25T05:45:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:45:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:45:29.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.737 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:29.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:29.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:29.874 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:29.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:29.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:30.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:30.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:30.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:30.241 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:30.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:30.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:30.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:30 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:45:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:30 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:45:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:30 smithi060 ceph-mon[30898]: 5.11 deep-scrub starts 2024-04-25T05:45:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:30 smithi060 ceph-mon[30898]: 5.11 deep-scrub ok 2024-04-25T05:45:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:30 smithi060 ceph-mon[30898]: osdmap e351: 8 total, 8 up, 5 in 2024-04-25T05:45:30.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.544 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:30.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:45:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:45:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[37492]: 5.11 deep-scrub starts 2024-04-25T05:45:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[37492]: 5.11 deep-scrub ok 2024-04-25T05:45:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[37492]: osdmap e351: 8 total, 8 up, 5 in 2024-04-25T05:45:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:45:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:45:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[27010]: 5.11 deep-scrub starts 2024-04-25T05:45:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[27010]: 5.11 deep-scrub ok 2024-04-25T05:45:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:30 smithi043 ceph-mon[27010]: osdmap e351: 8 total, 8 up, 5 in 2024-04-25T05:45:30.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:30.666 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:30.768 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:30.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:30.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:30.933 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:30.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:31.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:31.062 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:31.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:31.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:31.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:31.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:31.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:31.465 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:31 smithi060 ceph-mon[30898]: pgmap v604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:31 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:45:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:31 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:45:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:31 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:45:31.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:31.596 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[37492]: pgmap v604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:45:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:45:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:45:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[27010]: pgmap v604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:45:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:45:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:31 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:45:31.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.727 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:31.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:31.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:31.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:31.988 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:31.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:32.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:32.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:32.196 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:32.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:32.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:32.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:32.365 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:32.366 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:32.383 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:32.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 6.d deep-scrub starts 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 6.d deep-scrub ok 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:45:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:32 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 6.d deep-scrub starts 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 6.d deep-scrub ok 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:45:32.509 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 6.d deep-scrub starts 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 6.d deep-scrub ok 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:45:32.510 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:32 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:45:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:32.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:32.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:32.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:32.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:32.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:32.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:32.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:32.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:33.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:33.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:33.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:33.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:45:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:45:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: pgmap v606: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.a deep-scrub starts 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.a deep-scrub ok 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: pgmap v606: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.a deep-scrub starts 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.a deep-scrub ok 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:45:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:45:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:33.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:33.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:33.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: pgmap v606: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s; 4408/217437 objects misplaced (2.027%) 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.a deep-scrub starts 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.a deep-scrub ok 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:33.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:33.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:33.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:33.700 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:33.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:33.833 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:33.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:33.970 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:33.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:34.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:34.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:34.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:34.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:45:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:34.347 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:34.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:34.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:34 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:45:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:34 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:45:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:34.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:34.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:45:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:45:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:45:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:34 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:45:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.667 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:34.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:34.768 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:34.799 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:34.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:34.932 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:34.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:35.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:35.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:35.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:35.191 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:35.243 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: pgmap v607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]: dispatch 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]: dispatch 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: pgmap v607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]: dispatch 2024-04-25T05:45:35.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]: dispatch 2024-04-25T05:45:35.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:45:35.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:35 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:45:35.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:35.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:35.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:35.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: pgmap v607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]: dispatch 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]: dispatch 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:45:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:35 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:45:35.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:35.591 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:35.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:35.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:35.874 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:35.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:35.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:36.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:36.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.138 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:36.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:36.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:36.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:36.312 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:36.362 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]': finished 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[27010]: osdmap e352: 8 total, 8 up, 5 in 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]': finished 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[37492]: osdmap e352: 8 total, 8 up, 5 in 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:36 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:36.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:36.404 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:36.410 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:36.456 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:36.495 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:36.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.0", "id": [1, 6]}]': finished 2024-04-25T05:45:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [4, 3]}]': finished 2024-04-25T05:45:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:36 smithi060 ceph-mon[30898]: osdmap e352: 8 total, 8 up, 5 in 2024-04-25T05:45:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:36 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:36 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:36.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:36.540 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:36.578 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:36.618 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:36.628 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:36.628 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:36.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:36.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:36.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:36.837 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:36.939 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:36.968 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:36.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25487.0:183330 9.0 e353/325 9:0d61b4b8:::benchmark_data_smithi043_129820_object62016:head v 353'11385, mlcod=353'11382)", 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:36.869225+0000", 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00065818300000000001, 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00069207299999999997, 2024-04-25T05:45:37.017 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869225+0000", 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869225+0000", 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:45:37.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869227+0000", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3519999999999999e-06 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869237+0000", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8700000000000004e-06 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869257+0000", 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0675000000000001e-05 2024-04-25T05:45:37.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869267+0000", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8190000000000003e-06 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869360+0000", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3134000000000006e-05 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:36.869394+0000", 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4037e-05 2024-04-25T05:45:37.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 1 2024-04-25T05:45:37.021 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:45:37.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:37.099 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:37.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:37.229 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:37.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:37.360 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:37.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: pgmap v609: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: osdmap e353: 8 total, 8 up, 5 in 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4200901866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 6.7 scrub starts 2024-04-25T05:45:37.471 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[27010]: 6.7 scrub ok 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: pgmap v609: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: osdmap e353: 8 total, 8 up, 5 in 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4200901866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 6.7 scrub starts 2024-04-25T05:45:37.472 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:37 smithi043 ceph-mon[37492]: 6.7 scrub ok 2024-04-25T05:45:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.492 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:37.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: pgmap v609: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: osdmap e353: 8 total, 8 up, 5 in 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4200901866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 6.7 scrub starts 2024-04-25T05:45:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:37 smithi060 ceph-mon[30898]: 6.7 scrub ok 2024-04-25T05:45:37.593 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:37.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:37.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.755 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:37.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:37.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:37.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:37.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:37.996 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:45:37.999 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:45:38.022 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:38.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:38.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:38.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:38.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:38.380 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:38.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[27010]: 9.b scrub starts 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:45:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:38 smithi043 ceph-mon[37492]: 9.b scrub starts 2024-04-25T05:45:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:38.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:38 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:45:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:38 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:45:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:38 smithi060 ceph-mon[30898]: 9.b scrub starts 2024-04-25T05:45:38.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:38.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:38.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:38.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:38.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:38.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:38.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:38.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:39.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:45:39.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:39.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:39.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:39.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:39.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: pgmap v611: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 951 op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: from='client.25700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:45:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 9.b scrub ok 2024-04-25T05:45:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:45:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:45:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:45:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:39 smithi060 ceph-mon[30898]: osdmap e354: 8 total, 8 up, 5 in 2024-04-25T05:45:39.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:39.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:45:39.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:45:39.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: pgmap v611: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 951 op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:39.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: from='client.25700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 9.b scrub ok 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[27010]: osdmap e354: 8 total, 8 up, 5 in 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:45:39.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: pgmap v611: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 951 op/s; 3941/194676 objects misplaced (2.024%) 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: from='client.25700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 9.b scrub ok 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:45:39.533 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-mon[37492]: osdmap e354: 8 total, 8 up, 5 in 2024-04-25T05:45:39.538 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:39.626 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:39.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:39.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.671 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:39.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:39.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:39.802 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:39.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:39.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:40.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.075 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:40.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:40.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:45:40.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:40.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:40.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:40.324 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2024-04-25T05:45:37.902172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4238219827,"num_objects":64892,"num_object_clones":0,"num_object_copies":194676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3941,"num_objects_unfound":0,"num_objects_dirty":64892,"num_whiteouts":0,"num_read":2091,"num_read_kb":1811,"num_write":178876,"num_write_kb":7763950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34181,"num_bytes_recovered":2207985442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52677,"ondisk_log_size":52677,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":473,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":15089064,"kb_used_data":12153664,"kb_used_omap":7,"kb_used_meta":2935224,"kb_avail":453636696,"statfs":{"total":479975178240,"available":464523976704,"internally_reserved":0,"allocated":12445351936,"data_stored":12442537648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":3005669601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-497221632,"num_objects":-7587,"num_object_clones":0,"num_object_copies":-22761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-467,"num_objects_unfound":0,"num_objects_dirty":-7587,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.978042"},"pg_stats":[{"pgid":"9.1","version":"351'11149","reported_seq":11269,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.688936+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:45:33.688936+0000","last_peered":"2024-04-25T05:45:33.688936+0000","last_clean":"2024-04-25T05:45:33.688936+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:45:33.688936+0000","last_undegraded":"2024-04-25T05:45:33.688936+0000","last_fullsized":"2024-04-25T05:45:33.688936+0000","mapping_epoch":296,"log_start":"335'7987","ondisk_log_start":"335'7987","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262078464,"num_objects":3999,"num_object_clones":0,"num_object_copies":11997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11149,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":326,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976091+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:45:29.976091+0000","last_peered":"2024-04-25T05:45:29.976091+0000","last_clean":"2024-04-25T05:45:29.976091+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:45:29.976091+0000","last_undegraded":"2024-04-25T05:45:29.976091+0000","last_fullsized":"2024-04-25T05:45:29.976091+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":617,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977093+0000","last_change":"2024-04-25T05:44:28.173941+0000","last_active":"2024-04-25T05:45:29.977093+0000","last_peered":"2024-04-25T05:45:29.977093+0000","last_clean":"2024-04-25T05:45:29.977093+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:29.977093+0000","last_undegraded":"2024-04-25T05:45:29.977093+0000","last_fullsized":"2024-04-25T05:45:29.977093+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"325'12","reported_seq":419,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955453+0000","last_change":"2024-04-25T05:44:28.198744+0000","last_active":"2024-04-25T05:45:29.955453+0000","last_peered":"2024-04-25T05:45:29.955453+0000","last_clean":"2024-04-25T05:45:29.955453+0000","last_became_active":"2024-04-25T05:44:28.171810+0000","last_became_peered":"2024-04-25T05:44:28.171810+0000","last_unstale":"2024-04-25T05:45:29.955453+0000","last_undegraded":"2024-04-25T05:45:29.955453+0000","last_fullsized":"2024-04-25T05:45:29.955453+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'10","last_scrub_stamp":"2024-04-25T05:42:21.213432+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.213432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":289,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975347+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:45:29.975347+0000","last_peered":"2024-04-25T05:45:29.975347+0000","last_clean":"2024-04-25T05:45:29.975347+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:45:29.975347+0000","last_undegraded":"2024-04-25T05:45:29.975347+0000","last_fullsized":"2024-04-25T05:45:29.975347+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":923,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953117+0000","last_change":"2024-04-25T05:43:19.210320+0000","last_active":"2024-04-25T05:45:29.953117+0000","last_peered":"2024-04-25T05:45:29.953117+0000","last_clean":"2024-04-25T05:45:29.953117+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:45:29.953117+0000","last_undegraded":"2024-04-25T05:45:29.953117+0000","last_fullsized":"2024-04-25T05:45:29.953117+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:20.218608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.218608+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"351'11117","reported_seq":11251,"reported_epoch":351,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:45:33.800960+0000","last_change":"2024-04-25T05:45:29.228939+0000","last_active":"2024-04-25T05:45:33.800960+0000","last_peered":"2024-04-25T05:45:33.800960+0000","last_clean":"2024-04-25T05:44:27.095992+0000","last_became_active":"2024-04-25T05:44:29.189111+0000","last_became_peered":"2024-04-25T05:44:29.189111+0000","last_unstale":"2024-04-25T05:45:33.800960+0000","last_undegraded":"2024-04-25T05:45:33.800960+0000","last_fullsized":"2024-04-25T05:45:33.800960+0000","mapping_epoch":325,"log_start":"336'8187","ondisk_log_start":"336'8187","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2930,"log_dups_size":3000,"ondisk_log_size":2930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258277376,"num_objects":3941,"num_object_clones":0,"num_object_copies":11823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3941,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11117,"num_write_kb":481856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":3941}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":335,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974274+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:45:29.974274+0000","last_peered":"2024-04-25T05:45:29.974274+0000","last_clean":"2024-04-25T05:45:29.974274+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:45:29.974274+0000","last_undegraded":"2024-04-25T05:45:29.974274+0000","last_fullsized":"2024-04-25T05:45:29.974274+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":330,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971596+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:45:29.971596+0000","last_peered":"2024-04-25T05:45:29.971596+0000","last_clean":"2024-04-25T05:45:29.971596+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:45:29.971596+0000","last_undegraded":"2024-04-25T05:45:29.971596+0000","last_fullsized":"2024-04-25T05:45:29.971596+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.765264+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:45:33.765264+0000","last_peered":"2024-04-25T05:45:33.765264+0000","last_clean":"2024-04-25T05:45:33.765264+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:45:33.765264+0000","last_undegraded":"2024-04-25T05:45:33.765264+0000","last_fullsized":"2024-04-25T05:45:33.765264+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:45.261878+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":242,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.980026+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:45:29.980026+0000","last_peered":"2024-04-25T05:45:29.980026+0000","last_clean":"2024-04-25T05:45:29.980026+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:45:29.980026+0000","last_undegraded":"2024-04-25T05:45:29.980026+0000","last_fullsized":"2024-04-25T05:45:29.980026+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":523,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972071+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:45:29.972071+0000","last_peered":"2024-04-25T05:45:29.972071+0000","last_clean":"2024-04-25T05:45:29.972071+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:45:29.972071+0000","last_undegraded":"2024-04-25T05:45:29.972071+0000","last_fullsized":"2024-04-25T05:45:29.972071+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"351'11026","reported_seq":11145,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.972055+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:45:32.972055+0000","last_peered":"2024-04-25T05:45:32.972055+0000","last_clean":"2024-04-25T05:45:32.972055+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:45:32.972055+0000","last_undegraded":"2024-04-25T05:45:32.972055+0000","last_fullsized":"2024-04-25T05:45:32.972055+0000","mapping_epoch":296,"log_start":"333'7777","ondisk_log_start":"333'7777","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262275072,"num_objects":4002,"num_object_clones":0,"num_object_copies":12006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11026,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":643,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971781+0000","last_change":"2024-04-25T05:45:26.763116+0000","last_active":"2024-04-25T05:45:29.971781+0000","last_peered":"2024-04-25T05:45:29.971781+0000","last_clean":"2024-04-25T05:45:29.971781+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:45:29.971781+0000","last_undegraded":"2024-04-25T05:45:29.971781+0000","last_fullsized":"2024-04-25T05:45:29.971781+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:45:26.762873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:26.762873+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:48.188664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":633,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.191276+0000","last_change":"2024-04-25T05:44:46.503302+0000","last_active":"2024-04-25T05:45:36.191276+0000","last_peered":"2024-04-25T05:45:36.191276+0000","last_clean":"2024-04-25T05:45:36.191276+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:45:36.191276+0000","last_undegraded":"2024-04-25T05:45:36.191276+0000","last_fullsized":"2024-04-25T05:45:36.191276+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:46.503173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:46.503173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:56.738128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":429,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.195397+0000","last_change":"2024-04-25T05:44:57.457953+0000","last_active":"2024-04-25T05:45:36.195397+0000","last_peered":"2024-04-25T05:45:36.195397+0000","last_clean":"2024-04-25T05:45:36.195397+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:45:36.195397+0000","last_undegraded":"2024-04-25T05:45:36.195397+0000","last_fullsized":"2024-04-25T05:45:36.195397+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:44:57.457842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:18.204723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":605,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975710+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:45:29.975710+0000","last_peered":"2024-04-25T05:45:29.975710+0000","last_clean":"2024-04-25T05:45:29.975710+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:45:29.975710+0000","last_undegraded":"2024-04-25T05:45:29.975710+0000","last_fullsized":"2024-04-25T05:45:29.975710+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"351'11298","reported_seq":11418,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.687782+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:45:33.687782+0000","last_peered":"2024-04-25T05:45:33.687782+0000","last_clean":"2024-04-25T05:45:33.687782+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:45:33.687782+0000","last_undegraded":"2024-04-25T05:45:33.687782+0000","last_fullsized":"2024-04-25T05:45:33.687782+0000","mapping_epoch":296,"log_start":"335'8181","ondisk_log_start":"335'8181","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270008320,"num_objects":4120,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":653,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.980900+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:45:29.980900+0000","last_peered":"2024-04-25T05:45:29.980900+0000","last_clean":"2024-04-25T05:45:29.980900+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:45:29.980900+0000","last_undegraded":"2024-04-25T05:45:29.980900+0000","last_fullsized":"2024-04-25T05:45:29.980900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":607,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972665+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:45:29.972665+0000","last_peered":"2024-04-25T05:45:29.972665+0000","last_clean":"2024-04-25T05:45:29.972665+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:45:29.972665+0000","last_undegraded":"2024-04-25T05:45:29.972665+0000","last_fullsized":"2024-04-25T05:45:29.972665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"325'13","reported_seq":153,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975818+0000","last_change":"2024-04-25T05:45:28.975997+0000","last_active":"2024-04-25T05:45:29.975818+0000","last_peered":"2024-04-25T05:45:29.975818+0000","last_clean":"2024-04-25T05:45:29.975818+0000","last_became_active":"2024-04-25T05:44:28.178692+0000","last_became_peered":"2024-04-25T05:44:28.178692+0000","last_unstale":"2024-04-25T05:45:29.975818+0000","last_undegraded":"2024-04-25T05:45:29.975818+0000","last_fullsized":"2024-04-25T05:45:29.975818+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.085930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":477,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:31.796893+0000","last_change":"2024-04-25T05:45:31.796893+0000","last_active":"2024-04-25T05:45:31.796893+0000","last_peered":"2024-04-25T05:45:31.796893+0000","last_clean":"2024-04-25T05:45:31.796893+0000","last_became_active":"2024-04-25T05:44:28.177729+0000","last_became_peered":"2024-04-25T05:44:28.177729+0000","last_unstale":"2024-04-25T05:45:31.796893+0000","last_undegraded":"2024-04-25T05:45:31.796893+0000","last_fullsized":"2024-04-25T05:45:31.796893+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:53.427025+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":599,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971388+0000","last_change":"2024-04-25T05:44:30.932605+0000","last_active":"2024-04-25T05:45:29.971388+0000","last_peered":"2024-04-25T05:45:29.971388+0000","last_clean":"2024-04-25T05:45:29.971388+0000","last_became_active":"2024-04-25T05:44:30.932221+0000","last_became_peered":"2024-04-25T05:44:30.932221+0000","last_unstale":"2024-04-25T05:45:29.971388+0000","last_undegraded":"2024-04-25T05:45:29.971388+0000","last_fullsized":"2024-04-25T05:45:29.971388+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.344182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"351'11084","reported_seq":11211,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.958619+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:45:32.958619+0000","last_peered":"2024-04-25T05:45:32.958619+0000","last_clean":"2024-04-25T05:45:32.958619+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:45:32.958619+0000","last_undegraded":"2024-04-25T05:45:32.958619+0000","last_fullsized":"2024-04-25T05:45:32.958619+0000","mapping_epoch":301,"log_start":"333'7853","ondisk_log_start":"333'7853","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270598168,"num_objects":4130,"num_object_clones":0,"num_object_copies":12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11084,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":619,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.191186+0000","last_change":"2024-04-25T05:45:29.415240+0000","last_active":"2024-04-25T05:45:36.191186+0000","last_peered":"2024-04-25T05:45:36.191186+0000","last_clean":"2024-04-25T05:45:36.191186+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:45:36.191186+0000","last_undegraded":"2024-04-25T05:45:36.191186+0000","last_fullsized":"2024-04-25T05:45:36.191186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.415117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.415117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:43.082744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":761,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972758+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:45:29.972758+0000","last_peered":"2024-04-25T05:45:29.972758+0000","last_clean":"2024-04-25T05:45:29.972758+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:45:29.972758+0000","last_undegraded":"2024-04-25T05:45:29.972758+0000","last_fullsized":"2024-04-25T05:45:29.972758+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982347+0000","last_change":"2024-04-25T05:45:21.024234+0000","last_active":"2024-04-25T05:45:29.982347+0000","last_peered":"2024-04-25T05:45:29.982347+0000","last_clean":"2024-04-25T05:45:29.982347+0000","last_became_active":"2024-04-25T05:44:28.178969+0000","last_became_peered":"2024-04-25T05:44:28.178969+0000","last_unstale":"2024-04-25T05:45:29.982347+0000","last_undegraded":"2024-04-25T05:45:29.982347+0000","last_fullsized":"2024-04-25T05:45:29.982347+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:21.023940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:21.023940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:35.928250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":283,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976240+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:45:29.976240+0000","last_peered":"2024-04-25T05:45:29.976240+0000","last_clean":"2024-04-25T05:45:29.976240+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:45:29.976240+0000","last_undegraded":"2024-04-25T05:45:29.976240+0000","last_fullsized":"2024-04-25T05:45:29.976240+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":839,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.587473+0000","last_change":"2024-04-25T05:45:36.587515+0000","last_active":"2024-04-25T05:45:36.587473+0000","last_peered":"2024-04-25T05:45:36.587473+0000","last_clean":"2024-04-25T05:45:36.587473+0000","last_became_active":"2024-04-25T05:44:30.915718+0000","last_became_peered":"2024-04-25T05:44:30.915718+0000","last_unstale":"2024-04-25T05:45:36.587473+0000","last_undegraded":"2024-04-25T05:45:36.587473+0000","last_fullsized":"2024-04-25T05:45:36.587473+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:36.587388+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.587388+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:39.493160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"351'11326","reported_seq":11446,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.688001+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:45:33.688001+0000","last_peered":"2024-04-25T05:45:33.688001+0000","last_clean":"2024-04-25T05:45:33.688001+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:45:33.688001+0000","last_undegraded":"2024-04-25T05:45:33.688001+0000","last_fullsized":"2024-04-25T05:45:33.688001+0000","mapping_epoch":296,"log_start":"335'8216","ondisk_log_start":"335'8216","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3110,"log_dups_size":0,"ondisk_log_size":3110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":738,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972929+0000","last_change":"2024-04-25T05:42:09.358000+0000","last_active":"2024-04-25T05:45:29.972929+0000","last_peered":"2024-04-25T05:45:29.972929+0000","last_clean":"2024-04-25T05:45:29.972929+0000","last_became_active":"2024-04-25T05:34:02.701427+0000","last_became_peered":"2024-04-25T05:34:02.701427+0000","last_unstale":"2024-04-25T05:45:29.972929+0000","last_undegraded":"2024-04-25T05:45:29.972929+0000","last_fullsized":"2024-04-25T05:45:29.972929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:42:09.357875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:42:09.357875+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":559,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971859+0000","last_change":"2024-04-25T05:45:27.764855+0000","last_active":"2024-04-25T05:45:29.971859+0000","last_peered":"2024-04-25T05:45:29.971859+0000","last_clean":"2024-04-25T05:45:29.971859+0000","last_became_active":"2024-04-25T05:44:28.124913+0000","last_became_peered":"2024-04-25T05:44:28.124913+0000","last_unstale":"2024-04-25T05:45:29.971859+0000","last_undegraded":"2024-04-25T05:45:29.971859+0000","last_fullsized":"2024-04-25T05:45:29.971859+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:45:27.764695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.764695+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:40.827850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":311,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975480+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:45:29.975480+0000","last_peered":"2024-04-25T05:45:29.975480+0000","last_clean":"2024-04-25T05:45:29.975480+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:45:29.975480+0000","last_undegraded":"2024-04-25T05:45:29.975480+0000","last_fullsized":"2024-04-25T05:45:29.975480+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":240,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953409+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:45:29.953409+0000","last_peered":"2024-04-25T05:45:29.953409+0000","last_clean":"2024-04-25T05:45:29.953409+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:45:29.953409+0000","last_undegraded":"2024-04-25T05:45:29.953409+0000","last_fullsized":"2024-04-25T05:45:29.953409+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":787,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975556+0000","last_change":"2024-04-25T05:43:19.215636+0000","last_active":"2024-04-25T05:45:29.975556+0000","last_peered":"2024-04-25T05:45:29.975556+0000","last_clean":"2024-04-25T05:45:29.975556+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:45:29.975556+0000","last_undegraded":"2024-04-25T05:45:29.975556+0000","last_fullsized":"2024-04-25T05:45:29.975556+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:22.285287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.285287+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"351'11115","reported_seq":11226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.815548+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:45:33.815548+0000","last_peered":"2024-04-25T05:45:33.815548+0000","last_clean":"2024-04-25T05:45:33.815548+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:45:33.815548+0000","last_undegraded":"2024-04-25T05:45:33.815548+0000","last_fullsized":"2024-04-25T05:45:33.815548+0000","mapping_epoch":301,"log_start":"335'8138","ondisk_log_start":"335'8138","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2977,"log_dups_size":3000,"ondisk_log_size":2977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269156352,"num_objects":4107,"num_object_clones":0,"num_object_copies":12321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":329,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971513+0000","last_change":"2024-04-25T05:44:28.202828+0000","last_active":"2024-04-25T05:45:29.971513+0000","last_peered":"2024-04-25T05:45:29.971513+0000","last_clean":"2024-04-25T05:45:29.971513+0000","last_became_active":"2024-04-25T05:44:28.182043+0000","last_became_peered":"2024-04-25T05:44:28.182043+0000","last_unstale":"2024-04-25T05:45:29.971513+0000","last_undegraded":"2024-04-25T05:45:29.971513+0000","last_fullsized":"2024-04-25T05:45:29.971513+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":309,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.973210+0000","last_change":"2024-04-25T05:44:28.217831+0000","last_active":"2024-04-25T05:45:29.973210+0000","last_peered":"2024-04-25T05:45:29.973210+0000","last_clean":"2024-04-25T05:45:29.973210+0000","last_became_active":"2024-04-25T05:44:28.170458+0000","last_became_peered":"2024-04-25T05:44:28.170458+0000","last_unstale":"2024-04-25T05:45:29.973210+0000","last_undegraded":"2024-04-25T05:45:29.973210+0000","last_fullsized":"2024-04-25T05:45:29.973210+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":494,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.973230+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:45:29.973230+0000","last_peered":"2024-04-25T05:45:29.973230+0000","last_clean":"2024-04-25T05:45:29.973230+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:45:29.973230+0000","last_undegraded":"2024-04-25T05:45:29.973230+0000","last_fullsized":"2024-04-25T05:45:29.973230+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":233,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:31.048293+0000","last_change":"2024-04-25T05:45:31.048330+0000","last_active":"2024-04-25T05:45:31.048293+0000","last_peered":"2024-04-25T05:45:31.048293+0000","last_clean":"2024-04-25T05:45:31.048293+0000","last_became_active":"2024-04-25T05:44:28.168333+0000","last_became_peered":"2024-04-25T05:44:28.168333+0000","last_unstale":"2024-04-25T05:45:31.048293+0000","last_undegraded":"2024-04-25T05:45:31.048293+0000","last_fullsized":"2024-04-25T05:45:31.048293+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.047907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.314401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":785,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972015+0000","last_change":"2024-04-25T05:43:28.638040+0000","last_active":"2024-04-25T05:45:29.972015+0000","last_peered":"2024-04-25T05:45:29.972015+0000","last_clean":"2024-04-25T05:45:29.972015+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:45:29.972015+0000","last_undegraded":"2024-04-25T05:45:29.972015+0000","last_fullsized":"2024-04-25T05:45:29.972015+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:28.637882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:28.637882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"351'11073","reported_seq":11192,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.964120+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:45:32.964120+0000","last_peered":"2024-04-25T05:45:32.964120+0000","last_clean":"2024-04-25T05:45:32.964120+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:45:32.964120+0000","last_undegraded":"2024-04-25T05:45:32.964120+0000","last_fullsized":"2024-04-25T05:45:32.964120+0000","mapping_epoch":296,"log_start":"334'7830","ondisk_log_start":"334'7830","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258277376,"num_objects":3941,"num_object_clones":0,"num_object_copies":11823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11073,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":348,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971714+0000","last_change":"2024-04-25T05:44:28.125054+0000","last_active":"2024-04-25T05:45:29.971714+0000","last_peered":"2024-04-25T05:45:29.971714+0000","last_clean":"2024-04-25T05:45:29.971714+0000","last_became_active":"2024-04-25T05:44:28.124674+0000","last_became_peered":"2024-04-25T05:44:28.124674+0000","last_unstale":"2024-04-25T05:45:29.971714+0000","last_undegraded":"2024-04-25T05:45:29.971714+0000","last_fullsized":"2024-04-25T05:45:29.971714+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":337,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953696+0000","last_change":"2024-04-25T05:44:28.198134+0000","last_active":"2024-04-25T05:45:29.953696+0000","last_peered":"2024-04-25T05:45:29.953696+0000","last_clean":"2024-04-25T05:45:29.953696+0000","last_became_active":"2024-04-25T05:44:28.168410+0000","last_became_peered":"2024-04-25T05:44:28.168410+0000","last_unstale":"2024-04-25T05:45:29.953696+0000","last_undegraded":"2024-04-25T05:45:29.953696+0000","last_fullsized":"2024-04-25T05:45:29.953696+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972520+0000","last_change":"2024-04-25T05:44:28.167276+0000","last_active":"2024-04-25T05:45:29.972520+0000","last_peered":"2024-04-25T05:45:29.972520+0000","last_clean":"2024-04-25T05:45:29.972520+0000","last_became_active":"2024-04-25T05:44:28.166968+0000","last_became_peered":"2024-04-25T05:44:28.166968+0000","last_unstale":"2024-04-25T05:45:29.972520+0000","last_undegraded":"2024-04-25T05:45:29.972520+0000","last_fullsized":"2024-04-25T05:45:29.972520+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":613,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.969059+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:45:29.969059+0000","last_peered":"2024-04-25T05:45:29.969059+0000","last_clean":"2024-04-25T05:45:29.969059+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:45:29.969059+0000","last_undegraded":"2024-04-25T05:45:29.969059+0000","last_fullsized":"2024-04-25T05:45:29.969059+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":811,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971553+0000","last_change":"2024-04-25T05:44:30.918290+0000","last_active":"2024-04-25T05:45:29.971553+0000","last_peered":"2024-04-25T05:45:29.971553+0000","last_clean":"2024-04-25T05:45:29.971553+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:45:29.971553+0000","last_undegraded":"2024-04-25T05:45:29.971553+0000","last_fullsized":"2024-04-25T05:45:29.971553+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:20.780175+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.780175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:42.506221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"351'11002","reported_seq":24612,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.908714+0000","last_change":"2024-04-25T05:45:10.952360+0000","last_active":"2024-04-25T05:45:32.908714+0000","last_peered":"2024-04-25T05:45:32.908714+0000","last_clean":"2024-04-25T05:45:32.908714+0000","last_became_active":"2024-04-25T05:45:10.941174+0000","last_became_peered":"2024-04-25T05:45:10.941174+0000","last_unstale":"2024-04-25T05:45:32.908714+0000","last_undegraded":"2024-04-25T05:45:32.908714+0000","last_fullsized":"2024-04-25T05:45:32.908714+0000","mapping_epoch":343,"log_start":"331'7511","ondisk_log_start":"331'7511","created":296,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262012928,"num_objects":3998,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6735,"num_bytes_recovered":441384960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":482,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.978436+0000","last_change":"2024-04-25T05:44:28.180356+0000","last_active":"2024-04-25T05:45:29.978436+0000","last_peered":"2024-04-25T05:45:29.978436+0000","last_clean":"2024-04-25T05:45:29.978436+0000","last_became_active":"2024-04-25T05:44:28.180059+0000","last_became_peered":"2024-04-25T05:44:28.180059+0000","last_unstale":"2024-04-25T05:45:29.978436+0000","last_undegraded":"2024-04-25T05:45:29.978436+0000","last_fullsized":"2024-04-25T05:45:29.978436+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":155,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975770+0000","last_change":"2024-04-25T05:44:28.234082+0000","last_active":"2024-04-25T05:45:29.975770+0000","last_peered":"2024-04-25T05:45:29.975770+0000","last_clean":"2024-04-25T05:45:29.975770+0000","last_became_active":"2024-04-25T05:44:28.177715+0000","last_became_peered":"2024-04-25T05:44:28.177715+0000","last_unstale":"2024-04-25T05:45:29.975770+0000","last_undegraded":"2024-04-25T05:45:29.975770+0000","last_fullsized":"2024-04-25T05:45:29.975770+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:44.249745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1342,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:31.380456+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:45:31.380456+0000","last_peered":"2024-04-25T05:45:31.380456+0000","last_clean":"2024-04-25T05:45:31.380456+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:45:31.380456+0000","last_undegraded":"2024-04-25T05:45:31.380456+0000","last_fullsized":"2024-04-25T05:45:31.380456+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":676,"num_read_kb":671,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":238,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.191365+0000","last_change":"2024-04-25T05:45:30.434222+0000","last_active":"2024-04-25T05:45:36.191365+0000","last_peered":"2024-04-25T05:45:36.191365+0000","last_clean":"2024-04-25T05:45:36.191365+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:36.191365+0000","last_undegraded":"2024-04-25T05:45:36.191365+0000","last_fullsized":"2024-04-25T05:45:36.191365+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:30.434101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.434101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.921750+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":601,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972236+0000","last_change":"2024-04-25T05:43:16.189190+0000","last_active":"2024-04-25T05:45:29.972236+0000","last_peered":"2024-04-25T05:45:29.972236+0000","last_clean":"2024-04-25T05:45:29.972236+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:45:29.972236+0000","last_undegraded":"2024-04-25T05:45:29.972236+0000","last_fullsized":"2024-04-25T05:45:29.972236+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:40:47.469764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:40:47.469764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":789,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.973554+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:45:29.973554+0000","last_peered":"2024-04-25T05:45:29.973554+0000","last_clean":"2024-04-25T05:45:29.973554+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:45:29.973554+0000","last_undegraded":"2024-04-25T05:45:29.973554+0000","last_fullsized":"2024-04-25T05:45:29.973554+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"351'11067","reported_seq":24210,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.707121+0000","last_change":"2024-04-25T05:45:20.779526+0000","last_active":"2024-04-25T05:45:33.707121+0000","last_peered":"2024-04-25T05:45:33.707121+0000","last_clean":"2024-04-25T05:45:33.707121+0000","last_became_active":"2024-04-25T05:45:18.104956+0000","last_became_peered":"2024-04-25T05:45:18.104956+0000","last_unstale":"2024-04-25T05:45:33.707121+0000","last_undegraded":"2024-04-25T05:45:33.707121+0000","last_fullsized":"2024-04-25T05:45:33.707121+0000","mapping_epoch":347,"log_start":"334'7973","ondisk_log_start":"334'7973","created":296,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'9941","last_scrub_stamp":"2024-04-25T05:45:20.778952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:45:20.778952+0000","objects_scrubbed":5279,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:45.792628+0000","scrub_duration":1839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266928128,"num_objects":4073,"num_object_clones":0,"num_object_copies":12219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6470,"num_bytes_recovered":424017920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":625,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954236+0000","last_change":"2024-04-25T05:44:28.178984+0000","last_active":"2024-04-25T05:45:29.954236+0000","last_peered":"2024-04-25T05:45:29.954236+0000","last_clean":"2024-04-25T05:45:29.954236+0000","last_became_active":"2024-04-25T05:44:28.178614+0000","last_became_peered":"2024-04-25T05:44:28.178614+0000","last_unstale":"2024-04-25T05:45:29.954236+0000","last_undegraded":"2024-04-25T05:45:29.954236+0000","last_fullsized":"2024-04-25T05:45:29.954236+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":430,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972311+0000","last_change":"2024-04-25T05:43:22.635428+0000","last_active":"2024-04-25T05:45:29.972311+0000","last_peered":"2024-04-25T05:45:29.972311+0000","last_clean":"2024-04-25T05:45:29.972311+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:45:29.972311+0000","last_undegraded":"2024-04-25T05:45:29.972311+0000","last_fullsized":"2024-04-25T05:45:29.972311+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'19","last_scrub_stamp":"2024-04-25T05:43:22.635290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.635290+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":617,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.973157+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:45:29.973157+0000","last_peered":"2024-04-25T05:45:29.973157+0000","last_clean":"2024-04-25T05:45:29.973157+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:45:29.973157+0000","last_undegraded":"2024-04-25T05:45:29.973157+0000","last_fullsized":"2024-04-25T05:45:29.973157+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":924,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952959+0000","last_change":"2024-04-25T05:44:30.948007+0000","last_active":"2024-04-25T05:45:29.952959+0000","last_peered":"2024-04-25T05:45:29.952959+0000","last_clean":"2024-04-25T05:45:29.952959+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:45:29.952959+0000","last_undegraded":"2024-04-25T05:45:29.952959+0000","last_fullsized":"2024-04-25T05:45:29.952959+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":111,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982550+0000","last_change":"2024-04-25T05:43:16.188274+0000","last_active":"2024-04-25T05:45:29.982550+0000","last_peered":"2024-04-25T05:45:29.982550+0000","last_clean":"2024-04-25T05:45:29.982550+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:45:29.982550+0000","last_undegraded":"2024-04-25T05:45:29.982550+0000","last_fullsized":"2024-04-25T05:45:29.982550+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:15.011340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.011340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"351'11358","reported_seq":11469,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.795723+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:45:33.795723+0000","last_peered":"2024-04-25T05:45:33.795723+0000","last_clean":"2024-04-25T05:45:33.795723+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:45:33.795723+0000","last_undegraded":"2024-04-25T05:45:33.795723+0000","last_fullsized":"2024-04-25T05:45:33.795723+0000","mapping_epoch":301,"log_start":"338'8470","ondisk_log_start":"338'8470","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":3000,"ondisk_log_size":2888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268435456,"num_objects":4096,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11358,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":121,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954484+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:45:29.954484+0000","last_peered":"2024-04-25T05:45:29.954484+0000","last_clean":"2024-04-25T05:45:29.954484+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:45:29.954484+0000","last_undegraded":"2024-04-25T05:45:29.954484+0000","last_fullsized":"2024-04-25T05:45:29.954484+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":734,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.978035+0000","last_change":"2024-04-25T05:44:28.221414+0000","last_active":"2024-04-25T05:45:29.978035+0000","last_peered":"2024-04-25T05:45:29.978035+0000","last_clean":"2024-04-25T05:45:29.978035+0000","last_became_active":"2024-04-25T05:44:28.179973+0000","last_became_peered":"2024-04-25T05:44:28.179973+0000","last_unstale":"2024-04-25T05:45:29.978035+0000","last_undegraded":"2024-04-25T05:45:29.978035+0000","last_fullsized":"2024-04-25T05:45:29.978035+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":627,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976371+0000","last_change":"2024-04-25T05:44:28.186329+0000","last_active":"2024-04-25T05:45:29.976371+0000","last_peered":"2024-04-25T05:45:29.976371+0000","last_clean":"2024-04-25T05:45:29.976371+0000","last_became_active":"2024-04-25T05:44:28.184251+0000","last_became_peered":"2024-04-25T05:44:28.184251+0000","last_unstale":"2024-04-25T05:45:29.976371+0000","last_undegraded":"2024-04-25T05:45:29.976371+0000","last_fullsized":"2024-04-25T05:45:29.976371+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":645,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952406+0000","last_change":"2024-04-25T05:44:28.183966+0000","last_active":"2024-04-25T05:45:29.952406+0000","last_peered":"2024-04-25T05:45:29.952406+0000","last_clean":"2024-04-25T05:45:29.952406+0000","last_became_active":"2024-04-25T05:44:28.183528+0000","last_became_peered":"2024-04-25T05:44:28.183528+0000","last_unstale":"2024-04-25T05:45:29.952406+0000","last_undegraded":"2024-04-25T05:45:29.952406+0000","last_fullsized":"2024-04-25T05:45:29.952406+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:35.104924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":615,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955533+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:45:29.955533+0000","last_peered":"2024-04-25T05:45:29.955533+0000","last_clean":"2024-04-25T05:45:29.955533+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:45:29.955533+0000","last_undegraded":"2024-04-25T05:45:29.955533+0000","last_fullsized":"2024-04-25T05:45:29.955533+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":931,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:30.250183+0000","last_change":"2024-04-25T05:45:30.250224+0000","last_active":"2024-04-25T05:45:30.250183+0000","last_peered":"2024-04-25T05:45:30.250183+0000","last_clean":"2024-04-25T05:45:30.250183+0000","last_became_active":"2024-04-25T05:44:30.946674+0000","last_became_peered":"2024-04-25T05:44:30.946674+0000","last_unstale":"2024-04-25T05:45:30.250183+0000","last_undegraded":"2024-04-25T05:45:30.250183+0000","last_fullsized":"2024-04-25T05:45:30.250183+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.319675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"351'11118","reported_seq":25380,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.968581+0000","last_change":"2024-04-25T05:44:47.251341+0000","last_active":"2024-04-25T05:45:32.968581+0000","last_peered":"2024-04-25T05:45:32.968581+0000","last_clean":"2024-04-25T05:45:32.968581+0000","last_became_active":"2024-04-25T05:44:47.245906+0000","last_became_peered":"2024-04-25T05:44:47.245906+0000","last_unstale":"2024-04-25T05:45:32.968581+0000","last_undegraded":"2024-04-25T05:45:32.968581+0000","last_fullsized":"2024-04-25T05:45:32.968581+0000","mapping_epoch":332,"log_start":"333'7808","ondisk_log_start":"333'7808","created":296,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267649024,"num_objects":4084,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11118,"num_write_kb":486464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7061,"num_bytes_recovered":462749696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":240,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974580+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:45:29.974580+0000","last_peered":"2024-04-25T05:45:29.974580+0000","last_clean":"2024-04-25T05:45:29.974580+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:45:29.974580+0000","last_undegraded":"2024-04-25T05:45:29.974580+0000","last_fullsized":"2024-04-25T05:45:29.974580+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.475952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":861,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972669+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:45:29.972669+0000","last_peered":"2024-04-25T05:45:29.972669+0000","last_clean":"2024-04-25T05:45:29.972669+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:45:29.972669+0000","last_undegraded":"2024-04-25T05:45:29.972669+0000","last_fullsized":"2024-04-25T05:45:29.972669+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":402,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971327+0000","last_change":"2024-04-25T05:44:28.160664+0000","last_active":"2024-04-25T05:45:29.971327+0000","last_peered":"2024-04-25T05:45:29.971327+0000","last_clean":"2024-04-25T05:45:29.971327+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:45:29.971327+0000","last_undegraded":"2024-04-25T05:45:29.971327+0000","last_fullsized":"2024-04-25T05:45:29.971327+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:30.320979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":147,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971574+0000","last_change":"2024-04-25T05:44:28.205511+0000","last_active":"2024-04-25T05:45:29.971574+0000","last_peered":"2024-04-25T05:45:29.971574+0000","last_clean":"2024-04-25T05:45:29.971574+0000","last_became_active":"2024-04-25T05:44:28.176104+0000","last_became_peered":"2024-04-25T05:44:28.176104+0000","last_unstale":"2024-04-25T05:45:29.971574+0000","last_undegraded":"2024-04-25T05:45:29.971574+0000","last_fullsized":"2024-04-25T05:45:29.971574+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":587,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976685+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:45:29.976685+0000","last_peered":"2024-04-25T05:45:29.976685+0000","last_clean":"2024-04-25T05:45:29.976685+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:45:29.976685+0000","last_undegraded":"2024-04-25T05:45:29.976685+0000","last_fullsized":"2024-04-25T05:45:29.976685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":631,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.190946+0000","last_change":"2024-04-25T05:44:30.918203+0000","last_active":"2024-04-25T05:45:36.190946+0000","last_peered":"2024-04-25T05:45:36.190946+0000","last_clean":"2024-04-25T05:45:36.190946+0000","last_became_active":"2024-04-25T05:44:30.917713+0000","last_became_peered":"2024-04-25T05:44:30.917713+0000","last_unstale":"2024-04-25T05:45:36.190946+0000","last_undegraded":"2024-04-25T05:45:36.190946+0000","last_fullsized":"2024-04-25T05:45:36.190946+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"351'11122","reported_seq":24392,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.809774+0000","last_change":"2024-04-25T05:45:22.530073+0000","last_active":"2024-04-25T05:45:33.809774+0000","last_peered":"2024-04-25T05:45:33.809774+0000","last_clean":"2024-04-25T05:45:33.809774+0000","last_became_active":"2024-04-25T05:45:17.505735+0000","last_became_peered":"2024-04-25T05:45:17.505735+0000","last_unstale":"2024-04-25T05:45:33.809774+0000","last_undegraded":"2024-04-25T05:45:33.809774+0000","last_fullsized":"2024-04-25T05:45:33.809774+0000","mapping_epoch":346,"log_start":"336'8191","ondisk_log_start":"336'8191","created":296,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10068","last_scrub_stamp":"2024-04-25T05:45:22.529746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.529746+0000","objects_scrubbed":5089,"log_size":2931,"log_dups_size":3000,"ondisk_log_size":2931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:27.120793+0000","scrub_duration":1818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258867200,"num_objects":3950,"num_object_clones":0,"num_object_copies":11850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11122,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":428539904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":325,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971459+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:45:29.971459+0000","last_peered":"2024-04-25T05:45:29.971459+0000","last_clean":"2024-04-25T05:45:29.971459+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:45:29.971459+0000","last_undegraded":"2024-04-25T05:45:29.971459+0000","last_fullsized":"2024-04-25T05:45:29.971459+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":425,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975941+0000","last_change":"2024-04-25T05:45:27.997352+0000","last_active":"2024-04-25T05:45:29.975941+0000","last_peered":"2024-04-25T05:45:29.975941+0000","last_clean":"2024-04-25T05:45:29.975941+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:45:29.975941+0000","last_undegraded":"2024-04-25T05:45:29.975941+0000","last_fullsized":"2024-04-25T05:45:29.975941+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.581422+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":268,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.192516+0000","last_change":"2024-04-25T05:45:32.512149+0000","last_active":"2024-04-25T05:45:36.192516+0000","last_peered":"2024-04-25T05:45:36.192516+0000","last_clean":"2024-04-25T05:45:36.192516+0000","last_became_active":"2024-04-25T05:44:28.171004+0000","last_became_peered":"2024-04-25T05:44:28.171004+0000","last_unstale":"2024-04-25T05:45:36.192516+0000","last_undegraded":"2024-04-25T05:45:36.192516+0000","last_fullsized":"2024-04-25T05:45:36.192516+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:00.371199+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953185+0000","last_change":"2024-04-25T05:44:28.169508+0000","last_active":"2024-04-25T05:45:29.953185+0000","last_peered":"2024-04-25T05:45:29.953185+0000","last_clean":"2024-04-25T05:45:29.953185+0000","last_became_active":"2024-04-25T05:44:28.169146+0000","last_became_peered":"2024-04-25T05:44:28.169146+0000","last_unstale":"2024-04-25T05:45:29.953185+0000","last_undegraded":"2024-04-25T05:45:29.953185+0000","last_fullsized":"2024-04-25T05:45:29.953185+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"351'10861","reported_seq":10972,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.958274+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:45:32.958274+0000","last_peered":"2024-04-25T05:45:32.958274+0000","last_clean":"2024-04-25T05:45:32.958274+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:45:32.958274+0000","last_undegraded":"2024-04-25T05:45:32.958274+0000","last_fullsized":"2024-04-25T05:45:32.958274+0000","mapping_epoch":301,"log_start":"333'7555","ondisk_log_start":"333'7555","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3306,"log_dups_size":0,"ondisk_log_size":3306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265486336,"num_objects":4051,"num_object_clones":0,"num_object_copies":12153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10861,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":376,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971495+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:45:29.971495+0000","last_peered":"2024-04-25T05:45:29.971495+0000","last_clean":"2024-04-25T05:45:29.971495+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:45:29.971495+0000","last_undegraded":"2024-04-25T05:45:29.971495+0000","last_fullsized":"2024-04-25T05:45:29.971495+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":651,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955477+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:45:29.955477+0000","last_peered":"2024-04-25T05:45:29.955477+0000","last_clean":"2024-04-25T05:45:29.955477+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:45:29.955477+0000","last_undegraded":"2024-04-25T05:45:29.955477+0000","last_fullsized":"2024-04-25T05:45:29.955477+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":559,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953306+0000","last_change":"2024-04-25T05:44:28.180755+0000","last_active":"2024-04-25T05:45:29.953306+0000","last_peered":"2024-04-25T05:45:29.953306+0000","last_clean":"2024-04-25T05:45:29.953306+0000","last_became_active":"2024-04-25T05:44:28.180393+0000","last_became_peered":"2024-04-25T05:44:28.180393+0000","last_unstale":"2024-04-25T05:45:29.953306+0000","last_undegraded":"2024-04-25T05:45:29.953306+0000","last_fullsized":"2024-04-25T05:45:29.953306+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":609,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952814+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:45:29.952814+0000","last_peered":"2024-04-25T05:45:29.952814+0000","last_clean":"2024-04-25T05:45:29.952814+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:45:29.952814+0000","last_undegraded":"2024-04-25T05:45:29.952814+0000","last_fullsized":"2024-04-25T05:45:29.952814+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"351'11218","reported_seq":11338,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:33.802106+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:45:33.802106+0000","last_peered":"2024-04-25T05:45:33.802106+0000","last_clean":"2024-04-25T05:45:33.802106+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:45:33.802106+0000","last_undegraded":"2024-04-25T05:45:33.802106+0000","last_fullsized":"2024-04-25T05:45:33.802106+0000","mapping_epoch":296,"log_start":"336'8286","ondisk_log_start":"336'8286","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":3000,"ondisk_log_size":2932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268435456,"num_objects":4096,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11218,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":560,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.970311+0000","last_change":"2024-04-25T05:44:28.169954+0000","last_active":"2024-04-25T05:45:29.970311+0000","last_peered":"2024-04-25T05:45:29.970311+0000","last_clean":"2024-04-25T05:45:29.970311+0000","last_became_active":"2024-04-25T05:44:28.168753+0000","last_became_peered":"2024-04-25T05:44:28.168753+0000","last_unstale":"2024-04-25T05:45:29.970311+0000","last_undegraded":"2024-04-25T05:45:29.970311+0000","last_fullsized":"2024-04-25T05:45:29.970311+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":265,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952576+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:45:29.952576+0000","last_peered":"2024-04-25T05:45:29.952576+0000","last_clean":"2024-04-25T05:45:29.952576+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:45:29.952576+0000","last_undegraded":"2024-04-25T05:45:29.952576+0000","last_fullsized":"2024-04-25T05:45:29.952576+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":615,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954310+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:45:29.954310+0000","last_peered":"2024-04-25T05:45:29.954310+0000","last_clean":"2024-04-25T05:45:29.954310+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:45:29.954310+0000","last_undegraded":"2024-04-25T05:45:29.954310+0000","last_fullsized":"2024-04-25T05:45:29.954310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":599,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977244+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:45:29.977244+0000","last_peered":"2024-04-25T05:45:29.977244+0000","last_clean":"2024-04-25T05:45:29.977244+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:45:29.977244+0000","last_undegraded":"2024-04-25T05:45:29.977244+0000","last_fullsized":"2024-04-25T05:45:29.977244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"351'11028","reported_seq":24932,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.961717+0000","last_change":"2024-04-25T05:45:09.944174+0000","last_active":"2024-04-25T05:45:32.961717+0000","last_peered":"2024-04-25T05:45:32.961717+0000","last_clean":"2024-04-25T05:45:32.961717+0000","last_became_active":"2024-04-25T05:45:09.927273+0000","last_became_peered":"2024-04-25T05:45:09.927273+0000","last_unstale":"2024-04-25T05:45:32.961717+0000","last_undegraded":"2024-04-25T05:45:32.961717+0000","last_fullsized":"2024-04-25T05:45:32.961717+0000","mapping_epoch":342,"log_start":"333'7793","ondisk_log_start":"333'7793","created":296,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11028,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6881,"num_bytes_recovered":450953216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":778,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953553+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:45:29.953553+0000","last_peered":"2024-04-25T05:45:29.953553+0000","last_clean":"2024-04-25T05:45:29.953553+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:45:29.953553+0000","last_undegraded":"2024-04-25T05:45:29.953553+0000","last_fullsized":"2024-04-25T05:45:29.953553+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":217,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953206+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:45:29.953206+0000","last_peered":"2024-04-25T05:45:29.953206+0000","last_clean":"2024-04-25T05:45:29.953206+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:45:29.953206+0000","last_undegraded":"2024-04-25T05:45:29.953206+0000","last_fullsized":"2024-04-25T05:45:29.953206+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":663,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954405+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:45:29.954405+0000","last_peered":"2024-04-25T05:45:29.954405+0000","last_clean":"2024-04-25T05:45:29.954405+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:45:29.954405+0000","last_undegraded":"2024-04-25T05:45:29.954405+0000","last_fullsized":"2024-04-25T05:45:29.954405+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":490,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971274+0000","last_change":"2024-04-25T05:44:28.174865+0000","last_active":"2024-04-25T05:45:29.971274+0000","last_peered":"2024-04-25T05:45:29.971274+0000","last_clean":"2024-04-25T05:45:29.971274+0000","last_became_active":"2024-04-25T05:44:28.174499+0000","last_became_peered":"2024-04-25T05:44:28.174499+0000","last_unstale":"2024-04-25T05:45:29.971274+0000","last_undegraded":"2024-04-25T05:45:29.971274+0000","last_fullsized":"2024-04-25T05:45:29.971274+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982834+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:45:29.982834+0000","last_peered":"2024-04-25T05:45:29.982834+0000","last_clean":"2024-04-25T05:45:29.982834+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:45:29.982834+0000","last_undegraded":"2024-04-25T05:45:29.982834+0000","last_fullsized":"2024-04-25T05:45:29.982834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:31.253497+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":348,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.970882+0000","last_change":"2024-04-25T05:44:28.191586+0000","last_active":"2024-04-25T05:45:29.970882+0000","last_peered":"2024-04-25T05:45:29.970882+0000","last_clean":"2024-04-25T05:45:29.970882+0000","last_became_active":"2024-04-25T05:44:28.191288+0000","last_became_peered":"2024-04-25T05:44:28.191288+0000","last_unstale":"2024-04-25T05:45:29.970882+0000","last_undegraded":"2024-04-25T05:45:29.970882+0000","last_fullsized":"2024-04-25T05:45:29.970882+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.539924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":118,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976519+0000","last_change":"2024-04-25T05:44:19.931291+0000","last_active":"2024-04-25T05:45:29.976519+0000","last_peered":"2024-04-25T05:45:29.976519+0000","last_clean":"2024-04-25T05:45:29.976519+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:45:29.976519+0000","last_undegraded":"2024-04-25T05:45:29.976519+0000","last_fullsized":"2024-04-25T05:45:29.976519+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":614,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955384+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:45:29.955384+0000","last_peered":"2024-04-25T05:45:29.955384+0000","last_clean":"2024-04-25T05:45:29.955384+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:45:29.955384+0000","last_undegraded":"2024-04-25T05:45:29.955384+0000","last_fullsized":"2024-04-25T05:45:29.955384+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":293,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976587+0000","last_change":"2024-04-25T05:44:28.203127+0000","last_active":"2024-04-25T05:45:29.976587+0000","last_peered":"2024-04-25T05:45:29.976587+0000","last_clean":"2024-04-25T05:45:29.976587+0000","last_became_active":"2024-04-25T05:44:28.184563+0000","last_became_peered":"2024-04-25T05:44:28.184563+0000","last_unstale":"2024-04-25T05:45:29.976587+0000","last_undegraded":"2024-04-25T05:45:29.976587+0000","last_fullsized":"2024-04-25T05:45:29.976587+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953384+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:45:29.953384+0000","last_peered":"2024-04-25T05:45:29.953384+0000","last_clean":"2024-04-25T05:45:29.953384+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:45:29.953384+0000","last_undegraded":"2024-04-25T05:45:29.953384+0000","last_fullsized":"2024-04-25T05:45:29.953384+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":590,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982647+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:45:29.982647+0000","last_peered":"2024-04-25T05:45:29.982647+0000","last_clean":"2024-04-25T05:45:29.982647+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:45:29.982647+0000","last_undegraded":"2024-04-25T05:45:29.982647+0000","last_fullsized":"2024-04-25T05:45:29.982647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:33.522496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":261,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.192882+0000","last_change":"2024-04-25T05:44:28.120205+0000","last_active":"2024-04-25T05:45:36.192882+0000","last_peered":"2024-04-25T05:45:36.192882+0000","last_clean":"2024-04-25T05:45:36.192882+0000","last_became_active":"2024-04-25T05:44:28.119749+0000","last_became_peered":"2024-04-25T05:44:28.119749+0000","last_unstale":"2024-04-25T05:45:36.192882+0000","last_undegraded":"2024-04-25T05:45:36.192882+0000","last_fullsized":"2024-04-25T05:45:36.192882+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":235,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955510+0000","last_change":"2024-04-25T05:43:16.229957+0000","last_active":"2024-04-25T05:45:29.955510+0000","last_peered":"2024-04-25T05:45:29.955510+0000","last_clean":"2024-04-25T05:45:29.955510+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:45:29.955510+0000","last_undegraded":"2024-04-25T05:45:29.955510+0000","last_fullsized":"2024-04-25T05:45:29.955510+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'20","last_scrub_stamp":"2024-04-25T05:42:08.829275+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.829275+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953388+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:45:29.953388+0000","last_peered":"2024-04-25T05:45:29.953388+0000","last_clean":"2024-04-25T05:45:29.953388+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:45:29.953388+0000","last_undegraded":"2024-04-25T05:45:29.953388+0000","last_fullsized":"2024-04-25T05:45:29.953388+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":747,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976324+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:45:29.976324+0000","last_peered":"2024-04-25T05:45:29.976324+0000","last_clean":"2024-04-25T05:45:29.976324+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:45:29.976324+0000","last_undegraded":"2024-04-25T05:45:29.976324+0000","last_fullsized":"2024-04-25T05:45:29.976324+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":70,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953952+0000","last_change":"2024-04-25T05:44:28.167266+0000","last_active":"2024-04-25T05:45:29.953952+0000","last_peered":"2024-04-25T05:45:29.953952+0000","last_clean":"2024-04-25T05:45:29.953952+0000","last_became_active":"2024-04-25T05:44:28.166838+0000","last_became_peered":"2024-04-25T05:44:28.166838+0000","last_unstale":"2024-04-25T05:45:29.953952+0000","last_undegraded":"2024-04-25T05:45:29.953952+0000","last_fullsized":"2024-04-25T05:45:29.953952+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":656,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.015410+0000","last_change":"2024-04-25T05:45:32.015466+0000","last_active":"2024-04-25T05:45:32.015410+0000","last_peered":"2024-04-25T05:45:32.015410+0000","last_clean":"2024-04-25T05:45:32.015410+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:45:32.015410+0000","last_undegraded":"2024-04-25T05:45:32.015410+0000","last_fullsized":"2024-04-25T05:45:32.015410+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:45:32.015242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.015242+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:44.736841+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":620,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953417+0000","last_change":"2024-04-25T05:44:13.015441+0000","last_active":"2024-04-25T05:45:29.953417+0000","last_peered":"2024-04-25T05:45:29.953417+0000","last_clean":"2024-04-25T05:45:29.953417+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:45:29.953417+0000","last_undegraded":"2024-04-25T05:45:29.953417+0000","last_fullsized":"2024-04-25T05:45:29.953417+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":176,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974364+0000","last_change":"2024-04-25T05:44:28.236616+0000","last_active":"2024-04-25T05:45:29.974364+0000","last_peered":"2024-04-25T05:45:29.974364+0000","last_clean":"2024-04-25T05:45:29.974364+0000","last_became_active":"2024-04-25T05:44:28.184915+0000","last_became_peered":"2024-04-25T05:44:28.184915+0000","last_unstale":"2024-04-25T05:45:29.974364+0000","last_undegraded":"2024-04-25T05:45:29.974364+0000","last_fullsized":"2024-04-25T05:45:29.974364+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954074+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:45:29.954074+0000","last_peered":"2024-04-25T05:45:29.954074+0000","last_clean":"2024-04-25T05:45:29.954074+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:45:29.954074+0000","last_undegraded":"2024-04-25T05:45:29.954074+0000","last_fullsized":"2024-04-25T05:45:29.954074+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":730,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.192914+0000","last_change":"2024-04-25T05:44:19.254245+0000","last_active":"2024-04-25T05:45:36.192914+0000","last_peered":"2024-04-25T05:45:36.192914+0000","last_clean":"2024-04-25T05:45:36.192914+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:45:36.192914+0000","last_undegraded":"2024-04-25T05:45:36.192914+0000","last_fullsized":"2024-04-25T05:45:36.192914+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:19.254124+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.254124+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:41.554540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":286,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953809+0000","last_change":"2024-04-25T05:42:08.182448+0000","last_active":"2024-04-25T05:45:29.953809+0000","last_peered":"2024-04-25T05:45:29.953809+0000","last_clean":"2024-04-25T05:45:29.953809+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:45:29.953809+0000","last_undegraded":"2024-04-25T05:45:29.953809+0000","last_fullsized":"2024-04-25T05:45:29.953809+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:08.181630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.181630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":607,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974724+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:45:29.974724+0000","last_peered":"2024-04-25T05:45:29.974724+0000","last_clean":"2024-04-25T05:45:29.974724+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:45:29.974724+0000","last_undegraded":"2024-04-25T05:45:29.974724+0000","last_fullsized":"2024-04-25T05:45:29.974724+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":618,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972576+0000","last_change":"2024-04-25T05:44:28.186508+0000","last_active":"2024-04-25T05:45:29.972576+0000","last_peered":"2024-04-25T05:45:29.972576+0000","last_clean":"2024-04-25T05:45:29.972576+0000","last_became_active":"2024-04-25T05:44:28.184914+0000","last_became_peered":"2024-04-25T05:44:28.184914+0000","last_unstale":"2024-04-25T05:45:29.972576+0000","last_undegraded":"2024-04-25T05:45:29.972576+0000","last_fullsized":"2024-04-25T05:45:29.972576+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":144,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.970830+0000","last_change":"2024-04-25T05:44:28.208196+0000","last_active":"2024-04-25T05:45:29.970830+0000","last_peered":"2024-04-25T05:45:29.970830+0000","last_clean":"2024-04-25T05:45:29.970830+0000","last_became_active":"2024-04-25T05:44:28.160392+0000","last_became_peered":"2024-04-25T05:44:28.160392+0000","last_unstale":"2024-04-25T05:45:29.970830+0000","last_undegraded":"2024-04-25T05:45:29.970830+0000","last_fullsized":"2024-04-25T05:45:29.970830+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":614,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.955386+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:45:29.955386+0000","last_peered":"2024-04-25T05:45:29.955386+0000","last_clean":"2024-04-25T05:45:29.955386+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:45:29.955386+0000","last_undegraded":"2024-04-25T05:45:29.955386+0000","last_fullsized":"2024-04-25T05:45:29.955386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":608,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977149+0000","last_change":"2024-04-25T05:44:28.174271+0000","last_active":"2024-04-25T05:45:29.977149+0000","last_peered":"2024-04-25T05:45:29.977149+0000","last_clean":"2024-04-25T05:45:29.977149+0000","last_became_active":"2024-04-25T05:44:28.172179+0000","last_became_peered":"2024-04-25T05:44:28.172179+0000","last_unstale":"2024-04-25T05:45:29.977149+0000","last_undegraded":"2024-04-25T05:45:29.977149+0000","last_fullsized":"2024-04-25T05:45:29.977149+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":219,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952660+0000","last_change":"2024-04-25T05:44:28.179459+0000","last_active":"2024-04-25T05:45:29.952660+0000","last_peered":"2024-04-25T05:45:29.952660+0000","last_clean":"2024-04-25T05:45:29.952660+0000","last_became_active":"2024-04-25T05:44:28.176134+0000","last_became_peered":"2024-04-25T05:44:28.176134+0000","last_unstale":"2024-04-25T05:45:29.952660+0000","last_undegraded":"2024-04-25T05:45:29.952660+0000","last_fullsized":"2024-04-25T05:45:29.952660+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":685,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972839+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:45:29.972839+0000","last_peered":"2024-04-25T05:45:29.972839+0000","last_clean":"2024-04-25T05:45:29.972839+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:45:29.972839+0000","last_undegraded":"2024-04-25T05:45:29.972839+0000","last_fullsized":"2024-04-25T05:45:29.972839+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":118,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954278+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:45:29.954278+0000","last_peered":"2024-04-25T05:45:29.954278+0000","last_clean":"2024-04-25T05:45:29.954278+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:45:29.954278+0000","last_undegraded":"2024-04-25T05:45:29.954278+0000","last_fullsized":"2024-04-25T05:45:29.954278+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":624,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953193+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:45:29.953193+0000","last_peered":"2024-04-25T05:45:29.953193+0000","last_clean":"2024-04-25T05:45:29.953193+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:45:29.953193+0000","last_undegraded":"2024-04-25T05:45:29.953193+0000","last_fullsized":"2024-04-25T05:45:29.953193+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":616,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977990+0000","last_change":"2024-04-25T05:44:28.179442+0000","last_active":"2024-04-25T05:45:29.977990+0000","last_peered":"2024-04-25T05:45:29.977990+0000","last_clean":"2024-04-25T05:45:29.977990+0000","last_became_active":"2024-04-25T05:44:28.179103+0000","last_became_peered":"2024-04-25T05:44:28.179103+0000","last_unstale":"2024-04-25T05:45:29.977990+0000","last_undegraded":"2024-04-25T05:45:29.977990+0000","last_fullsized":"2024-04-25T05:45:29.977990+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":638,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972562+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:45:29.972562+0000","last_peered":"2024-04-25T05:45:29.972562+0000","last_clean":"2024-04-25T05:45:29.972562+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:45:29.972562+0000","last_undegraded":"2024-04-25T05:45:29.972562+0000","last_fullsized":"2024-04-25T05:45:29.972562+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:44.810725+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":211,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982260+0000","last_change":"2024-04-25T05:44:28.188135+0000","last_active":"2024-04-25T05:45:29.982260+0000","last_peered":"2024-04-25T05:45:29.982260+0000","last_clean":"2024-04-25T05:45:29.982260+0000","last_became_active":"2024-04-25T05:44:28.187806+0000","last_became_peered":"2024-04-25T05:44:28.187806+0000","last_unstale":"2024-04-25T05:45:29.982260+0000","last_undegraded":"2024-04-25T05:45:29.982260+0000","last_fullsized":"2024-04-25T05:45:29.982260+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":330,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972131+0000","last_change":"2024-04-25T05:45:29.823062+0000","last_active":"2024-04-25T05:45:29.972131+0000","last_peered":"2024-04-25T05:45:29.972131+0000","last_clean":"2024-04-25T05:45:29.972131+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:45:29.972131+0000","last_undegraded":"2024-04-25T05:45:29.972131+0000","last_fullsized":"2024-04-25T05:45:29.972131+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.822937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.473503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":613,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954050+0000","last_change":"2024-04-25T05:44:19.960606+0000","last_active":"2024-04-25T05:45:29.954050+0000","last_peered":"2024-04-25T05:45:29.954050+0000","last_clean":"2024-04-25T05:45:29.954050+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:45:29.954050+0000","last_undegraded":"2024-04-25T05:45:29.954050+0000","last_fullsized":"2024-04-25T05:45:29.954050+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":402,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972164+0000","last_change":"2024-04-25T05:44:28.212080+0000","last_active":"2024-04-25T05:45:29.972164+0000","last_peered":"2024-04-25T05:45:29.972164+0000","last_clean":"2024-04-25T05:45:29.972164+0000","last_became_active":"2024-04-25T05:44:28.181996+0000","last_became_peered":"2024-04-25T05:44:28.181996+0000","last_unstale":"2024-04-25T05:45:29.972164+0000","last_undegraded":"2024-04-25T05:45:29.972164+0000","last_fullsized":"2024-04-25T05:45:29.972164+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":624,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972402+0000","last_change":"2024-04-25T05:44:28.172975+0000","last_active":"2024-04-25T05:45:29.972402+0000","last_peered":"2024-04-25T05:45:29.972402+0000","last_clean":"2024-04-25T05:45:29.972402+0000","last_became_active":"2024-04-25T05:44:28.171311+0000","last_became_peered":"2024-04-25T05:44:28.171311+0000","last_unstale":"2024-04-25T05:45:29.972402+0000","last_undegraded":"2024-04-25T05:45:29.972402+0000","last_fullsized":"2024-04-25T05:45:29.972402+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:37.398601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954385+0000","last_change":"2024-04-25T05:44:28.179599+0000","last_active":"2024-04-25T05:45:29.954385+0000","last_peered":"2024-04-25T05:45:29.954385+0000","last_clean":"2024-04-25T05:45:29.954385+0000","last_became_active":"2024-04-25T05:44:28.178710+0000","last_became_peered":"2024-04-25T05:44:28.178710+0000","last_unstale":"2024-04-25T05:45:29.954385+0000","last_undegraded":"2024-04-25T05:45:29.954385+0000","last_fullsized":"2024-04-25T05:45:29.954385+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:32.286921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":600,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974190+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:45:29.974190+0000","last_peered":"2024-04-25T05:45:29.974190+0000","last_clean":"2024-04-25T05:45:29.974190+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:45:29.974190+0000","last_undegraded":"2024-04-25T05:45:29.974190+0000","last_fullsized":"2024-04-25T05:45:29.974190+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":299,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954472+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:45:29.954472+0000","last_peered":"2024-04-25T05:45:29.954472+0000","last_clean":"2024-04-25T05:45:29.954472+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:45:29.954472+0000","last_undegraded":"2024-04-25T05:45:29.954472+0000","last_fullsized":"2024-04-25T05:45:29.954472+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":615,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.952579+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:45:29.952579+0000","last_peered":"2024-04-25T05:45:29.952579+0000","last_clean":"2024-04-25T05:45:29.952579+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:45:29.952579+0000","last_undegraded":"2024-04-25T05:45:29.952579+0000","last_fullsized":"2024-04-25T05:45:29.952579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:40.500720+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":406,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971994+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:45:29.971994+0000","last_peered":"2024-04-25T05:45:29.971994+0000","last_clean":"2024-04-25T05:45:29.971994+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:45:29.971994+0000","last_undegraded":"2024-04-25T05:45:29.971994+0000","last_fullsized":"2024-04-25T05:45:29.971994+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":623,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.954026+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:45:29.954026+0000","last_peered":"2024-04-25T05:45:29.954026+0000","last_clean":"2024-04-25T05:45:29.954026+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:45:29.954026+0000","last_undegraded":"2024-04-25T05:45:29.954026+0000","last_fullsized":"2024-04-25T05:45:29.954026+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":638,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975918+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:45:29.975918+0000","last_peered":"2024-04-25T05:45:29.975918+0000","last_clean":"2024-04-25T05:45:29.975918+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:45:29.975918+0000","last_undegraded":"2024-04-25T05:45:29.975918+0000","last_fullsized":"2024-04-25T05:45:29.975918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":638,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.192235+0000","last_change":"2024-04-25T05:45:02.437804+0000","last_active":"2024-04-25T05:45:36.192235+0000","last_peered":"2024-04-25T05:45:36.192235+0000","last_clean":"2024-04-25T05:45:36.192235+0000","last_became_active":"2024-04-25T05:44:28.167563+0000","last_became_peered":"2024-04-25T05:44:28.167563+0000","last_unstale":"2024-04-25T05:45:36.192235+0000","last_undegraded":"2024-04-25T05:45:36.192235+0000","last_fullsized":"2024-04-25T05:45:36.192235+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:02.437727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:02.437727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:27.778001+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":284,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975997+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:45:29.975997+0000","last_peered":"2024-04-25T05:45:29.975997+0000","last_clean":"2024-04-25T05:45:29.975997+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:45:29.975997+0000","last_undegraded":"2024-04-25T05:45:29.975997+0000","last_fullsized":"2024-04-25T05:45:29.975997+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":605,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.982737+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:45:29.982737+0000","last_peered":"2024-04-25T05:45:29.982737+0000","last_clean":"2024-04-25T05:45:29.982737+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:45:29.982737+0000","last_undegraded":"2024-04-25T05:45:29.982737+0000","last_fullsized":"2024-04-25T05:45:29.982737+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":505,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972413+0000","last_change":"2024-04-25T05:44:28.237495+0000","last_active":"2024-04-25T05:45:29.972413+0000","last_peered":"2024-04-25T05:45:29.972413+0000","last_clean":"2024-04-25T05:45:29.972413+0000","last_became_active":"2024-04-25T05:44:28.182958+0000","last_became_peered":"2024-04-25T05:44:28.182958+0000","last_unstale":"2024-04-25T05:45:29.972413+0000","last_undegraded":"2024-04-25T05:45:29.972413+0000","last_fullsized":"2024-04-25T05:45:29.972413+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":102,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953598+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:45:29.953598+0000","last_peered":"2024-04-25T05:45:29.953598+0000","last_clean":"2024-04-25T05:45:29.953598+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:45:29.953598+0000","last_undegraded":"2024-04-25T05:45:29.953598+0000","last_fullsized":"2024-04-25T05:45:29.953598+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":624,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.974504+0000","last_change":"2024-04-25T05:44:28.189092+0000","last_active":"2024-04-25T05:45:29.974504+0000","last_peered":"2024-04-25T05:45:29.974504+0000","last_clean":"2024-04-25T05:45:29.974504+0000","last_became_active":"2024-04-25T05:44:28.185597+0000","last_became_peered":"2024-04-25T05:44:28.185597+0000","last_unstale":"2024-04-25T05:45:29.974504+0000","last_undegraded":"2024-04-25T05:45:29.974504+0000","last_fullsized":"2024-04-25T05:45:29.974504+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":633,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.195214+0000","last_change":"2024-04-25T05:45:20.391712+0000","last_active":"2024-04-25T05:45:36.195214+0000","last_peered":"2024-04-25T05:45:36.195214+0000","last_clean":"2024-04-25T05:45:36.195214+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:45:36.195214+0000","last_undegraded":"2024-04-25T05:45:36.195214+0000","last_fullsized":"2024-04-25T05:45:36.195214+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:20.391485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:45:20.391485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:34.472250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":349,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977050+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:45:29.977050+0000","last_peered":"2024-04-25T05:45:29.977050+0000","last_clean":"2024-04-25T05:45:29.977050+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:45:29.977050+0000","last_undegraded":"2024-04-25T05:45:29.977050+0000","last_fullsized":"2024-04-25T05:45:29.977050+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":130,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.192453+0000","last_change":"2024-04-25T05:45:00.435729+0000","last_active":"2024-04-25T05:45:36.192453+0000","last_peered":"2024-04-25T05:45:36.192453+0000","last_clean":"2024-04-25T05:45:36.192453+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:45:36.192453+0000","last_undegraded":"2024-04-25T05:45:36.192453+0000","last_fullsized":"2024-04-25T05:45:36.192453+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:00.435460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:45:00.435460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:25.355468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":606,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:31.300186+0000","last_change":"2024-04-25T05:45:31.300230+0000","last_active":"2024-04-25T05:45:31.300186+0000","last_peered":"2024-04-25T05:45:31.300186+0000","last_clean":"2024-04-25T05:45:31.300186+0000","last_became_active":"2024-04-25T05:34:02.701305+0000","last_became_peered":"2024-04-25T05:34:02.701305+0000","last_unstale":"2024-04-25T05:45:31.300186+0000","last_undegraded":"2024-04-25T05:45:31.300186+0000","last_fullsized":"2024-04-25T05:45:31.300186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.654603+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":116,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.969161+0000","last_change":"2024-04-25T05:44:28.201019+0000","last_active":"2024-04-25T05:45:29.969161+0000","last_peered":"2024-04-25T05:45:29.969161+0000","last_clean":"2024-04-25T05:45:29.969161+0000","last_became_active":"2024-04-25T05:44:28.200623+0000","last_became_peered":"2024-04-25T05:44:28.200623+0000","last_unstale":"2024-04-25T05:45:29.969161+0000","last_undegraded":"2024-04-25T05:45:29.969161+0000","last_fullsized":"2024-04-25T05:45:29.969161+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:31.571359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:31.571359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":386,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.975735+0000","last_change":"2024-04-25T05:44:28.198166+0000","last_active":"2024-04-25T05:45:29.975735+0000","last_peered":"2024-04-25T05:45:29.975735+0000","last_clean":"2024-04-25T05:45:29.975735+0000","last_became_active":"2024-04-25T05:44:28.179156+0000","last_became_peered":"2024-04-25T05:44:28.179156+0000","last_unstale":"2024-04-25T05:45:29.975735+0000","last_undegraded":"2024-04-25T05:45:29.975735+0000","last_fullsized":"2024-04-25T05:45:29.975735+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":424,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972339+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:45:29.972339+0000","last_peered":"2024-04-25T05:45:29.972339+0000","last_clean":"2024-04-25T05:45:29.972339+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:45:29.972339+0000","last_undegraded":"2024-04-25T05:45:29.972339+0000","last_fullsized":"2024-04-25T05:45:29.972339+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":430,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.953768+0000","last_change":"2024-04-25T05:45:23.299259+0000","last_active":"2024-04-25T05:45:29.953768+0000","last_peered":"2024-04-25T05:45:29.953768+0000","last_clean":"2024-04-25T05:45:29.953768+0000","last_became_active":"2024-04-25T05:44:28.125552+0000","last_became_peered":"2024-04-25T05:44:28.125552+0000","last_unstale":"2024-04-25T05:45:29.953768+0000","last_undegraded":"2024-04-25T05:45:29.953768+0000","last_fullsized":"2024-04-25T05:45:29.953768+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:43.393282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":226,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.970369+0000","last_change":"2024-04-25T05:44:28.169494+0000","last_active":"2024-04-25T05:45:29.970369+0000","last_peered":"2024-04-25T05:45:29.970369+0000","last_clean":"2024-04-25T05:45:29.970369+0000","last_became_active":"2024-04-25T05:44:28.168461+0000","last_became_peered":"2024-04-25T05:44:28.168461+0000","last_unstale":"2024-04-25T05:45:29.970369+0000","last_undegraded":"2024-04-25T05:45:29.970369+0000","last_fullsized":"2024-04-25T05:45:29.970369+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":835,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.972959+0000","last_change":"2024-04-25T05:44:28.225524+0000","last_active":"2024-04-25T05:45:29.972959+0000","last_peered":"2024-04-25T05:45:29.972959+0000","last_clean":"2024-04-25T05:45:29.972959+0000","last_became_active":"2024-04-25T05:44:28.182800+0000","last_became_peered":"2024-04-25T05:44:28.182800+0000","last_unstale":"2024-04-25T05:45:29.972959+0000","last_undegraded":"2024-04-25T05:45:29.972959+0000","last_fullsized":"2024-04-25T05:45:29.972959+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":625,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.976038+0000","last_change":"2024-04-25T05:44:16.995982+0000","last_active":"2024-04-25T05:45:29.976038+0000","last_peered":"2024-04-25T05:45:29.976038+0000","last_clean":"2024-04-25T05:45:29.976038+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:45:29.976038+0000","last_undegraded":"2024-04-25T05:45:29.976038+0000","last_fullsized":"2024-04-25T05:45:29.976038+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:38.671823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":582,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:32.772679+0000","last_change":"2024-04-25T05:45:32.772720+0000","last_active":"2024-04-25T05:45:32.772679+0000","last_peered":"2024-04-25T05:45:32.772679+0000","last_clean":"2024-04-25T05:45:32.772679+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:45:32.772679+0000","last_undegraded":"2024-04-25T05:45:32.772679+0000","last_fullsized":"2024-04-25T05:45:32.772679+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.772481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.772481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:36.168927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":598,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.977189+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:45:29.977189+0000","last_peered":"2024-04-25T05:45:29.977189+0000","last_clean":"2024-04-25T05:45:29.977189+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:45:29.977189+0000","last_undegraded":"2024-04-25T05:45:29.977189+0000","last_fullsized":"2024-04-25T05:45:29.977189+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":700,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-04-25T05:45:36.191003+0000","last_change":"2024-04-25T05:45:24.354902+0000","last_active":"2024-04-25T05:45:36.191003+0000","last_peered":"2024-04-25T05:45:36.191003+0000","last_clean":"2024-04-25T05:45:36.191003+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:45:36.191003+0000","last_undegraded":"2024-04-25T05:45:36.191003+0000","last_fullsized":"2024-04-25T05:45:36.191003+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:24.354764+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:45:24.354764+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:42.834643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":233,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971782+0000","last_change":"2024-04-25T05:45:22.646228+0000","last_active":"2024-04-25T05:45:29.971782+0000","last_peered":"2024-04-25T05:45:29.971782+0000","last_clean":"2024-04-25T05:45:29.971782+0000","last_became_active":"2024-04-25T05:44:28.181951+0000","last_became_peered":"2024-04-25T05:44:28.181951+0000","last_unstale":"2024-04-25T05:45:29.971782+0000","last_undegraded":"2024-04-25T05:45:29.971782+0000","last_fullsized":"2024-04-25T05:45:29.971782+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.646148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.646148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:45.468679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":612,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.968956+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:45:29.968956+0000","last_peered":"2024-04-25T05:45:29.968956+0000","last_clean":"2024-04-25T05:45:29.968956+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:45:29.968956+0000","last_undegraded":"2024-04-25T05:45:29.968956+0000","last_fullsized":"2024-04-25T05:45:29.968956+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":112,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-04-25T05:45:29.971086+0000","last_change":"2024-04-25T05:43:16.198408+0000","last_active":"2024-04-25T05:45:29.971086+0000","last_peered":"2024-04-25T05:45:29.971086+0000","last_clean":"2024-04-25T05:45:29.971086+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:45:29.971086+0000","last_undegraded":"2024-04-25T05:45:29.971086+0000","last_fullsized":"2024-04-25T05:45:29.971086+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:41:35.582841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:41:35.582841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4237623320,"num_objects":64662,"num_object_clones":0,"num_object_copies":193986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3941,"num_objects_unfound":0,"num_objects_dirty":64662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177962,"num_write_kb":7763905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33687,"num_bytes_recovered":2207645696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14288826368,"data_stored":14288814152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50206,"ondisk_log_size":50206,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":676,"num_read_kb":671,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":8366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":14104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070608,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3773868,"kb_used_data":3187236,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":89971284,"statfs":{"total":95995035648,"available":92130594816,"internally_reserved":0,"allocated":3263729664,"data_stored":3263179707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":600570314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364948,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767968,"kb_used_data":1303640,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91977184,"statfs":{"total":95995035648,"available":94184636416,"internally_reserved":0,"allocated":1334927360,"data_stored":1334389425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":475462086},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575021,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4024912,"kb_used_data":3335688,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":89720240,"statfs":{"total":95995035648,"available":91873525760,"internally_reserved":0,"allocated":3415744512,"data_stored":3415156995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738547,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411496,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3923328,"kb_used_data":3156344,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":89821824,"statfs":{"total":95995035648,"available":91977547776,"internally_reserved":0,"allocated":3232096256,"data_stored":3231525917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":785381831},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1598988,"kb_used_data":1170756,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92146164,"statfs":{"total":95995035648,"available":94357671936,"internally_reserved":0,"allocated":1198854144,"data_stored":1198285604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1986658304,"data_stored":1986658304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3406958592,"data_stored":3406954520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3167551488,"data_stored":3167551488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1187909632,"data_stored":1187905560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1327562752,"data_stored":1327562752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3212185600,"data_stored":3212181528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:45:40.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-04-25T05:45:40.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:40.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:40.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:40.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:40.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:40.969 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:45:40.972 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:45:40.972 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:45:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:45:40.973 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:45:40.973 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:45:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:40.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:41.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: osdmap e355: 8 total, 8 up, 5 in 2024-04-25T05:45:41.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: osdmap e355: 8 total, 8 up, 5 in 2024-04-25T05:45:41.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:41.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:45:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:45:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:45:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:45:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:45:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:45:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.32k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: osdmap e355: 8 total, 8 up, 5 in 2024-04-25T05:45:41.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:45:41.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:41.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:41.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:41.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:41.666 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:41.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:41.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:41.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:41.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:42.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:42.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:45:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:45:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:42.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[27010]: osdmap e356: 8 total, 8 up, 5 in 2024-04-25T05:45:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:45:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:45:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:42.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:41 smithi043 ceph-mon[37492]: osdmap e356: 8 total, 8 up, 5 in 2024-04-25T05:45:42.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:42.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:41 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:45:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:41 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:45:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:41 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2143184469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:45:42.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:41 smithi060 ceph-mon[30898]: osdmap e356: 8 total, 8 up, 5 in 2024-04-25T05:45:42.314 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:42.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:42.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:42.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:42.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:42.731 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:42.864 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:42.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:43.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:43.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:43.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:42 smithi043 ceph-mon[27010]: pgmap v616: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 895 B/s rd, 0 B/s wr, 1.38k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:43.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:42 smithi043 ceph-mon[37492]: pgmap v616: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 895 B/s rd, 0 B/s wr, 1.38k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:43.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:43.198 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:43.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:45:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:42 smithi060 ceph-mon[30898]: pgmap v616: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 156 active+clean; 3.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 895 B/s rd, 0 B/s wr, 1.38k op/s; 3477/170982 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:43.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:43.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:43.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:43.451 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:43.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:43.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:43.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:43.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:43.777 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:43.822 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:43.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:43.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:43.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:44.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:44.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:44.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:43 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:44.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:43 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:44.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:44.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:43 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:44.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:44.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:44.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:44.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:44.548 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:44.587 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:44.629 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:44.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:44.672 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:44.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:44.722 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:44.763 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:44.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:44.804 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:44.852 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:44.885 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:44.896 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:44.907 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:44.934 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:44.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:44.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[37492]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.61k op/s; 2985/147648 objects misplaced (2.022%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/579058529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[37492]: osdmap e357: 8 total, 8 up, 5 in 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[27010]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.61k op/s; 2985/147648 objects misplaced (2.022%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/579058529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:45.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:44 smithi043 ceph-mon[27010]: osdmap e357: 8 total, 8 up, 5 in 2024-04-25T05:45:45.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:45.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:45.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:44 smithi060 ceph-mon[30898]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.61k op/s; 2985/147648 objects misplaced (2.022%); 0 B/s, 0 objects/s recovering 2024-04-25T05:45:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/579058529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:44 smithi060 ceph-mon[30898]: osdmap e357: 8 total, 8 up, 5 in 2024-04-25T05:45:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:45.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:45.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:45.373 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:45.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.508 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:45.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:45.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:45.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:45.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:45.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:45.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:45:45.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:45.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:45.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:45.997 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 9 pgp_num to 16 2024-04-25T05:45:46.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:46.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:46.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/142052664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/142052664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/142052664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:45:46.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:46.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:46.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:46.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:46.530 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:46.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:46.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:46.686 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:46.686 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:46.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:46.812 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:46.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:46.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:46.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:47.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:47.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:47.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[27010]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:47.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:45:47.144 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[27010]: osdmap e358: 8 total, 8 up, 5 in 2024-04-25T05:45:47.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[37492]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:47.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:45:47.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:46 smithi043 ceph-mon[37492]: osdmap e358: 8 total, 8 up, 5 in 2024-04-25T05:45:47.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:47.206 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:46 smithi060 ceph-mon[30898]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:45:47.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:46 smithi060 ceph-mon[30898]: osdmap e358: 8 total, 8 up, 5 in 2024-04-25T05:45:47.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:47.342 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:47.444 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:47.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:47.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.599 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:47.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:47.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.728 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:47.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:47.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:47.866 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:47.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:47.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.001 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:48.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:48.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:48.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:48.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:48.272 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:48.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:48.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:48.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:48.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:48.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:48.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:48.808 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:48.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:45:48.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:48.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:48.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:49.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:45:49.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.084 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:49.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:49.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:49.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:49.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[27010]: pgmap v621: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 1.12k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:49.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:49.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:49.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:48 smithi043 ceph-mon[37492]: pgmap v621: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 1.12k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:49.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:49.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:49.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:45:49.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:45:49.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:48 smithi060 ceph-mon[30898]: pgmap v621: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 737 B/s rd, 0 B/s wr, 1.12k op/s; 2985/147648 objects misplaced (2.022%) 2024-04-25T05:45:49.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.367 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:49.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:49.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:49.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:49.549 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:49.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.652 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:49.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:49.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:49.758 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:45:49.775 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:49.781 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:49.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:49.913 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:49.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:50.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:50.043 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:50.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:50.173 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:50.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.307 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:50.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:50.445 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:50.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:50.573 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:50.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:50.695 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:45:50.695 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:45:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.703 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:50.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:50.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:50.963 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:50.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:51.043 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:51.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:51.110 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:51.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:50 smithi060 ceph-mon[30898]: pgmap v622: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.94k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:50 smithi060 ceph-mon[30898]: osdmap e359: 8 total, 8 up, 5 in 2024-04-25T05:45:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:51.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:51.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:51.381 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:50 smithi043 ceph-mon[27010]: pgmap v622: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.94k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:50 smithi043 ceph-mon[27010]: osdmap e359: 8 total, 8 up, 5 in 2024-04-25T05:45:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:50 smithi043 ceph-mon[37492]: pgmap v622: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.94k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:50 smithi043 ceph-mon[37492]: osdmap e359: 8 total, 8 up, 5 in 2024-04-25T05:45:51.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:51.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:51.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:51.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:51.643 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:51.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:45:51.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:51.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:51.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:51.920 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:51.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:51.959 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:45:52.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:52.052 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:52.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:52.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2047311742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:52.302 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2047311742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.303 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:51 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.303 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2047311742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.304 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:51 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:45:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.313 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:52.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:52.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:52.421 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:45:52.421 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:45:52.422 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:45:52.423 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:45:52.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:52.471 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:45:52.508 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:45:52.547 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:45:52.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:52.584 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:52.591 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:45:52.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:45:52.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:45:52.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:45:52.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:45:52.595 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:45:52.595 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:45:52.595 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:45:52.595 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:45:52.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:52.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:52.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:52.857 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:52.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:52.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:52.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:53.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:53.125 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:53.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:52 smithi060 ceph-mon[30898]: pgmap v624: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 1.11k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:52 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:52 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:52 smithi060 ceph-mon[30898]: osdmap e360: 8 total, 8 up, 5 in 2024-04-25T05:45:53.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/513806335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:53.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:53.298 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[27010]: pgmap v624: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 1.11k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:53.298 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[27010]: osdmap e360: 8 total, 8 up, 5 in 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/513806335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[37492]: pgmap v624: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 733 B/s rd, 0 B/s wr, 1.11k op/s; 2506/124323 objects misplaced (2.016%) 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[37492]: osdmap e360: 8 total, 8 up, 5 in 2024-04-25T05:45:53.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/513806335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:45:53.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:53.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:53.505 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:53.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:53.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:53.668 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:53.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:45:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:45:53.770 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:53.804 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:53.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:53.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:53.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:54.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:54.071 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:54.173 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:54.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:54.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:53 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:54.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:54.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:53 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:54.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:53 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:45:54.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:54.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:54.547 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:45:54.548 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:45:54.548 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-04-25T05:45:54.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:45:54.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:54.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:54.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:54.721 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:54.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:54.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:54.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:45:54.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:54.982 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:54.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:55.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:55.110 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:55.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:55.234 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:54 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:54 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:54 smithi060 ceph-mon[30898]: pgmap v626: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.00k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:55.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:54 smithi060 ceph-mon[30898]: osdmap e361: 8 total, 8 up, 5 in 2024-04-25T05:45:55.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:55.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:55.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:54 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[27010]: pgmap v626: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.00k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[27010]: osdmap e361: 8 total, 8 up, 5 in 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:54 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[37492]: pgmap v626: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.00k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:55 smithi043 ceph-mon[37492]: osdmap e361: 8 total, 8 up, 5 in 2024-04-25T05:45:55.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:55.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:55.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:55.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:55.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:55.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:55.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:55.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:55.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:56.027 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:56.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:56.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:56.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:56.299 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:56.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:56.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:56.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:56.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:45:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:56.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:45:56.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.708 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:56.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:56.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:56.837 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:56.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:56.900 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:56.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:57.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:57.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:56 smithi043 ceph-mon[27010]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:57.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:56 smithi043 ceph-mon[37492]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:57.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:57.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:57.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:56 smithi060 ceph-mon[30898]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:57.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:57.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:57.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:57.544 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:57.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:57.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:57.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:57.801 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:57.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:57.932 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:57.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:58.034 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:58.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:58.169 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:45:58.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:45:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:57 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:45:58.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:57 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:45:58.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:58.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:58.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:57 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:45:58.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:57 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:45:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:57 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:45:58.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:57 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:45:58.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:58.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:58.570 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:45:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:58.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:45:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.745 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:58.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:45:58.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:45:58.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:58.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:45:58.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:45:58.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:59.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:59.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:59.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:59.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:58 smithi060 ceph-mon[30898]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 1.01k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:59.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:58 smithi060 ceph-mon[30898]: osdmap e362: 8 total, 8 up, 5 in 2024-04-25T05:45:59.260 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:59.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:58 smithi043 ceph-mon[27010]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 1.01k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:59.264 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:58 smithi043 ceph-mon[27010]: osdmap e362: 8 total, 8 up, 5 in 2024-04-25T05:45:59.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:58 smithi043 ceph-mon[37492]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 641 B/s rd, 0 B/s wr, 1.01k op/s; 2037/100149 objects misplaced (2.034%) 2024-04-25T05:45:59.264 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:58 smithi043 ceph-mon[37492]: osdmap e362: 8 total, 8 up, 5 in 2024-04-25T05:45:59.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:45:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:59.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:45:59.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:45:59.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:45:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:45:59.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:59.665 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:59.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:45:59.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:45:59.842 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:45:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:45:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:45:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:45:59.842 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:45:59.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:45:59.927 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:45:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:45:59.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:45:59.986 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:45:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:45:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:00.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:00.063 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:00.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:00.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:00.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:59 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:00.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:59 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:00.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:45:59 smithi060 ceph-mon[30898]: osdmap e363: 8 total, 8 up, 5 in 2024-04-25T05:46:00.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.326 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:00.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:00.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:45:59 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:00.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:00.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[27010]: osdmap e363: 8 total, 8 up, 5 in 2024-04-25T05:46:00.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:45:59 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:00.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:00.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[37492]: osdmap e363: 8 total, 8 up, 5 in 2024-04-25T05:46:00.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.458 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:00.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:00.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:00.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:00.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:00.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:00.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:00.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:00.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:46:00.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:01.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:01.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:01.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:00 smithi060 ceph-mon[30898]: pgmap v631: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.90k op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:01.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:01.288 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:01.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[27010]: pgmap v631: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.90k op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:00 smithi043 ceph-mon[37492]: pgmap v631: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.90k op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:01.413 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:01.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:01.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:01.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.676 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:01.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:01.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:01.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:01.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:01.955 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:01.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:02.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:02.084 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:02.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:02.328 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:02.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.389 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:02.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:02.491 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:02.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:02.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:02.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:02.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:02.846 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:46:02.855 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:46:02.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:02.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:02.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:03.015 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:46:03.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:03.091 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[27010]: pgmap v633: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 644 B/s rd, 0 B/s wr, 895 op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:03.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[37492]: pgmap v633: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 644 B/s rd, 0 B/s wr, 895 op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:03.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:03.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:03.193 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:03.230 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:03.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:03.372 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:03 smithi060 ceph-mon[30898]: pgmap v633: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 644 B/s rd, 0 B/s wr, 895 op/s; 1589/78822 objects misplaced (2.016%) 2024-04-25T05:46:03.372 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:03.372 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:03.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:03.590 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.622 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:03.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:03.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:03.754 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:03.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:03.891 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:03.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:03.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:04.025 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:04.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:04.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:04.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:04.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:04.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[27010]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[27010]: osdmap e364: 8 total, 8 up, 5 in 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[37492]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[37492]: osdmap e364: 8 total, 8 up, 5 in 2024-04-25T05:46:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:04.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:04.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:04 smithi060 ceph-mon[30898]: from='client.25727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:04 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1693514164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:04 smithi060 ceph-mon[30898]: osdmap e364: 8 total, 8 up, 5 in 2024-04-25T05:46:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:04.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.566 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:04.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:04.667 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:04.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:04.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:04.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:04.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:04.931 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:04.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:04.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:04.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:04.974 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:05.017 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:05.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:05.067 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:05.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:05.116 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:05.159 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:05.196 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:05.196 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: pgmap v635: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.37k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2335199390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[37492]: osdmap e365: 8 total, 8 up, 5 in 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: pgmap v635: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.37k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2335199390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:05.197 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:05 smithi043 ceph-mon[27010]: osdmap e365: 8 total, 8 up, 5 in 2024-04-25T05:46:05.199 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:05.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:05.205 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:05.206 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:05.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:05.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:05.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:05.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: pgmap v635: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.37k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2335199390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:05 smithi060 ceph-mon[30898]: osdmap e365: 8 total, 8 up, 5 in 2024-04-25T05:46:05.514 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:05.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:05.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:05.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.781 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:05.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:05.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:05.916 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:05.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:06.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:06.050 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:06.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:06.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:06.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:06.318 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:06.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:06.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:06.703 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.736 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:06.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:06.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:06.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:06.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:06.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:07.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.143 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:07.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:07.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[27010]: pgmap v637: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[37492]: pgmap v637: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:46:07.258 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:07 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:46:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:07.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:07.283 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2024-04-25T05:46:01.910390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1707350579,"num_objects":26274,"num_object_clones":0,"num_object_copies":78822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1589,"num_objects_unfound":0,"num_objects_dirty":26274,"num_whiteouts":0,"num_read":2116,"num_read_kb":1836,"num_write":217494,"num_write_kb":7763950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34182,"num_bytes_recovered":2207985442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52896,"ondisk_log_size":52896,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":8057784,"kb_used_data":5037772,"kb_used_omap":7,"kb_used_meta":3019896,"kb_avail":460667976,"statfs":{"total":479975178240,"available":471724007424,"internally_reserved":0,"allocated":5158678528,"data_stored":5155608711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7967,"internal_metadata":3092373729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,8,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-465895424,"num_objects":-7109,"num_object_clones":0,"num_object_copies":-21327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-448,"num_objects_unfound":0,"num_objects_dirty":-7109,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.944992"},"pg_stats":[{"pgid":"9.1","version":"362'13592","reported_seq":13735,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.673967+0000","last_change":"2024-04-25T05:43:10.483768+0000","last_active":"2024-04-25T05:45:58.673967+0000","last_peered":"2024-04-25T05:45:58.673967+0000","last_clean":"2024-04-25T05:45:58.673967+0000","last_became_active":"2024-04-25T05:43:10.483376+0000","last_became_peered":"2024-04-25T05:43:10.483376+0000","last_unstale":"2024-04-25T05:45:58.673967+0000","last_undegraded":"2024-04-25T05:45:58.673967+0000","last_fullsized":"2024-04-25T05:45:58.673967+0000","mapping_epoch":296,"log_start":"350'10487","ondisk_log_start":"350'10487","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3105,"log_dups_size":0,"ondisk_log_size":3105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13592,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":348,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989448+0000","last_change":"2024-04-25T05:42:15.391020+0000","last_active":"2024-04-25T05:45:57.989448+0000","last_peered":"2024-04-25T05:45:57.989448+0000","last_clean":"2024-04-25T05:45:57.989448+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:45:57.989448+0000","last_undegraded":"2024-04-25T05:45:57.989448+0000","last_fullsized":"2024-04-25T05:45:57.989448+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T05:42:15.390858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:15.390858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992260+0000","last_change":"2024-04-25T05:44:28.173941+0000","last_active":"2024-04-25T05:45:57.992260+0000","last_peered":"2024-04-25T05:45:57.992260+0000","last_clean":"2024-04-25T05:45:57.992260+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:57.992260+0000","last_undegraded":"2024-04-25T05:45:57.992260+0000","last_fullsized":"2024-04-25T05:45:57.992260+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:17.311163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:17.311163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"325'12","reported_seq":454,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986335+0000","last_change":"2024-04-25T05:45:36.284410+0000","last_active":"2024-04-25T05:45:57.986335+0000","last_peered":"2024-04-25T05:45:57.986335+0000","last_clean":"2024-04-25T05:45:57.986335+0000","last_became_active":"2024-04-25T05:44:28.171810+0000","last_became_peered":"2024-04-25T05:44:28.171810+0000","last_unstale":"2024-04-25T05:45:57.986335+0000","last_undegraded":"2024-04-25T05:45:57.986335+0000","last_fullsized":"2024-04-25T05:45:57.986335+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:45:36.284279+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.284279+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:03.999263+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":311,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985422+0000","last_change":"2024-04-25T05:43:17.275828+0000","last_active":"2024-04-25T05:45:57.985422+0000","last_peered":"2024-04-25T05:45:57.985422+0000","last_clean":"2024-04-25T05:45:57.985422+0000","last_became_active":"2024-04-25T05:43:16.200421+0000","last_became_peered":"2024-04-25T05:43:16.200421+0000","last_unstale":"2024-04-25T05:45:57.985422+0000","last_undegraded":"2024-04-25T05:45:57.985422+0000","last_fullsized":"2024-04-25T05:45:57.985422+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":953,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985026+0000","last_change":"2024-04-25T05:45:39.362002+0000","last_active":"2024-04-25T05:45:57.985026+0000","last_peered":"2024-04-25T05:45:57.985026+0000","last_clean":"2024-04-25T05:45:57.985026+0000","last_became_active":"2024-04-25T05:43:19.209902+0000","last_became_peered":"2024-04-25T05:43:19.209902+0000","last_unstale":"2024-04-25T05:45:57.985026+0000","last_undegraded":"2024-04-25T05:45:57.985026+0000","last_fullsized":"2024-04-25T05:45:57.985026+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:55.835348+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"362'13469","reported_seq":13626,"reported_epoch":362,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:45:58.838522+0000","last_change":"2024-04-25T05:45:29.228939+0000","last_active":"2024-04-25T05:45:58.838522+0000","last_peered":"2024-04-25T05:45:58.838522+0000","last_clean":"2024-04-25T05:44:27.095992+0000","last_became_active":"2024-04-25T05:44:29.189111+0000","last_became_peered":"2024-04-25T05:44:29.189111+0000","last_unstale":"2024-04-25T05:45:58.838522+0000","last_undegraded":"2024-04-25T05:45:58.838522+0000","last_fullsized":"2024-04-25T05:45:58.838522+0000","mapping_epoch":325,"log_start":"350'10487","ondisk_log_start":"350'10487","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":3000,"ondisk_log_size":2982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104136704,"num_objects":1589,"num_object_clones":0,"num_object_copies":4767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1589,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13469,"num_write_kb":481856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":1589}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":357,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987084+0000","last_change":"2024-04-25T05:42:24.144147+0000","last_active":"2024-04-25T05:45:57.987084+0000","last_peered":"2024-04-25T05:45:57.987084+0000","last_clean":"2024-04-25T05:45:57.987084+0000","last_became_active":"2024-04-25T05:40:55.965287+0000","last_became_peered":"2024-04-25T05:40:55.965287+0000","last_unstale":"2024-04-25T05:45:57.987084+0000","last_undegraded":"2024-04-25T05:45:57.987084+0000","last_fullsized":"2024-04-25T05:45:57.987084+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'25","last_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:42:24.143971+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975044+0000","last_change":"2024-04-25T05:44:00.808266+0000","last_active":"2024-04-25T05:45:53.975044+0000","last_peered":"2024-04-25T05:45:53.975044+0000","last_clean":"2024-04-25T05:45:53.975044+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:45:53.975044+0000","last_undegraded":"2024-04-25T05:45:53.975044+0000","last_fullsized":"2024-04-25T05:45:53.975044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.808123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.808123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756235+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":653,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986445+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:45:57.986445+0000","last_peered":"2024-04-25T05:45:57.986445+0000","last_clean":"2024-04-25T05:45:57.986445+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:45:57.986445+0000","last_undegraded":"2024-04-25T05:45:57.986445+0000","last_fullsized":"2024-04-25T05:45:57.986445+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:45.261878+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":264,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989665+0000","last_change":"2024-04-25T05:44:09.175841+0000","last_active":"2024-04-25T05:45:57.989665+0000","last_peered":"2024-04-25T05:45:57.989665+0000","last_clean":"2024-04-25T05:45:57.989665+0000","last_became_active":"2024-04-25T05:40:55.974117+0000","last_became_peered":"2024-04-25T05:40:55.974117+0000","last_unstale":"2024-04-25T05:45:57.989665+0000","last_undegraded":"2024-04-25T05:45:57.989665+0000","last_fullsized":"2024-04-25T05:45:57.989665+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900036+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":545,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989163+0000","last_change":"2024-04-25T05:43:19.441231+0000","last_active":"2024-04-25T05:45:57.989163+0000","last_peered":"2024-04-25T05:45:57.989163+0000","last_clean":"2024-04-25T05:45:57.989163+0000","last_became_active":"2024-04-25T05:43:19.209795+0000","last_became_peered":"2024-04-25T05:43:19.209795+0000","last_unstale":"2024-04-25T05:45:57.989163+0000","last_undegraded":"2024-04-25T05:45:57.989163+0000","last_fullsized":"2024-04-25T05:45:57.989163+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899899+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"362'13428","reported_seq":13571,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.000727+0000","last_change":"2024-04-25T05:43:10.142785+0000","last_active":"2024-04-25T05:45:58.000727+0000","last_peered":"2024-04-25T05:45:58.000727+0000","last_clean":"2024-04-25T05:45:58.000727+0000","last_became_active":"2024-04-25T05:43:10.142425+0000","last_became_peered":"2024-04-25T05:43:10.142425+0000","last_unstale":"2024-04-25T05:45:58.000727+0000","last_undegraded":"2024-04-25T05:45:58.000727+0000","last_fullsized":"2024-04-25T05:45:58.000727+0000","mapping_epoch":296,"log_start":"349'10177","ondisk_log_start":"349'10177","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3251,"log_dups_size":0,"ondisk_log_size":3251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":1600,"num_object_clones":0,"num_object_copies":4800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":663,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975172+0000","last_change":"2024-04-25T05:45:26.763116+0000","last_active":"2024-04-25T05:45:53.975172+0000","last_peered":"2024-04-25T05:45:53.975172+0000","last_clean":"2024-04-25T05:45:53.975172+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:45:53.975172+0000","last_undegraded":"2024-04-25T05:45:53.975172+0000","last_fullsized":"2024-04-25T05:45:53.975172+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:45:26.762873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:26.762873+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:48.188664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":649,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967766+0000","last_change":"2024-04-25T05:44:46.503302+0000","last_active":"2024-04-25T05:45:53.967766+0000","last_peered":"2024-04-25T05:45:53.967766+0000","last_clean":"2024-04-25T05:45:53.967766+0000","last_became_active":"2024-04-25T05:40:55.964581+0000","last_became_peered":"2024-04-25T05:40:55.964581+0000","last_unstale":"2024-04-25T05:45:53.967766+0000","last_undegraded":"2024-04-25T05:45:53.967766+0000","last_fullsized":"2024-04-25T05:45:53.967766+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:46.503173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:46.503173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:56.738128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":445,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.970934+0000","last_change":"2024-04-25T05:44:57.457953+0000","last_active":"2024-04-25T05:45:53.970934+0000","last_peered":"2024-04-25T05:45:53.970934+0000","last_clean":"2024-04-25T05:45:53.970934+0000","last_became_active":"2024-04-25T05:43:16.189398+0000","last_became_peered":"2024-04-25T05:43:16.189398+0000","last_unstale":"2024-04-25T05:45:53.970934+0000","last_undegraded":"2024-04-25T05:45:53.970934+0000","last_fullsized":"2024-04-25T05:45:53.970934+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:44:57.457842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:18.204723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":627,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.988998+0000","last_change":"2024-04-25T05:43:16.198287+0000","last_active":"2024-04-25T05:45:57.988998+0000","last_peered":"2024-04-25T05:45:57.988998+0000","last_clean":"2024-04-25T05:45:57.988998+0000","last_became_active":"2024-04-25T05:43:16.197363+0000","last_became_peered":"2024-04-25T05:43:16.197363+0000","last_unstale":"2024-04-25T05:45:57.988998+0000","last_undegraded":"2024-04-25T05:45:57.988998+0000","last_fullsized":"2024-04-25T05:45:57.988998+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:59.402237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:59.402237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"362'13789","reported_seq":13932,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.744101+0000","last_change":"2024-04-25T05:43:10.140591+0000","last_active":"2024-04-25T05:45:58.744101+0000","last_peered":"2024-04-25T05:45:58.744101+0000","last_clean":"2024-04-25T05:45:58.744101+0000","last_became_active":"2024-04-25T05:43:10.140301+0000","last_became_peered":"2024-04-25T05:43:10.140301+0000","last_unstale":"2024-04-25T05:45:58.744101+0000","last_undegraded":"2024-04-25T05:45:58.744101+0000","last_fullsized":"2024-04-25T05:45:58.744101+0000","mapping_epoch":296,"log_start":"350'10681","ondisk_log_start":"350'10681","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3108,"log_dups_size":0,"ondisk_log_size":3108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13789,"num_write_kb":493376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":675,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985909+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:45:57.985909+0000","last_peered":"2024-04-25T05:45:57.985909+0000","last_clean":"2024-04-25T05:45:57.985909+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:45:57.985909+0000","last_undegraded":"2024-04-25T05:45:57.985909+0000","last_fullsized":"2024-04-25T05:45:57.985909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":629,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989538+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:45:57.989538+0000","last_peered":"2024-04-25T05:45:57.989538+0000","last_clean":"2024-04-25T05:45:57.989538+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:45:57.989538+0000","last_undegraded":"2024-04-25T05:45:57.989538+0000","last_fullsized":"2024-04-25T05:45:57.989538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"325'13","reported_seq":180,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.988958+0000","last_change":"2024-04-25T05:45:28.975997+0000","last_active":"2024-04-25T05:45:57.988958+0000","last_peered":"2024-04-25T05:45:57.988958+0000","last_clean":"2024-04-25T05:45:57.988958+0000","last_became_active":"2024-04-25T05:44:28.178692+0000","last_became_peered":"2024-04-25T05:44:28.178692+0000","last_unstale":"2024-04-25T05:45:57.988958+0000","last_undegraded":"2024-04-25T05:45:57.988958+0000","last_fullsized":"2024-04-25T05:45:57.988958+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.085930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":499,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986874+0000","last_change":"2024-04-25T05:45:31.796893+0000","last_active":"2024-04-25T05:45:57.986874+0000","last_peered":"2024-04-25T05:45:57.986874+0000","last_clean":"2024-04-25T05:45:57.986874+0000","last_became_active":"2024-04-25T05:44:28.177729+0000","last_became_peered":"2024-04-25T05:44:28.177729+0000","last_unstale":"2024-04-25T05:45:57.986874+0000","last_undegraded":"2024-04-25T05:45:57.986874+0000","last_fullsized":"2024-04-25T05:45:57.986874+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:53.427025+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":619,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.974784+0000","last_change":"2024-04-25T05:44:30.932605+0000","last_active":"2024-04-25T05:45:53.974784+0000","last_peered":"2024-04-25T05:45:53.974784+0000","last_clean":"2024-04-25T05:45:53.974784+0000","last_became_active":"2024-04-25T05:44:30.932221+0000","last_became_peered":"2024-04-25T05:44:30.932221+0000","last_unstale":"2024-04-25T05:45:53.974784+0000","last_undegraded":"2024-04-25T05:45:53.974784+0000","last_fullsized":"2024-04-25T05:45:53.974784+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"362'13568","reported_seq":13718,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.996774+0000","last_change":"2024-04-25T05:43:16.204887+0000","last_active":"2024-04-25T05:45:57.996774+0000","last_peered":"2024-04-25T05:45:57.996774+0000","last_clean":"2024-04-25T05:45:57.996774+0000","last_became_active":"2024-04-25T05:43:16.204361+0000","last_became_peered":"2024-04-25T05:43:16.204361+0000","last_unstale":"2024-04-25T05:45:57.996774+0000","last_undegraded":"2024-04-25T05:45:57.996774+0000","last_fullsized":"2024-04-25T05:45:57.996774+0000","mapping_epoch":301,"log_start":"349'10253","ondisk_log_start":"349'10253","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3315,"log_dups_size":0,"ondisk_log_size":3315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806744,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13568,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":635,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967676+0000","last_change":"2024-04-25T05:45:29.415240+0000","last_active":"2024-04-25T05:45:53.967676+0000","last_peered":"2024-04-25T05:45:53.967676+0000","last_clean":"2024-04-25T05:45:53.967676+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:45:53.967676+0000","last_undegraded":"2024-04-25T05:45:53.967676+0000","last_fullsized":"2024-04-25T05:45:53.967676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.415117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.415117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:43.082744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":783,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989669+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:45:57.989669+0000","last_peered":"2024-04-25T05:45:57.989669+0000","last_clean":"2024-04-25T05:45:57.989669+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:45:57.989669+0000","last_undegraded":"2024-04-25T05:45:57.989669+0000","last_fullsized":"2024-04-25T05:45:57.989669+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900172+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":655,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989946+0000","last_change":"2024-04-25T05:45:21.024234+0000","last_active":"2024-04-25T05:45:57.989946+0000","last_peered":"2024-04-25T05:45:57.989946+0000","last_clean":"2024-04-25T05:45:57.989946+0000","last_became_active":"2024-04-25T05:44:28.178969+0000","last_became_peered":"2024-04-25T05:44:28.178969+0000","last_unstale":"2024-04-25T05:45:57.989946+0000","last_undegraded":"2024-04-25T05:45:57.989946+0000","last_fullsized":"2024-04-25T05:45:57.989946+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:21.023940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:21.023940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:35.928250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":305,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.991887+0000","last_change":"2024-04-25T05:44:05.919316+0000","last_active":"2024-04-25T05:45:57.991887+0000","last_peered":"2024-04-25T05:45:57.991887+0000","last_clean":"2024-04-25T05:45:57.991887+0000","last_became_active":"2024-04-25T05:40:55.974602+0000","last_became_peered":"2024-04-25T05:40:55.974602+0000","last_unstale":"2024-04-25T05:45:57.991887+0000","last_undegraded":"2024-04-25T05:45:57.991887+0000","last_fullsized":"2024-04-25T05:45:57.991887+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":855,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.968640+0000","last_change":"2024-04-25T05:45:36.587515+0000","last_active":"2024-04-25T05:45:53.968640+0000","last_peered":"2024-04-25T05:45:53.968640+0000","last_clean":"2024-04-25T05:45:53.968640+0000","last_became_active":"2024-04-25T05:44:30.915718+0000","last_became_peered":"2024-04-25T05:44:30.915718+0000","last_unstale":"2024-04-25T05:45:53.968640+0000","last_undegraded":"2024-04-25T05:45:53.968640+0000","last_fullsized":"2024-04-25T05:45:53.968640+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:36.587388+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.587388+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:39.493160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"362'13729","reported_seq":13872,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.768682+0000","last_change":"2024-04-25T05:43:10.138792+0000","last_active":"2024-04-25T05:45:58.768682+0000","last_peered":"2024-04-25T05:45:58.768682+0000","last_clean":"2024-04-25T05:45:58.768682+0000","last_became_active":"2024-04-25T05:43:10.138484+0000","last_became_peered":"2024-04-25T05:43:10.138484+0000","last_unstale":"2024-04-25T05:45:58.768682+0000","last_undegraded":"2024-04-25T05:45:58.768682+0000","last_fullsized":"2024-04-25T05:45:58.768682+0000","mapping_epoch":296,"log_start":"349'10616","ondisk_log_start":"349'10616","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107413504,"num_objects":1639,"num_object_clones":0,"num_object_copies":4917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13729,"num_write_kb":491776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":783,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989874+0000","last_change":"2024-04-25T05:45:36.207163+0000","last_active":"2024-04-25T05:45:57.989874+0000","last_peered":"2024-04-25T05:45:57.989874+0000","last_clean":"2024-04-25T05:45:57.989874+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:45:57.989874+0000","last_undegraded":"2024-04-25T05:45:57.989874+0000","last_fullsized":"2024-04-25T05:45:57.989874+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:45:34.969209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:34.969209+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.116597+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":579,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.977551+0000","last_change":"2024-04-25T05:45:27.764855+0000","last_active":"2024-04-25T05:45:53.977551+0000","last_peered":"2024-04-25T05:45:53.977551+0000","last_clean":"2024-04-25T05:45:53.977551+0000","last_became_active":"2024-04-25T05:44:28.124913+0000","last_became_peered":"2024-04-25T05:44:28.124913+0000","last_unstale":"2024-04-25T05:45:53.977551+0000","last_undegraded":"2024-04-25T05:45:53.977551+0000","last_fullsized":"2024-04-25T05:45:53.977551+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:45:27.764695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.764695+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:40.827850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":333,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985594+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:45:57.985594+0000","last_peered":"2024-04-25T05:45:57.985594+0000","last_clean":"2024-04-25T05:45:57.985594+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:45:57.985594+0000","last_undegraded":"2024-04-25T05:45:57.985594+0000","last_fullsized":"2024-04-25T05:45:57.985594+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":262,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984588+0000","last_change":"2024-04-25T05:43:00.031433+0000","last_active":"2024-04-25T05:45:57.984588+0000","last_peered":"2024-04-25T05:45:57.984588+0000","last_clean":"2024-04-25T05:45:57.984588+0000","last_became_active":"2024-04-25T05:40:55.981177+0000","last_became_peered":"2024-04-25T05:40:55.981177+0000","last_unstale":"2024-04-25T05:45:57.984588+0000","last_undegraded":"2024-04-25T05:45:57.984588+0000","last_fullsized":"2024-04-25T05:45:57.984588+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":817,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.988813+0000","last_change":"2024-04-25T05:45:39.882932+0000","last_active":"2024-04-25T05:45:57.988813+0000","last_peered":"2024-04-25T05:45:57.988813+0000","last_clean":"2024-04-25T05:45:57.988813+0000","last_became_active":"2024-04-25T05:43:19.215232+0000","last_became_peered":"2024-04-25T05:43:19.215232+0000","last_unstale":"2024-04-25T05:45:57.988813+0000","last_undegraded":"2024-04-25T05:45:57.988813+0000","last_fullsized":"2024-04-25T05:45:57.988813+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.882669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.882669+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:44.329481+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"362'13599","reported_seq":13733,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.828477+0000","last_change":"2024-04-25T05:43:16.189073+0000","last_active":"2024-04-25T05:45:58.828477+0000","last_peered":"2024-04-25T05:45:58.828477+0000","last_clean":"2024-04-25T05:45:58.828477+0000","last_became_active":"2024-04-25T05:43:16.188666+0000","last_became_peered":"2024-04-25T05:43:16.188666+0000","last_unstale":"2024-04-25T05:45:58.828477+0000","last_undegraded":"2024-04-25T05:45:58.828477+0000","last_fullsized":"2024-04-25T05:45:58.828477+0000","mapping_epoch":301,"log_start":"350'10638","ondisk_log_start":"350'10638","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2961,"log_dups_size":3000,"ondisk_log_size":2961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":4869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13599,"num_write_kb":487104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":354,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975403+0000","last_change":"2024-04-25T05:44:28.202828+0000","last_active":"2024-04-25T05:45:53.975403+0000","last_peered":"2024-04-25T05:45:53.975403+0000","last_clean":"2024-04-25T05:45:53.975403+0000","last_became_active":"2024-04-25T05:44:28.182043+0000","last_became_peered":"2024-04-25T05:44:28.182043+0000","last_unstale":"2024-04-25T05:45:53.975403+0000","last_undegraded":"2024-04-25T05:45:53.975403+0000","last_fullsized":"2024-04-25T05:45:53.975403+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'14","last_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.494293+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756307+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":331,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987224+0000","last_change":"2024-04-25T05:44:28.217831+0000","last_active":"2024-04-25T05:45:57.987224+0000","last_peered":"2024-04-25T05:45:57.987224+0000","last_clean":"2024-04-25T05:45:57.987224+0000","last_became_active":"2024-04-25T05:44:28.170458+0000","last_became_peered":"2024-04-25T05:44:28.170458+0000","last_unstale":"2024-04-25T05:45:57.987224+0000","last_undegraded":"2024-04-25T05:45:57.987224+0000","last_fullsized":"2024-04-25T05:45:57.987224+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'22","last_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.092050+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":516,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990895+0000","last_change":"2024-04-25T05:43:20.410521+0000","last_active":"2024-04-25T05:45:57.990895+0000","last_peered":"2024-04-25T05:45:57.990895+0000","last_clean":"2024-04-25T05:45:57.990895+0000","last_became_active":"2024-04-25T05:43:16.197606+0000","last_became_peered":"2024-04-25T05:43:16.197606+0000","last_unstale":"2024-04-25T05:45:57.990895+0000","last_undegraded":"2024-04-25T05:45:57.990895+0000","last_fullsized":"2024-04-25T05:45:57.990895+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":255,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989766+0000","last_change":"2024-04-25T05:45:31.048330+0000","last_active":"2024-04-25T05:45:57.989766+0000","last_peered":"2024-04-25T05:45:57.989766+0000","last_clean":"2024-04-25T05:45:57.989766+0000","last_became_active":"2024-04-25T05:44:28.168333+0000","last_became_peered":"2024-04-25T05:44:28.168333+0000","last_unstale":"2024-04-25T05:45:57.989766+0000","last_undegraded":"2024-04-25T05:45:57.989766+0000","last_fullsized":"2024-04-25T05:45:57.989766+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.047907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.314401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":813,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976750+0000","last_change":"2024-04-25T05:45:40.844494+0000","last_active":"2024-04-25T05:45:53.976750+0000","last_peered":"2024-04-25T05:45:53.976750+0000","last_clean":"2024-04-25T05:45:53.976750+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:45:53.976750+0000","last_undegraded":"2024-04-25T05:45:53.976750+0000","last_fullsized":"2024-04-25T05:45:53.976750+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:40.844354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:40.844354+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:47.315570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"362'13385","reported_seq":13528,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.996850+0000","last_change":"2024-04-25T05:43:10.145192+0000","last_active":"2024-04-25T05:45:57.996850+0000","last_peered":"2024-04-25T05:45:57.996850+0000","last_clean":"2024-04-25T05:45:57.996850+0000","last_became_active":"2024-04-25T05:43:10.144491+0000","last_became_peered":"2024-04-25T05:43:10.144491+0000","last_unstale":"2024-04-25T05:45:57.996850+0000","last_undegraded":"2024-04-25T05:45:57.996850+0000","last_fullsized":"2024-04-25T05:45:57.996850+0000","mapping_epoch":296,"log_start":"349'10130","ondisk_log_start":"349'10130","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3255,"log_dups_size":0,"ondisk_log_size":3255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13385,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":369,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975170+0000","last_change":"2024-04-25T05:44:28.125054+0000","last_active":"2024-04-25T05:45:53.975170+0000","last_peered":"2024-04-25T05:45:53.975170+0000","last_clean":"2024-04-25T05:45:53.975170+0000","last_became_active":"2024-04-25T05:44:28.124674+0000","last_became_peered":"2024-04-25T05:44:28.124674+0000","last_unstale":"2024-04-25T05:45:53.975170+0000","last_undegraded":"2024-04-25T05:45:53.975170+0000","last_fullsized":"2024-04-25T05:45:53.975170+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.613840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.613840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":359,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984944+0000","last_change":"2024-04-25T05:44:28.198134+0000","last_active":"2024-04-25T05:45:57.984944+0000","last_peered":"2024-04-25T05:45:57.984944+0000","last_clean":"2024-04-25T05:45:57.984944+0000","last_became_active":"2024-04-25T05:44:28.168410+0000","last_became_peered":"2024-04-25T05:44:28.168410+0000","last_unstale":"2024-04-25T05:45:57.984944+0000","last_undegraded":"2024-04-25T05:45:57.984944+0000","last_fullsized":"2024-04-25T05:45:57.984944+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:42:22.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:22.674188+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":655,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985791+0000","last_change":"2024-04-25T05:44:28.167276+0000","last_active":"2024-04-25T05:45:57.985791+0000","last_peered":"2024-04-25T05:45:57.985791+0000","last_clean":"2024-04-25T05:45:57.985791+0000","last_became_active":"2024-04-25T05:44:28.166968+0000","last_became_peered":"2024-04-25T05:44:28.166968+0000","last_unstale":"2024-04-25T05:45:57.985791+0000","last_undegraded":"2024-04-25T05:45:57.985791+0000","last_fullsized":"2024-04-25T05:45:57.985791+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":635,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984389+0000","last_change":"2024-04-25T05:42:23.119384+0000","last_active":"2024-04-25T05:45:57.984389+0000","last_peered":"2024-04-25T05:45:57.984389+0000","last_clean":"2024-04-25T05:45:57.984389+0000","last_became_active":"2024-04-25T05:40:55.963795+0000","last_became_peered":"2024-04-25T05:40:55.963795+0000","last_unstale":"2024-04-25T05:45:57.984389+0000","last_undegraded":"2024-04-25T05:45:57.984389+0000","last_fullsized":"2024-04-25T05:45:57.984389+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:23.119155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:23.119155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":831,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.974735+0000","last_change":"2024-04-25T05:44:30.918290+0000","last_active":"2024-04-25T05:45:53.974735+0000","last_peered":"2024-04-25T05:45:53.974735+0000","last_clean":"2024-04-25T05:45:53.974735+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:45:53.974735+0000","last_undegraded":"2024-04-25T05:45:53.974735+0000","last_fullsized":"2024-04-25T05:45:53.974735+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:20.780175+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.780175+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756457+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"361'13393","reported_seq":27067,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.905166+0000","last_change":"2024-04-25T05:45:38.753153+0000","last_active":"2024-04-25T05:45:57.905166+0000","last_peered":"2024-04-25T05:45:57.905166+0000","last_clean":"2024-04-25T05:45:57.905166+0000","last_became_active":"2024-04-25T05:45:10.941174+0000","last_became_peered":"2024-04-25T05:45:10.941174+0000","last_unstale":"2024-04-25T05:45:57.905166+0000","last_undegraded":"2024-04-25T05:45:57.905166+0000","last_fullsized":"2024-04-25T05:45:57.905166+0000","mapping_epoch":343,"log_start":"349'9911","ondisk_log_start":"349'9911","created":296,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'11560","last_scrub_stamp":"2024-04-25T05:45:38.752800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.752800+0000","objects_scrubbed":3470,"log_size":3482,"log_dups_size":0,"ondisk_log_size":3482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:00.437579+0000","scrub_duration":904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105316352,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13393,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6735,"num_bytes_recovered":441384960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":504,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987368+0000","last_change":"2024-04-25T05:44:28.180356+0000","last_active":"2024-04-25T05:45:57.987368+0000","last_peered":"2024-04-25T05:45:57.987368+0000","last_clean":"2024-04-25T05:45:57.987368+0000","last_became_active":"2024-04-25T05:44:28.180059+0000","last_became_peered":"2024-04-25T05:44:28.180059+0000","last_unstale":"2024-04-25T05:45:57.987368+0000","last_undegraded":"2024-04-25T05:45:57.987368+0000","last_fullsized":"2024-04-25T05:45:57.987368+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":177,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986253+0000","last_change":"2024-04-25T05:44:28.234082+0000","last_active":"2024-04-25T05:45:57.986253+0000","last_peered":"2024-04-25T05:45:57.986253+0000","last_clean":"2024-04-25T05:45:57.986253+0000","last_became_active":"2024-04-25T05:44:28.177715+0000","last_became_peered":"2024-04-25T05:44:28.177715+0000","last_unstale":"2024-04-25T05:45:57.986253+0000","last_undegraded":"2024-04-25T05:45:57.986253+0000","last_fullsized":"2024-04-25T05:45:57.986253+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1389,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985271+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:45:57.985271+0000","last_peered":"2024-04-25T05:45:57.985271+0000","last_clean":"2024-04-25T05:45:57.985271+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:45:57.985271+0000","last_undegraded":"2024-04-25T05:45:57.985271+0000","last_fullsized":"2024-04-25T05:45:57.985271+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":701,"num_read_kb":696,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":254,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967856+0000","last_change":"2024-04-25T05:45:30.434222+0000","last_active":"2024-04-25T05:45:53.967856+0000","last_peered":"2024-04-25T05:45:53.967856+0000","last_clean":"2024-04-25T05:45:53.967856+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:45:53.967856+0000","last_undegraded":"2024-04-25T05:45:53.967856+0000","last_fullsized":"2024-04-25T05:45:53.967856+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:30.434101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.434101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.921750+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":631,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985518+0000","last_change":"2024-04-25T05:45:36.702620+0000","last_active":"2024-04-25T05:45:57.985518+0000","last_peered":"2024-04-25T05:45:57.985518+0000","last_clean":"2024-04-25T05:45:57.985518+0000","last_became_active":"2024-04-25T05:43:16.188851+0000","last_became_peered":"2024-04-25T05:43:16.188851+0000","last_unstale":"2024-04-25T05:45:57.985518+0000","last_undegraded":"2024-04-25T05:45:57.985518+0000","last_fullsized":"2024-04-25T05:45:57.985518+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:36.702085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.702085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.808010+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":811,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985933+0000","last_change":"2024-04-25T05:42:16.111591+0000","last_active":"2024-04-25T05:45:57.985933+0000","last_peered":"2024-04-25T05:45:57.985933+0000","last_clean":"2024-04-25T05:45:57.985933+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:45:57.985933+0000","last_undegraded":"2024-04-25T05:45:57.985933+0000","last_fullsized":"2024-04-25T05:45:57.985933+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:16.111452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.111452+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"362'13509","reported_seq":26676,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.673128+0000","last_change":"2024-04-25T05:45:20.779526+0000","last_active":"2024-04-25T05:45:58.673128+0000","last_peered":"2024-04-25T05:45:58.673128+0000","last_clean":"2024-04-25T05:45:58.673128+0000","last_became_active":"2024-04-25T05:45:18.104956+0000","last_became_peered":"2024-04-25T05:45:18.104956+0000","last_unstale":"2024-04-25T05:45:58.673128+0000","last_undegraded":"2024-04-25T05:45:58.673128+0000","last_fullsized":"2024-04-25T05:45:58.673128+0000","mapping_epoch":347,"log_start":"349'10373","ondisk_log_start":"349'10373","created":296,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'9941","last_scrub_stamp":"2024-04-25T05:45:20.778952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:45:20.778952+0000","objects_scrubbed":5279,"log_size":3136,"log_dups_size":0,"ondisk_log_size":3136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:31.668567+0000","scrub_duration":1839,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13509,"num_write_kb":484480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6470,"num_bytes_recovered":424017920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":647,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984939+0000","last_change":"2024-04-25T05:44:28.178984+0000","last_active":"2024-04-25T05:45:57.984939+0000","last_peered":"2024-04-25T05:45:57.984939+0000","last_clean":"2024-04-25T05:45:57.984939+0000","last_became_active":"2024-04-25T05:44:28.178614+0000","last_became_peered":"2024-04-25T05:44:28.178614+0000","last_unstale":"2024-04-25T05:45:57.984939+0000","last_undegraded":"2024-04-25T05:45:57.984939+0000","last_fullsized":"2024-04-25T05:45:57.984939+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":464,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976007+0000","last_change":"2024-04-25T05:45:38.816178+0000","last_active":"2024-04-25T05:45:53.976007+0000","last_peered":"2024-04-25T05:45:53.976007+0000","last_clean":"2024-04-25T05:45:53.976007+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:45:53.976007+0000","last_undegraded":"2024-04-25T05:45:53.976007+0000","last_fullsized":"2024-04-25T05:45:53.976007+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:45:38.815913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.815913+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:02.678590+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.991567+0000","last_change":"2024-04-25T05:43:16.189638+0000","last_active":"2024-04-25T05:45:57.991567+0000","last_peered":"2024-04-25T05:45:57.991567+0000","last_clean":"2024-04-25T05:45:57.991567+0000","last_became_active":"2024-04-25T05:43:16.189291+0000","last_became_peered":"2024-04-25T05:43:16.189291+0000","last_unstale":"2024-04-25T05:45:57.991567+0000","last_undegraded":"2024-04-25T05:45:57.991567+0000","last_fullsized":"2024-04-25T05:45:57.991567+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:08.356012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:08.356012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899207+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":947,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985713+0000","last_change":"2024-04-25T05:44:30.948007+0000","last_active":"2024-04-25T05:45:57.985713+0000","last_peered":"2024-04-25T05:45:57.985713+0000","last_clean":"2024-04-25T05:45:57.985713+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:45:57.985713+0000","last_undegraded":"2024-04-25T05:45:57.985713+0000","last_fullsized":"2024-04-25T05:45:57.985713+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:42:18.235819+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":141,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990110+0000","last_change":"2024-04-25T05:45:37.037715+0000","last_active":"2024-04-25T05:45:57.990110+0000","last_peered":"2024-04-25T05:45:57.990110+0000","last_clean":"2024-04-25T05:45:57.990110+0000","last_became_active":"2024-04-25T05:43:16.187499+0000","last_became_peered":"2024-04-25T05:43:16.187499+0000","last_unstale":"2024-04-25T05:45:57.990110+0000","last_undegraded":"2024-04-25T05:45:57.990110+0000","last_fullsized":"2024-04-25T05:45:57.990110+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:37.037435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.037435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.222112+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"362'13807","reported_seq":13941,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.832188+0000","last_change":"2024-04-25T05:43:16.196515+0000","last_active":"2024-04-25T05:45:58.832188+0000","last_peered":"2024-04-25T05:45:58.832188+0000","last_clean":"2024-04-25T05:45:58.832188+0000","last_became_active":"2024-04-25T05:43:16.196169+0000","last_became_peered":"2024-04-25T05:43:16.196169+0000","last_unstale":"2024-04-25T05:45:58.832188+0000","last_undegraded":"2024-04-25T05:45:58.832188+0000","last_fullsized":"2024-04-25T05:45:58.832188+0000","mapping_epoch":301,"log_start":"350'10870","ondisk_log_start":"350'10870","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2937,"log_dups_size":3000,"ondisk_log_size":2937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13807,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":143,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986742+0000","last_change":"2024-04-25T05:43:21.150005+0000","last_active":"2024-04-25T05:45:57.986742+0000","last_peered":"2024-04-25T05:45:57.986742+0000","last_clean":"2024-04-25T05:45:57.986742+0000","last_became_active":"2024-04-25T05:43:16.193882+0000","last_became_peered":"2024-04-25T05:43:16.193882+0000","last_unstale":"2024-04-25T05:45:57.986742+0000","last_undegraded":"2024-04-25T05:45:57.986742+0000","last_fullsized":"2024-04-25T05:45:57.986742+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":756,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985800+0000","last_change":"2024-04-25T05:44:28.221414+0000","last_active":"2024-04-25T05:45:57.985800+0000","last_peered":"2024-04-25T05:45:57.985800+0000","last_clean":"2024-04-25T05:45:57.985800+0000","last_became_active":"2024-04-25T05:44:28.179973+0000","last_became_peered":"2024-04-25T05:44:28.179973+0000","last_unstale":"2024-04-25T05:45:57.985800+0000","last_undegraded":"2024-04-25T05:45:57.985800+0000","last_fullsized":"2024-04-25T05:45:57.985800+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":649,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.988901+0000","last_change":"2024-04-25T05:44:28.186329+0000","last_active":"2024-04-25T05:45:57.988901+0000","last_peered":"2024-04-25T05:45:57.988901+0000","last_clean":"2024-04-25T05:45:57.988901+0000","last_became_active":"2024-04-25T05:44:28.184251+0000","last_became_peered":"2024-04-25T05:44:28.184251+0000","last_unstale":"2024-04-25T05:45:57.988901+0000","last_undegraded":"2024-04-25T05:45:57.988901+0000","last_fullsized":"2024-04-25T05:45:57.988901+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:55.459042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:55.459042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899644+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":667,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985336+0000","last_change":"2024-04-25T05:44:28.183966+0000","last_active":"2024-04-25T05:45:57.985336+0000","last_peered":"2024-04-25T05:45:57.985336+0000","last_clean":"2024-04-25T05:45:57.985336+0000","last_became_active":"2024-04-25T05:44:28.183528+0000","last_became_peered":"2024-04-25T05:44:28.183528+0000","last_unstale":"2024-04-25T05:45:57.985336+0000","last_undegraded":"2024-04-25T05:45:57.985336+0000","last_fullsized":"2024-04-25T05:45:57.985336+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986496+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:45:57.986496+0000","last_peered":"2024-04-25T05:45:57.986496+0000","last_clean":"2024-04-25T05:45:57.986496+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:45:57.986496+0000","last_undegraded":"2024-04-25T05:45:57.986496+0000","last_fullsized":"2024-04-25T05:45:57.986496+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":953,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986192+0000","last_change":"2024-04-25T05:45:30.250224+0000","last_active":"2024-04-25T05:45:57.986192+0000","last_peered":"2024-04-25T05:45:57.986192+0000","last_clean":"2024-04-25T05:45:57.986192+0000","last_became_active":"2024-04-25T05:44:30.946674+0000","last_became_peered":"2024-04-25T05:44:30.946674+0000","last_unstale":"2024-04-25T05:45:57.986192+0000","last_undegraded":"2024-04-25T05:45:57.986192+0000","last_fullsized":"2024-04-25T05:45:57.986192+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:57.319675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"361'13517","reported_seq":27802,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984765+0000","last_change":"2024-04-25T05:44:47.251341+0000","last_active":"2024-04-25T05:45:57.984765+0000","last_peered":"2024-04-25T05:45:57.984765+0000","last_clean":"2024-04-25T05:45:57.984765+0000","last_became_active":"2024-04-25T05:44:47.245906+0000","last_became_peered":"2024-04-25T05:44:47.245906+0000","last_unstale":"2024-04-25T05:45:57.984765+0000","last_undegraded":"2024-04-25T05:45:57.984765+0000","last_fullsized":"2024-04-25T05:45:57.984765+0000","mapping_epoch":332,"log_start":"349'10208","ondisk_log_start":"349'10208","created":296,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":5055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13517,"num_write_kb":486464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7061,"num_bytes_recovered":462749696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":262,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987468+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:45:57.987468+0000","last_peered":"2024-04-25T05:45:57.987468+0000","last_clean":"2024-04-25T05:45:57.987468+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:45:57.987468+0000","last_undegraded":"2024-04-25T05:45:57.987468+0000","last_fullsized":"2024-04-25T05:45:57.987468+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.475952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"320'58","reported_seq":888,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986063+0000","last_change":"2024-04-25T05:42:54.994076+0000","last_active":"2024-04-25T05:45:57.986063+0000","last_peered":"2024-04-25T05:45:57.986063+0000","last_clean":"2024-04-25T05:45:57.986063+0000","last_became_active":"2024-04-25T05:40:55.991813+0000","last_became_peered":"2024-04-25T05:40:55.991813+0000","last_unstale":"2024-04-25T05:45:57.986063+0000","last_undegraded":"2024-04-25T05:45:57.986063+0000","last_fullsized":"2024-04-25T05:45:57.986063+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'46","last_scrub_stamp":"2024-04-25T05:42:54.993908+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.993908+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":422,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975236+0000","last_change":"2024-04-25T05:44:28.160664+0000","last_active":"2024-04-25T05:45:53.975236+0000","last_peered":"2024-04-25T05:45:53.975236+0000","last_clean":"2024-04-25T05:45:53.975236+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:45:53.975236+0000","last_undegraded":"2024-04-25T05:45:53.975236+0000","last_fullsized":"2024-04-25T05:45:53.975236+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.447620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.447620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756370+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":169,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986929+0000","last_change":"2024-04-25T05:44:28.205511+0000","last_active":"2024-04-25T05:45:57.986929+0000","last_peered":"2024-04-25T05:45:57.986929+0000","last_clean":"2024-04-25T05:45:57.986929+0000","last_became_active":"2024-04-25T05:44:28.176104+0000","last_became_peered":"2024-04-25T05:44:28.176104+0000","last_unstale":"2024-04-25T05:45:57.986929+0000","last_undegraded":"2024-04-25T05:45:57.986929+0000","last_fullsized":"2024-04-25T05:45:57.986929+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:42:44.047795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:42:44.047795+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":609,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989567+0000","last_change":"2024-04-25T05:42:16.363519+0000","last_active":"2024-04-25T05:45:57.989567+0000","last_peered":"2024-04-25T05:45:57.989567+0000","last_clean":"2024-04-25T05:45:57.989567+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:45:57.989567+0000","last_undegraded":"2024-04-25T05:45:57.989567+0000","last_fullsized":"2024-04-25T05:45:57.989567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:16.363206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:16.363206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899291+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":647,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967460+0000","last_change":"2024-04-25T05:44:30.918203+0000","last_active":"2024-04-25T05:45:53.967460+0000","last_peered":"2024-04-25T05:45:53.967460+0000","last_clean":"2024-04-25T05:45:53.967460+0000","last_became_active":"2024-04-25T05:44:30.917713+0000","last_became_peered":"2024-04-25T05:44:30.917713+0000","last_unstale":"2024-04-25T05:45:53.967460+0000","last_undegraded":"2024-04-25T05:45:53.967460+0000","last_fullsized":"2024-04-25T05:45:53.967460+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:10.499299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:10.499299+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.533896+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"362'13472","reported_seq":26765,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.803004+0000","last_change":"2024-04-25T05:45:22.530073+0000","last_active":"2024-04-25T05:45:58.803004+0000","last_peered":"2024-04-25T05:45:58.803004+0000","last_clean":"2024-04-25T05:45:58.803004+0000","last_became_active":"2024-04-25T05:45:17.505735+0000","last_became_peered":"2024-04-25T05:45:17.505735+0000","last_unstale":"2024-04-25T05:45:58.803004+0000","last_undegraded":"2024-04-25T05:45:58.803004+0000","last_fullsized":"2024-04-25T05:45:58.803004+0000","mapping_epoch":346,"log_start":"350'10491","ondisk_log_start":"350'10491","created":296,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10068","last_scrub_stamp":"2024-04-25T05:45:22.529746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.529746+0000","objects_scrubbed":5089,"log_size":2981,"log_dups_size":3000,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:51.373198+0000","scrub_duration":1818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":1600,"num_object_clones":0,"num_object_copies":4800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13472,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6540,"num_bytes_recovered":428539904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":347,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984015+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:45:57.984015+0000","last_peered":"2024-04-25T05:45:57.984015+0000","last_clean":"2024-04-25T05:45:57.984015+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:45:57.984015+0000","last_undegraded":"2024-04-25T05:45:57.984015+0000","last_fullsized":"2024-04-25T05:45:57.984015+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":447,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989266+0000","last_change":"2024-04-25T05:45:27.997352+0000","last_active":"2024-04-25T05:45:57.989266+0000","last_peered":"2024-04-25T05:45:57.989266+0000","last_clean":"2024-04-25T05:45:57.989266+0000","last_became_active":"2024-04-25T05:43:16.191496+0000","last_became_peered":"2024-04-25T05:43:16.191496+0000","last_unstale":"2024-04-25T05:45:57.989266+0000","last_undegraded":"2024-04-25T05:45:57.989266+0000","last_fullsized":"2024-04-25T05:45:57.989266+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:41.581422+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":284,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967336+0000","last_change":"2024-04-25T05:45:32.512149+0000","last_active":"2024-04-25T05:45:53.967336+0000","last_peered":"2024-04-25T05:45:53.967336+0000","last_clean":"2024-04-25T05:45:53.967336+0000","last_became_active":"2024-04-25T05:44:28.171004+0000","last_became_peered":"2024-04-25T05:44:28.171004+0000","last_unstale":"2024-04-25T05:45:53.967336+0000","last_undegraded":"2024-04-25T05:45:53.967336+0000","last_fullsized":"2024-04-25T05:45:53.967336+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:00.371199+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":248,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986308+0000","last_change":"2024-04-25T05:44:28.169508+0000","last_active":"2024-04-25T05:45:57.986308+0000","last_peered":"2024-04-25T05:45:57.986308+0000","last_clean":"2024-04-25T05:45:57.986308+0000","last_became_active":"2024-04-25T05:44:28.169146+0000","last_became_peered":"2024-04-25T05:44:28.169146+0000","last_unstale":"2024-04-25T05:45:57.986308+0000","last_undegraded":"2024-04-25T05:45:57.986308+0000","last_fullsized":"2024-04-25T05:45:57.986308+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"361'13246","reported_seq":13380,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985459+0000","last_change":"2024-04-25T05:43:16.191972+0000","last_active":"2024-04-25T05:45:57.985459+0000","last_peered":"2024-04-25T05:45:57.985459+0000","last_clean":"2024-04-25T05:45:57.985459+0000","last_became_active":"2024-04-25T05:43:16.191575+0000","last_became_peered":"2024-04-25T05:43:16.191575+0000","last_unstale":"2024-04-25T05:45:57.985459+0000","last_undegraded":"2024-04-25T05:45:57.985459+0000","last_fullsized":"2024-04-25T05:45:57.985459+0000","mapping_epoch":301,"log_start":"349'9955","ondisk_log_start":"349'9955","created":296,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3291,"log_dups_size":0,"ondisk_log_size":3291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109182976,"num_objects":1666,"num_object_clones":0,"num_object_copies":4998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13246,"num_write_kb":477184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":396,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.974891+0000","last_change":"2024-04-25T05:43:31.634282+0000","last_active":"2024-04-25T05:45:53.974891+0000","last_peered":"2024-04-25T05:45:53.974891+0000","last_clean":"2024-04-25T05:45:53.974891+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:45:53.974891+0000","last_undegraded":"2024-04-25T05:45:53.974891+0000","last_fullsized":"2024-04-25T05:45:53.974891+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'25","last_scrub_stamp":"2024-04-25T05:43:31.634141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.634141+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":673,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986359+0000","last_change":"2024-04-25T05:43:16.225042+0000","last_active":"2024-04-25T05:45:57.986359+0000","last_peered":"2024-04-25T05:45:57.986359+0000","last_clean":"2024-04-25T05:45:57.986359+0000","last_became_active":"2024-04-25T05:43:16.203557+0000","last_became_peered":"2024-04-25T05:43:16.203557+0000","last_unstale":"2024-04-25T05:45:57.986359+0000","last_undegraded":"2024-04-25T05:45:57.986359+0000","last_fullsized":"2024-04-25T05:45:57.986359+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:42:58.053271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:42:58.053271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":581,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984527+0000","last_change":"2024-04-25T05:44:28.180755+0000","last_active":"2024-04-25T05:45:57.984527+0000","last_peered":"2024-04-25T05:45:57.984527+0000","last_clean":"2024-04-25T05:45:57.984527+0000","last_became_active":"2024-04-25T05:44:28.180393+0000","last_became_peered":"2024-04-25T05:44:28.180393+0000","last_unstale":"2024-04-25T05:45:57.984527+0000","last_undegraded":"2024-04-25T05:45:57.984527+0000","last_fullsized":"2024-04-25T05:45:57.984527+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":631,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985149+0000","last_change":"2024-04-25T05:44:00.144990+0000","last_active":"2024-04-25T05:45:57.985149+0000","last_peered":"2024-04-25T05:45:57.985149+0000","last_clean":"2024-04-25T05:45:57.985149+0000","last_became_active":"2024-04-25T05:40:55.981111+0000","last_became_peered":"2024-04-25T05:40:55.981111+0000","last_unstale":"2024-04-25T05:45:57.985149+0000","last_undegraded":"2024-04-25T05:45:57.985149+0000","last_fullsized":"2024-04-25T05:45:57.985149+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"362'13644","reported_seq":13786,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:58.849254+0000","last_change":"2024-04-25T05:43:10.136848+0000","last_active":"2024-04-25T05:45:58.849254+0000","last_peered":"2024-04-25T05:45:58.849254+0000","last_clean":"2024-04-25T05:45:58.849254+0000","last_became_active":"2024-04-25T05:43:10.136556+0000","last_became_peered":"2024-04-25T05:43:10.136556+0000","last_unstale":"2024-04-25T05:45:58.849254+0000","last_undegraded":"2024-04-25T05:45:58.849254+0000","last_fullsized":"2024-04-25T05:45:58.849254+0000","mapping_epoch":296,"log_start":"350'10686","ondisk_log_start":"350'10686","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":3000,"ondisk_log_size":2958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109445120,"num_objects":1670,"num_object_clones":0,"num_object_copies":5010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13644,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":582,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986386+0000","last_change":"2024-04-25T05:44:28.169954+0000","last_active":"2024-04-25T05:45:57.986386+0000","last_peered":"2024-04-25T05:45:57.986386+0000","last_clean":"2024-04-25T05:45:57.986386+0000","last_became_active":"2024-04-25T05:44:28.168753+0000","last_became_peered":"2024-04-25T05:44:28.168753+0000","last_unstale":"2024-04-25T05:45:57.986386+0000","last_undegraded":"2024-04-25T05:45:57.986386+0000","last_fullsized":"2024-04-25T05:45:57.986386+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":287,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.983980+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:45:57.983980+0000","last_peered":"2024-04-25T05:45:57.983980+0000","last_clean":"2024-04-25T05:45:57.983980+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:45:57.983980+0000","last_undegraded":"2024-04-25T05:45:57.983980+0000","last_fullsized":"2024-04-25T05:45:57.983980+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986574+0000","last_change":"2024-04-25T05:42:51.060336+0000","last_active":"2024-04-25T05:45:57.986574+0000","last_peered":"2024-04-25T05:45:57.986574+0000","last_clean":"2024-04-25T05:45:57.986574+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:45:57.986574+0000","last_undegraded":"2024-04-25T05:45:57.986574+0000","last_fullsized":"2024-04-25T05:45:57.986574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.060223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":621,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992528+0000","last_change":"2024-04-25T05:43:27.607471+0000","last_active":"2024-04-25T05:45:57.992528+0000","last_peered":"2024-04-25T05:45:57.992528+0000","last_clean":"2024-04-25T05:45:57.992528+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:45:57.992528+0000","last_undegraded":"2024-04-25T05:45:57.992528+0000","last_fullsized":"2024-04-25T05:45:57.992528+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:27.607347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:27.607347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899850+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"362'13433","reported_seq":27360,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987682+0000","last_change":"2024-04-25T05:45:09.944174+0000","last_active":"2024-04-25T05:45:57.987682+0000","last_peered":"2024-04-25T05:45:57.987682+0000","last_clean":"2024-04-25T05:45:57.987682+0000","last_became_active":"2024-04-25T05:45:09.927273+0000","last_became_peered":"2024-04-25T05:45:09.927273+0000","last_unstale":"2024-04-25T05:45:57.987682+0000","last_undegraded":"2024-04-25T05:45:57.987682+0000","last_fullsized":"2024-04-25T05:45:57.987682+0000","mapping_epoch":342,"log_start":"349'10193","ondisk_log_start":"349'10193","created":296,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:09.125645+0000","last_clean_scrub_stamp":"2024-04-25T05:43:09.125645+0000","objects_scrubbed":0,"log_size":3240,"log_dups_size":0,"ondisk_log_size":3240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13433,"num_write_kb":481920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6881,"num_bytes_recovered":450953216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":805,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986562+0000","last_change":"2024-04-25T05:42:42.050956+0000","last_active":"2024-04-25T05:45:57.986562+0000","last_peered":"2024-04-25T05:45:57.986562+0000","last_clean":"2024-04-25T05:45:57.986562+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:45:57.986562+0000","last_undegraded":"2024-04-25T05:45:57.986562+0000","last_fullsized":"2024-04-25T05:45:57.986562+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:42:42.050875+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:42:42.050875+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":239,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985792+0000","last_change":"2024-04-25T05:43:16.220722+0000","last_active":"2024-04-25T05:45:57.985792+0000","last_peered":"2024-04-25T05:45:57.985792+0000","last_clean":"2024-04-25T05:45:57.985792+0000","last_became_active":"2024-04-25T05:43:16.190381+0000","last_became_peered":"2024-04-25T05:43:16.190381+0000","last_unstale":"2024-04-25T05:45:57.985792+0000","last_undegraded":"2024-04-25T05:45:57.985792+0000","last_fullsized":"2024-04-25T05:45:57.985792+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":685,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986073+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:45:57.986073+0000","last_peered":"2024-04-25T05:45:57.986073+0000","last_clean":"2024-04-25T05:45:57.986073+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:45:57.986073+0000","last_undegraded":"2024-04-25T05:45:57.986073+0000","last_fullsized":"2024-04-25T05:45:57.986073+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":510,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.974669+0000","last_change":"2024-04-25T05:44:28.174865+0000","last_active":"2024-04-25T05:45:53.974669+0000","last_peered":"2024-04-25T05:45:53.974669+0000","last_clean":"2024-04-25T05:45:53.974669+0000","last_became_active":"2024-04-25T05:44:28.174499+0000","last_became_peered":"2024-04-25T05:44:28.174499+0000","last_unstale":"2024-04-25T05:45:53.974669+0000","last_undegraded":"2024-04-25T05:45:53.974669+0000","last_fullsized":"2024-04-25T05:45:53.974669+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.676519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.676519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":654,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990381+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:45:57.990381+0000","last_peered":"2024-04-25T05:45:57.990381+0000","last_clean":"2024-04-25T05:45:57.990381+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:45:57.990381+0000","last_undegraded":"2024-04-25T05:45:57.990381+0000","last_fullsized":"2024-04-25T05:45:57.990381+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900291+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":368,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.974822+0000","last_change":"2024-04-25T05:44:28.191586+0000","last_active":"2024-04-25T05:45:53.974822+0000","last_peered":"2024-04-25T05:45:53.974822+0000","last_clean":"2024-04-25T05:45:53.974822+0000","last_became_active":"2024-04-25T05:44:28.191288+0000","last_became_peered":"2024-04-25T05:44:28.191288+0000","last_unstale":"2024-04-25T05:45:53.974822+0000","last_undegraded":"2024-04-25T05:45:53.974822+0000","last_fullsized":"2024-04-25T05:45:53.974822+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.716119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.716119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756420+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":141,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989359+0000","last_change":"2024-04-25T05:44:19.931291+0000","last_active":"2024-04-25T05:45:57.989359+0000","last_peered":"2024-04-25T05:45:57.989359+0000","last_clean":"2024-04-25T05:45:57.989359+0000","last_became_active":"2024-04-25T05:43:16.189541+0000","last_became_peered":"2024-04-25T05:43:16.189541+0000","last_unstale":"2024-04-25T05:45:57.989359+0000","last_undegraded":"2024-04-25T05:45:57.989359+0000","last_fullsized":"2024-04-25T05:45:57.989359+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":636,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986227+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:45:57.986227+0000","last_peered":"2024-04-25T05:45:57.986227+0000","last_clean":"2024-04-25T05:45:57.986227+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:45:57.986227+0000","last_undegraded":"2024-04-25T05:45:57.986227+0000","last_fullsized":"2024-04-25T05:45:57.986227+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":315,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989466+0000","last_change":"2024-04-25T05:44:28.203127+0000","last_active":"2024-04-25T05:45:57.989466+0000","last_peered":"2024-04-25T05:45:57.989466+0000","last_clean":"2024-04-25T05:45:57.989466+0000","last_became_active":"2024-04-25T05:44:28.184563+0000","last_became_peered":"2024-04-25T05:44:28.184563+0000","last_unstale":"2024-04-25T05:45:57.989466+0000","last_undegraded":"2024-04-25T05:45:57.989466+0000","last_fullsized":"2024-04-25T05:45:57.989466+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'21","last_scrub_stamp":"2024-04-25T05:42:21.659389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:21.659389+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899363+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":655,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986358+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:45:57.986358+0000","last_peered":"2024-04-25T05:45:57.986358+0000","last_clean":"2024-04-25T05:45:57.986358+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:45:57.986358+0000","last_undegraded":"2024-04-25T05:45:57.986358+0000","last_fullsized":"2024-04-25T05:45:57.986358+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":612,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990199+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:45:57.990199+0000","last_peered":"2024-04-25T05:45:57.990199+0000","last_clean":"2024-04-25T05:45:57.990199+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:45:57.990199+0000","last_undegraded":"2024-04-25T05:45:57.990199+0000","last_fullsized":"2024-04-25T05:45:57.990199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900325+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":277,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.968701+0000","last_change":"2024-04-25T05:44:28.120205+0000","last_active":"2024-04-25T05:45:53.968701+0000","last_peered":"2024-04-25T05:45:53.968701+0000","last_clean":"2024-04-25T05:45:53.968701+0000","last_became_active":"2024-04-25T05:44:28.119749+0000","last_became_peered":"2024-04-25T05:44:28.119749+0000","last_unstale":"2024-04-25T05:45:53.968701+0000","last_undegraded":"2024-04-25T05:45:53.968701+0000","last_fullsized":"2024-04-25T05:45:53.968701+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.165638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.165638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.533859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":266,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986459+0000","last_change":"2024-04-25T05:45:37.317658+0000","last_active":"2024-04-25T05:45:57.986459+0000","last_peered":"2024-04-25T05:45:57.986459+0000","last_clean":"2024-04-25T05:45:57.986459+0000","last_became_active":"2024-04-25T05:43:16.203176+0000","last_became_peered":"2024-04-25T05:43:16.203176+0000","last_unstale":"2024-04-25T05:45:57.986459+0000","last_undegraded":"2024-04-25T05:45:57.986459+0000","last_fullsized":"2024-04-25T05:45:57.986459+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:37.317498+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.317498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:39.511458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":248,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985859+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:45:57.985859+0000","last_peered":"2024-04-25T05:45:57.985859+0000","last_clean":"2024-04-25T05:45:57.985859+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:45:57.985859+0000","last_undegraded":"2024-04-25T05:45:57.985859+0000","last_fullsized":"2024-04-25T05:45:57.985859+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"243'9","reported_seq":775,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989804+0000","last_change":"2024-04-25T05:44:11.019894+0000","last_active":"2024-04-25T05:45:57.989804+0000","last_peered":"2024-04-25T05:45:57.989804+0000","last_clean":"2024-04-25T05:45:57.989804+0000","last_became_active":"2024-04-25T05:40:55.982963+0000","last_became_peered":"2024-04-25T05:40:55.982963+0000","last_unstale":"2024-04-25T05:45:57.989804+0000","last_undegraded":"2024-04-25T05:45:57.989804+0000","last_fullsized":"2024-04-25T05:45:57.989804+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900222+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":92,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985320+0000","last_change":"2024-04-25T05:44:28.167266+0000","last_active":"2024-04-25T05:45:57.985320+0000","last_peered":"2024-04-25T05:45:57.985320+0000","last_clean":"2024-04-25T05:45:57.985320+0000","last_became_active":"2024-04-25T05:44:28.166838+0000","last_became_peered":"2024-04-25T05:44:28.166838+0000","last_unstale":"2024-04-25T05:45:57.985320+0000","last_undegraded":"2024-04-25T05:45:57.985320+0000","last_fullsized":"2024-04-25T05:45:57.985320+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":679,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989114+0000","last_change":"2024-04-25T05:45:32.015466+0000","last_active":"2024-04-25T05:45:57.989114+0000","last_peered":"2024-04-25T05:45:57.989114+0000","last_clean":"2024-04-25T05:45:57.989114+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:45:57.989114+0000","last_undegraded":"2024-04-25T05:45:57.989114+0000","last_fullsized":"2024-04-25T05:45:57.989114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:45:32.015242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.015242+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:44.736841+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":643,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984634+0000","last_change":"2024-04-25T05:44:13.015441+0000","last_active":"2024-04-25T05:45:57.984634+0000","last_peered":"2024-04-25T05:45:57.984634+0000","last_clean":"2024-04-25T05:45:57.984634+0000","last_became_active":"2024-04-25T05:40:55.913098+0000","last_became_peered":"2024-04-25T05:40:55.913098+0000","last_unstale":"2024-04-25T05:45:57.984634+0000","last_undegraded":"2024-04-25T05:45:57.984634+0000","last_fullsized":"2024-04-25T05:45:57.984634+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":204,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987200+0000","last_change":"2024-04-25T05:44:28.236616+0000","last_active":"2024-04-25T05:45:57.987200+0000","last_peered":"2024-04-25T05:45:57.987200+0000","last_clean":"2024-04-25T05:45:57.987200+0000","last_became_active":"2024-04-25T05:44:28.184915+0000","last_became_peered":"2024-04-25T05:44:28.184915+0000","last_unstale":"2024-04-25T05:45:57.987200+0000","last_undegraded":"2024-04-25T05:45:57.987200+0000","last_fullsized":"2024-04-25T05:45:57.987200+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'10","last_scrub_stamp":"2024-04-25T05:41:45.481621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:41:45.481621+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":248,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986306+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:45:57.986306+0000","last_peered":"2024-04-25T05:45:57.986306+0000","last_clean":"2024-04-25T05:45:57.986306+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:45:57.986306+0000","last_undegraded":"2024-04-25T05:45:57.986306+0000","last_fullsized":"2024-04-25T05:45:57.986306+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":747,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.968304+0000","last_change":"2024-04-25T05:44:19.254245+0000","last_active":"2024-04-25T05:45:53.968304+0000","last_peered":"2024-04-25T05:45:53.968304+0000","last_clean":"2024-04-25T05:45:53.968304+0000","last_became_active":"2024-04-25T05:40:55.979627+0000","last_became_peered":"2024-04-25T05:40:55.979627+0000","last_unstale":"2024-04-25T05:45:53.968304+0000","last_undegraded":"2024-04-25T05:45:53.968304+0000","last_fullsized":"2024-04-25T05:45:53.968304+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:19.254124+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.254124+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.533928+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":316,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984819+0000","last_change":"2024-04-25T05:45:33.294954+0000","last_active":"2024-04-25T05:45:57.984819+0000","last_peered":"2024-04-25T05:45:57.984819+0000","last_clean":"2024-04-25T05:45:57.984819+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:45:57.984819+0000","last_undegraded":"2024-04-25T05:45:57.984819+0000","last_fullsized":"2024-04-25T05:45:57.984819+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:03.065653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":630,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987344+0000","last_change":"2024-04-25T05:43:18.284063+0000","last_active":"2024-04-25T05:45:57.987344+0000","last_peered":"2024-04-25T05:45:57.987344+0000","last_clean":"2024-04-25T05:45:57.987344+0000","last_became_active":"2024-04-25T05:43:16.199098+0000","last_became_peered":"2024-04-25T05:43:16.199098+0000","last_unstale":"2024-04-25T05:45:57.987344+0000","last_undegraded":"2024-04-25T05:45:57.987344+0000","last_fullsized":"2024-04-25T05:45:57.987344+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":641,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.988719+0000","last_change":"2024-04-25T05:44:28.186508+0000","last_active":"2024-04-25T05:45:57.988719+0000","last_peered":"2024-04-25T05:45:57.988719+0000","last_clean":"2024-04-25T05:45:57.988719+0000","last_became_active":"2024-04-25T05:44:28.184914+0000","last_became_peered":"2024-04-25T05:44:28.184914+0000","last_unstale":"2024-04-25T05:45:57.988719+0000","last_undegraded":"2024-04-25T05:45:57.988719+0000","last_fullsized":"2024-04-25T05:45:57.988719+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.480279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899983+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":167,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984775+0000","last_change":"2024-04-25T05:44:28.208196+0000","last_active":"2024-04-25T05:45:57.984775+0000","last_peered":"2024-04-25T05:45:57.984775+0000","last_clean":"2024-04-25T05:45:57.984775+0000","last_became_active":"2024-04-25T05:44:28.160392+0000","last_became_peered":"2024-04-25T05:44:28.160392+0000","last_unstale":"2024-04-25T05:45:57.984775+0000","last_undegraded":"2024-04-25T05:45:57.984775+0000","last_fullsized":"2024-04-25T05:45:57.984775+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'13","last_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:41:30.532371+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":636,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986197+0000","last_change":"2024-04-25T05:42:56.099337+0000","last_active":"2024-04-25T05:45:57.986197+0000","last_peered":"2024-04-25T05:45:57.986197+0000","last_clean":"2024-04-25T05:45:57.986197+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:45:57.986197+0000","last_undegraded":"2024-04-25T05:45:57.986197+0000","last_fullsized":"2024-04-25T05:45:57.986197+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:56.099224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:42:56.099224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":631,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992344+0000","last_change":"2024-04-25T05:44:28.174271+0000","last_active":"2024-04-25T05:45:57.992344+0000","last_peered":"2024-04-25T05:45:57.992344+0000","last_clean":"2024-04-25T05:45:57.992344+0000","last_became_active":"2024-04-25T05:44:28.172179+0000","last_became_peered":"2024-04-25T05:44:28.172179+0000","last_unstale":"2024-04-25T05:45:57.992344+0000","last_undegraded":"2024-04-25T05:45:57.992344+0000","last_fullsized":"2024-04-25T05:45:57.992344+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900253+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":241,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986282+0000","last_change":"2024-04-25T05:44:28.179459+0000","last_active":"2024-04-25T05:45:57.986282+0000","last_peered":"2024-04-25T05:45:57.986282+0000","last_clean":"2024-04-25T05:45:57.986282+0000","last_became_active":"2024-04-25T05:44:28.176134+0000","last_became_peered":"2024-04-25T05:44:28.176134+0000","last_unstale":"2024-04-25T05:45:57.986282+0000","last_undegraded":"2024-04-25T05:45:57.986282+0000","last_fullsized":"2024-04-25T05:45:57.986282+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":708,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989173+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:45:57.989173+0000","last_peered":"2024-04-25T05:45:57.989173+0000","last_clean":"2024-04-25T05:45:57.989173+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:45:57.989173+0000","last_undegraded":"2024-04-25T05:45:57.989173+0000","last_fullsized":"2024-04-25T05:45:57.989173+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900100+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":141,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986477+0000","last_change":"2024-04-25T05:43:17.056662+0000","last_active":"2024-04-25T05:45:57.986477+0000","last_peered":"2024-04-25T05:45:57.986477+0000","last_clean":"2024-04-25T05:45:57.986477+0000","last_became_active":"2024-04-25T05:43:16.205164+0000","last_became_peered":"2024-04-25T05:43:16.205164+0000","last_unstale":"2024-04-25T05:45:57.986477+0000","last_undegraded":"2024-04-25T05:45:57.986477+0000","last_fullsized":"2024-04-25T05:45:57.986477+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":647,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985971+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:45:57.985971+0000","last_peered":"2024-04-25T05:45:57.985971+0000","last_clean":"2024-04-25T05:45:57.985971+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:45:57.985971+0000","last_undegraded":"2024-04-25T05:45:57.985971+0000","last_fullsized":"2024-04-25T05:45:57.985971+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992617+0000","last_change":"2024-04-25T05:44:28.179442+0000","last_active":"2024-04-25T05:45:57.992617+0000","last_peered":"2024-04-25T05:45:57.992617+0000","last_clean":"2024-04-25T05:45:57.992617+0000","last_became_active":"2024-04-25T05:44:28.179103+0000","last_became_peered":"2024-04-25T05:44:28.179103+0000","last_unstale":"2024-04-25T05:45:57.992617+0000","last_undegraded":"2024-04-25T05:45:57.992617+0000","last_fullsized":"2024-04-25T05:45:57.992617+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:20.379780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:42:20.379780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899329+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":661,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989441+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:45:57.989441+0000","last_peered":"2024-04-25T05:45:57.989441+0000","last_clean":"2024-04-25T05:45:57.989441+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:45:57.989441+0000","last_undegraded":"2024-04-25T05:45:57.989441+0000","last_fullsized":"2024-04-25T05:45:57.989441+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:44.810725+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":233,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989937+0000","last_change":"2024-04-25T05:44:28.188135+0000","last_active":"2024-04-25T05:45:57.989937+0000","last_peered":"2024-04-25T05:45:57.989937+0000","last_clean":"2024-04-25T05:45:57.989937+0000","last_became_active":"2024-04-25T05:44:28.187806+0000","last_became_peered":"2024-04-25T05:44:28.187806+0000","last_unstale":"2024-04-25T05:45:57.989937+0000","last_undegraded":"2024-04-25T05:45:57.989937+0000","last_fullsized":"2024-04-25T05:45:57.989937+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":350,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.977763+0000","last_change":"2024-04-25T05:45:29.823062+0000","last_active":"2024-04-25T05:45:53.977763+0000","last_peered":"2024-04-25T05:45:53.977763+0000","last_clean":"2024-04-25T05:45:53.977763+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:45:53.977763+0000","last_undegraded":"2024-04-25T05:45:53.977763+0000","last_fullsized":"2024-04-25T05:45:53.977763+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.822937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.473503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":636,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986307+0000","last_change":"2024-04-25T05:44:19.960606+0000","last_active":"2024-04-25T05:45:57.986307+0000","last_peered":"2024-04-25T05:45:57.986307+0000","last_clean":"2024-04-25T05:45:57.986307+0000","last_became_active":"2024-04-25T05:40:55.982143+0000","last_became_peered":"2024-04-25T05:40:55.982143+0000","last_unstale":"2024-04-25T05:45:57.986307+0000","last_undegraded":"2024-04-25T05:45:57.986307+0000","last_fullsized":"2024-04-25T05:45:57.986307+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":423,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976890+0000","last_change":"2024-04-25T05:44:28.212080+0000","last_active":"2024-04-25T05:45:53.976890+0000","last_peered":"2024-04-25T05:45:53.976890+0000","last_clean":"2024-04-25T05:45:53.976890+0000","last_became_active":"2024-04-25T05:44:28.181996+0000","last_became_peered":"2024-04-25T05:44:28.181996+0000","last_unstale":"2024-04-25T05:45:53.976890+0000","last_undegraded":"2024-04-25T05:45:53.976890+0000","last_fullsized":"2024-04-25T05:45:53.976890+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'23","last_scrub_stamp":"2024-04-25T05:44:03.711908+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.711908+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":646,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989294+0000","last_change":"2024-04-25T05:44:28.172975+0000","last_active":"2024-04-25T05:45:57.989294+0000","last_peered":"2024-04-25T05:45:57.989294+0000","last_clean":"2024-04-25T05:45:57.989294+0000","last_became_active":"2024-04-25T05:44:28.171311+0000","last_became_peered":"2024-04-25T05:44:28.171311+0000","last_unstale":"2024-04-25T05:45:57.989294+0000","last_undegraded":"2024-04-25T05:45:57.989294+0000","last_fullsized":"2024-04-25T05:45:57.989294+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900504+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":248,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986643+0000","last_change":"2024-04-25T05:44:28.179599+0000","last_active":"2024-04-25T05:45:57.986643+0000","last_peered":"2024-04-25T05:45:57.986643+0000","last_clean":"2024-04-25T05:45:57.986643+0000","last_became_active":"2024-04-25T05:44:28.178710+0000","last_became_peered":"2024-04-25T05:44:28.178710+0000","last_unstale":"2024-04-25T05:45:57.986643+0000","last_undegraded":"2024-04-25T05:45:57.986643+0000","last_fullsized":"2024-04-25T05:45:57.986643+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":622,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986974+0000","last_change":"2024-04-25T05:43:21.410956+0000","last_active":"2024-04-25T05:45:57.986974+0000","last_peered":"2024-04-25T05:45:57.986974+0000","last_clean":"2024-04-25T05:45:57.986974+0000","last_became_active":"2024-04-25T05:43:16.189944+0000","last_became_peered":"2024-04-25T05:43:16.189944+0000","last_unstale":"2024-04-25T05:45:57.986974+0000","last_undegraded":"2024-04-25T05:45:57.986974+0000","last_fullsized":"2024-04-25T05:45:57.986974+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":321,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986734+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:45:57.986734+0000","last_peered":"2024-04-25T05:45:57.986734+0000","last_clean":"2024-04-25T05:45:57.986734+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:45:57.986734+0000","last_undegraded":"2024-04-25T05:45:57.986734+0000","last_fullsized":"2024-04-25T05:45:57.986734+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984195+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:45:57.984195+0000","last_peered":"2024-04-25T05:45:57.984195+0000","last_clean":"2024-04-25T05:45:57.984195+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:45:57.984195+0000","last_undegraded":"2024-04-25T05:45:57.984195+0000","last_fullsized":"2024-04-25T05:45:57.984195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":432,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976623+0000","last_change":"2024-04-25T05:44:02.742949+0000","last_active":"2024-04-25T05:45:53.976623+0000","last_peered":"2024-04-25T05:45:53.976623+0000","last_clean":"2024-04-25T05:45:53.976623+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:45:53.976623+0000","last_undegraded":"2024-04-25T05:45:53.976623+0000","last_fullsized":"2024-04-25T05:45:53.976623+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:44:02.742866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.742866+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":645,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984935+0000","last_change":"2024-04-25T05:43:16.211690+0000","last_active":"2024-04-25T05:45:57.984935+0000","last_peered":"2024-04-25T05:45:57.984935+0000","last_clean":"2024-04-25T05:45:57.984935+0000","last_became_active":"2024-04-25T05:43:16.202634+0000","last_became_peered":"2024-04-25T05:43:16.202634+0000","last_unstale":"2024-04-25T05:45:57.984935+0000","last_undegraded":"2024-04-25T05:45:57.984935+0000","last_fullsized":"2024-04-25T05:45:57.984935+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:42:54.099964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.099964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":660,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.991773+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:45:57.991773+0000","last_peered":"2024-04-25T05:45:57.991773+0000","last_clean":"2024-04-25T05:45:57.991773+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:45:57.991773+0000","last_undegraded":"2024-04-25T05:45:57.991773+0000","last_fullsized":"2024-04-25T05:45:57.991773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899954+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":654,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967181+0000","last_change":"2024-04-25T05:45:02.437804+0000","last_active":"2024-04-25T05:45:53.967181+0000","last_peered":"2024-04-25T05:45:53.967181+0000","last_clean":"2024-04-25T05:45:53.967181+0000","last_became_active":"2024-04-25T05:44:28.167563+0000","last_became_peered":"2024-04-25T05:44:28.167563+0000","last_unstale":"2024-04-25T05:45:53.967181+0000","last_undegraded":"2024-04-25T05:45:53.967181+0000","last_fullsized":"2024-04-25T05:45:53.967181+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:02.437727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:02.437727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:27.778001+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":306,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989362+0000","last_change":"2024-04-25T05:42:48.415254+0000","last_active":"2024-04-25T05:45:57.989362+0000","last_peered":"2024-04-25T05:45:57.989362+0000","last_clean":"2024-04-25T05:45:57.989362+0000","last_became_active":"2024-04-25T05:40:55.973513+0000","last_became_peered":"2024-04-25T05:40:55.973513+0000","last_unstale":"2024-04-25T05:45:57.989362+0000","last_undegraded":"2024-04-25T05:45:57.989362+0000","last_fullsized":"2024-04-25T05:45:57.989362+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:42:48.415168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":627,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990287+0000","last_change":"2024-04-25T05:43:16.191463+0000","last_active":"2024-04-25T05:45:57.990287+0000","last_peered":"2024-04-25T05:45:57.990287+0000","last_clean":"2024-04-25T05:45:57.990287+0000","last_became_active":"2024-04-25T05:43:16.191087+0000","last_became_peered":"2024-04-25T05:43:16.191087+0000","last_unstale":"2024-04-25T05:45:57.990287+0000","last_undegraded":"2024-04-25T05:45:57.990287+0000","last_fullsized":"2024-04-25T05:45:57.990287+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.519375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899584+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":528,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985730+0000","last_change":"2024-04-25T05:44:28.237495+0000","last_active":"2024-04-25T05:45:57.985730+0000","last_peered":"2024-04-25T05:45:57.985730+0000","last_clean":"2024-04-25T05:45:57.985730+0000","last_became_active":"2024-04-25T05:44:28.182958+0000","last_became_peered":"2024-04-25T05:44:28.182958+0000","last_unstale":"2024-04-25T05:45:57.985730+0000","last_undegraded":"2024-04-25T05:45:57.985730+0000","last_fullsized":"2024-04-25T05:45:57.985730+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":124,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984834+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:45:57.984834+0000","last_peered":"2024-04-25T05:45:57.984834+0000","last_clean":"2024-04-25T05:45:57.984834+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:45:57.984834+0000","last_undegraded":"2024-04-25T05:45:57.984834+0000","last_fullsized":"2024-04-25T05:45:57.984834+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":646,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.987375+0000","last_change":"2024-04-25T05:44:28.189092+0000","last_active":"2024-04-25T05:45:57.987375+0000","last_peered":"2024-04-25T05:45:57.987375+0000","last_clean":"2024-04-25T05:45:57.987375+0000","last_became_active":"2024-04-25T05:44:28.185597+0000","last_became_peered":"2024-04-25T05:44:28.185597+0000","last_unstale":"2024-04-25T05:45:57.987375+0000","last_undegraded":"2024-04-25T05:45:57.987375+0000","last_fullsized":"2024-04-25T05:45:57.987375+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":649,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.969917+0000","last_change":"2024-04-25T05:45:20.391712+0000","last_active":"2024-04-25T05:45:53.969917+0000","last_peered":"2024-04-25T05:45:53.969917+0000","last_clean":"2024-04-25T05:45:53.969917+0000","last_became_active":"2024-04-25T05:43:16.188602+0000","last_became_peered":"2024-04-25T05:43:16.188602+0000","last_unstale":"2024-04-25T05:45:53.969917+0000","last_undegraded":"2024-04-25T05:45:53.969917+0000","last_fullsized":"2024-04-25T05:45:53.969917+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:20.391485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:45:20.391485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:34.472250+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":371,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992162+0000","last_change":"2024-04-25T05:43:35.365348+0000","last_active":"2024-04-25T05:45:57.992162+0000","last_peered":"2024-04-25T05:45:57.992162+0000","last_clean":"2024-04-25T05:45:57.992162+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:45:57.992162+0000","last_undegraded":"2024-04-25T05:45:57.992162+0000","last_fullsized":"2024-04-25T05:45:57.992162+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T05:42:13.373848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:13.373848+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899179+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":147,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.968253+0000","last_change":"2024-04-25T05:45:00.435729+0000","last_active":"2024-04-25T05:45:53.968253+0000","last_peered":"2024-04-25T05:45:53.968253+0000","last_clean":"2024-04-25T05:45:53.968253+0000","last_became_active":"2024-04-25T05:43:16.189264+0000","last_became_peered":"2024-04-25T05:43:16.189264+0000","last_unstale":"2024-04-25T05:45:53.968253+0000","last_undegraded":"2024-04-25T05:45:53.968253+0000","last_fullsized":"2024-04-25T05:45:53.968253+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:00.435460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:45:00.435460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:25.355468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":634,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.985942+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:45:57.985942+0000","last_peered":"2024-04-25T05:45:57.985942+0000","last_clean":"2024-04-25T05:45:57.985942+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:45:57.985942+0000","last_undegraded":"2024-04-25T05:45:57.985942+0000","last_fullsized":"2024-04-25T05:45:57.985942+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:54.988841+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":147,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984497+0000","last_change":"2024-04-25T05:45:38.767196+0000","last_active":"2024-04-25T05:45:57.984497+0000","last_peered":"2024-04-25T05:45:57.984497+0000","last_clean":"2024-04-25T05:45:57.984497+0000","last_became_active":"2024-04-25T05:44:28.200623+0000","last_became_peered":"2024-04-25T05:44:28.200623+0000","last_unstale":"2024-04-25T05:45:57.984497+0000","last_undegraded":"2024-04-25T05:45:57.984497+0000","last_fullsized":"2024-04-25T05:45:57.984497+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:38.767074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.767074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:55.134052+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":408,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.991717+0000","last_change":"2024-04-25T05:44:28.198166+0000","last_active":"2024-04-25T05:45:57.991717+0000","last_peered":"2024-04-25T05:45:57.991717+0000","last_clean":"2024-04-25T05:45:57.991717+0000","last_became_active":"2024-04-25T05:44:28.179156+0000","last_became_peered":"2024-04-25T05:44:28.179156+0000","last_unstale":"2024-04-25T05:45:57.991717+0000","last_undegraded":"2024-04-25T05:45:57.991717+0000","last_fullsized":"2024-04-25T05:45:57.991717+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'22","last_scrub_stamp":"2024-04-25T05:42:19.250272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:19.250272+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":445,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976113+0000","last_change":"2024-04-25T05:44:06.726332+0000","last_active":"2024-04-25T05:45:53.976113+0000","last_peered":"2024-04-25T05:45:53.976113+0000","last_clean":"2024-04-25T05:45:53.976113+0000","last_became_active":"2024-04-25T05:43:16.186397+0000","last_became_peered":"2024-04-25T05:43:16.186397+0000","last_unstale":"2024-04-25T05:45:53.976113+0000","last_undegraded":"2024-04-25T05:45:53.976113+0000","last_fullsized":"2024-04-25T05:45:53.976113+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:55.756352+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":452,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984741+0000","last_change":"2024-04-25T05:45:23.299259+0000","last_active":"2024-04-25T05:45:57.984741+0000","last_peered":"2024-04-25T05:45:57.984741+0000","last_clean":"2024-04-25T05:45:57.984741+0000","last_became_active":"2024-04-25T05:44:28.125552+0000","last_became_peered":"2024-04-25T05:44:28.125552+0000","last_unstale":"2024-04-25T05:45:57.984741+0000","last_undegraded":"2024-04-25T05:45:57.984741+0000","last_fullsized":"2024-04-25T05:45:57.984741+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:43.393282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":248,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986468+0000","last_change":"2024-04-25T05:44:28.169494+0000","last_active":"2024-04-25T05:45:57.986468+0000","last_peered":"2024-04-25T05:45:57.986468+0000","last_clean":"2024-04-25T05:45:57.986468+0000","last_became_active":"2024-04-25T05:44:28.168461+0000","last_became_peered":"2024-04-25T05:44:28.168461+0000","last_unstale":"2024-04-25T05:45:57.986468+0000","last_undegraded":"2024-04-25T05:45:57.986468+0000","last_fullsized":"2024-04-25T05:45:57.986468+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":857,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.990173+0000","last_change":"2024-04-25T05:44:28.225524+0000","last_active":"2024-04-25T05:45:57.990173+0000","last_peered":"2024-04-25T05:45:57.990173+0000","last_clean":"2024-04-25T05:45:57.990173+0000","last_became_active":"2024-04-25T05:44:28.182800+0000","last_became_peered":"2024-04-25T05:44:28.182800+0000","last_unstale":"2024-04-25T05:45:57.990173+0000","last_undegraded":"2024-04-25T05:45:57.990173+0000","last_fullsized":"2024-04-25T05:45:57.990173+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'39","last_scrub_stamp":"2024-04-25T05:42:57.487214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:57.487214+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899763+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":647,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.989194+0000","last_change":"2024-04-25T05:44:16.995982+0000","last_active":"2024-04-25T05:45:57.989194+0000","last_peered":"2024-04-25T05:45:57.989194+0000","last_clean":"2024-04-25T05:45:57.989194+0000","last_became_active":"2024-04-25T05:40:55.982195+0000","last_became_peered":"2024-04-25T05:40:55.982195+0000","last_unstale":"2024-04-25T05:45:57.989194+0000","last_undegraded":"2024-04-25T05:45:57.989194+0000","last_fullsized":"2024-04-25T05:45:57.989194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.900631+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":602,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.975173+0000","last_change":"2024-04-25T05:45:32.772720+0000","last_active":"2024-04-25T05:45:53.975173+0000","last_peered":"2024-04-25T05:45:53.975173+0000","last_clean":"2024-04-25T05:45:53.975173+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:45:53.975173+0000","last_undegraded":"2024-04-25T05:45:53.975173+0000","last_fullsized":"2024-04-25T05:45:53.975173+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.772481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.772481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:36.168927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":620,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.992436+0000","last_change":"2024-04-25T05:42:53.536936+0000","last_active":"2024-04-25T05:45:57.992436+0000","last_peered":"2024-04-25T05:45:57.992436+0000","last_clean":"2024-04-25T05:45:57.992436+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:45:57.992436+0000","last_undegraded":"2024-04-25T05:45:57.992436+0000","last_fullsized":"2024-04-25T05:45:57.992436+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:53.536809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:53.536809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:02.899392+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":716,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.967724+0000","last_change":"2024-04-25T05:45:24.354902+0000","last_active":"2024-04-25T05:45:53.967724+0000","last_peered":"2024-04-25T05:45:53.967724+0000","last_clean":"2024-04-25T05:45:53.967724+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:45:53.967724+0000","last_undegraded":"2024-04-25T05:45:53.967724+0000","last_fullsized":"2024-04-25T05:45:53.967724+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:24.354764+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:45:24.354764+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:42.834643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":253,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-04-25T05:45:53.976399+0000","last_change":"2024-04-25T05:45:22.646228+0000","last_active":"2024-04-25T05:45:53.976399+0000","last_peered":"2024-04-25T05:45:53.976399+0000","last_clean":"2024-04-25T05:45:53.976399+0000","last_became_active":"2024-04-25T05:44:28.181951+0000","last_became_peered":"2024-04-25T05:44:28.181951+0000","last_unstale":"2024-04-25T05:45:53.976399+0000","last_undegraded":"2024-04-25T05:45:53.976399+0000","last_fullsized":"2024-04-25T05:45:53.976399+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.646148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.646148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:45.468679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":634,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.984293+0000","last_change":"2024-04-25T05:44:05.242484+0000","last_active":"2024-04-25T05:45:57.984293+0000","last_peered":"2024-04-25T05:45:57.984293+0000","last_clean":"2024-04-25T05:45:57.984293+0000","last_became_active":"2024-04-25T05:43:16.196314+0000","last_became_peered":"2024-04-25T05:43:16.196314+0000","last_unstale":"2024-04-25T05:45:57.984293+0000","last_undegraded":"2024-04-25T05:45:57.984293+0000","last_fullsized":"2024-04-25T05:45:57.984293+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":143,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-04-25T05:45:57.986364+0000","last_change":"2024-04-25T05:45:39.734494+0000","last_active":"2024-04-25T05:45:57.986364+0000","last_peered":"2024-04-25T05:45:57.986364+0000","last_clean":"2024-04-25T05:45:57.986364+0000","last_became_active":"2024-04-25T05:43:16.198077+0000","last_became_peered":"2024-04-25T05:43:16.198077+0000","last_unstale":"2024-04-25T05:45:57.986364+0000","last_undegraded":"2024-04-25T05:45:57.986364+0000","last_fullsized":"2024-04-25T05:45:57.986364+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:39.734373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.734373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:03.595206+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1706754072,"num_objects":26044,"num_object_clones":0,"num_object_copies":78132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1589,"num_objects_unfound":0,"num_objects_dirty":26044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216580,"num_write_kb":7763905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33687,"num_bytes_recovered":2207645696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6608662528,"data_stored":6608650312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50424,"ondisk_log_size":50424,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":701,"num_read_kb":696,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1397,"num_read_kb":1122,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":8366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":14104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":957,"ondisk_log_size":957,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070613,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934308,"kb_used_data":1321436,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91810844,"statfs":{"total":95995035648,"available":94014304256,"internally_reserved":0,"allocated":1353150464,"data_stored":1352548371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":627571146},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364953,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1018328,"kb_used_data":535952,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92726824,"statfs":{"total":95995035648,"available":94952267776,"internally_reserved":0,"allocated":548814848,"data_stored":548224777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033944,"kb_used_data":1334988,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91711208,"statfs":{"total":95995035648,"available":93912276992,"internally_reserved":0,"allocated":1367027712,"data_stored":1366382834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738552,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2095224,"kb_used_data":1293488,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":91649928,"statfs":{"total":95995035648,"available":93849526272,"internally_reserved":0,"allocated":1324531712,"data_stored":1323909237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":820902343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975980,"kb_used_data":551908,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92769172,"statfs":{"total":95995035648,"available":94995632128,"internally_reserved":0,"allocated":565153792,"data_stored":564543492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":171798691998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1586429952,"data_stored":1586429952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1355223040,"data_stored":1355218968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1261371392,"data_stored":1261371392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":556077056,"data_stored":556072984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":541523968,"data_stored":541523968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1308037120,"data_stored":1308033048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:46:07.290 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-25T05:46:07.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_2 pg_num 26 2024-04-25T05:46:07.369 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:07.391 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:07.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:07 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:46:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:07 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:46:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:07 smithi060 ceph-mon[30898]: pgmap v637: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.4 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:07 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:46:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:07 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:46:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:07.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.643 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:07.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.769 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:07.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:07.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:46:07.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:07.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:07.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:07.912 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:07.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:08.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:08.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:08.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[27010]: osdmap e366: 8 total, 8 up, 5 in 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[37492]: osdmap e366: 8 total, 8 up, 5 in 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:46:08.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:08 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:46:08.152 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:46:08.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:08.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:08.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:08.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:08.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:08 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:46:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:08 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:46:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:08 smithi060 ceph-mon[30898]: osdmap e366: 8 total, 8 up, 5 in 2024-04-25T05:46:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:08 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:46:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:08 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:46:08.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:08.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:46:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:08.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:08.606 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:08.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:08.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:08.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:08.912 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:08.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:09.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:09.035 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:46:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 9 pg_num to 26 2024-04-25T05:46:09.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:09.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:09.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:09.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:09.388 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:09.389 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: pgmap v639: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: osdmap e367: 8 total, 8 up, 5 in 2024-04-25T05:46:09.389 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: pgmap v639: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: osdmap e367: 8 total, 8 up, 5 in 2024-04-25T05:46:09.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:46:09.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:09.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: pgmap v639: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.2 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.18k op/s; 580/57528 objects misplaced (1.008%); 7.0 MiB/s, 112 objects/s recovering 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: osdmap e367: 8 total, 8 up, 5 in 2024-04-25T05:46:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:46:09.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:09.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:09.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:09.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:09.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:09.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:09.920 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:09.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:10.015 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:10.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.143 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:10.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:10.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:10.266 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[37492]: osdmap e368: 8 total, 8 up, 5 in 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[37492]: osdmap e369: 8 total, 8 up, 5 in 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[27010]: osdmap e368: 8 total, 8 up, 5 in 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:10.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:46:10.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:10 smithi043 ceph-mon[27010]: osdmap e369: 8 total, 8 up, 5 in 2024-04-25T05:46:10.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:46:10.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:10.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:10.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:10.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3404605953' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:46:10.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:10 smithi060 ceph-mon[30898]: osdmap e368: 8 total, 8 up, 5 in 2024-04-25T05:46:10.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:10.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:10 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:46:10.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:10 smithi060 ceph-mon[30898]: osdmap e369: 8 total, 8 up, 5 in 2024-04-25T05:46:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.514 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:10.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:10.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:10.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:10.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.759 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:10.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:10.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:10.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:10.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:10.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:11.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:11.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.138 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:11.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:11.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:11.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: pgmap v642: 158 pgs: 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s; 6.3 MiB/s, 101 objects/s recovering 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:46:11.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: pgmap v642: 158 pgs: 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s; 6.3 MiB/s, 101 objects/s recovering 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:46:11.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:11 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:46:11.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:11.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: pgmap v642: 158 pgs: 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.25k op/s; 6.3 MiB/s, 101 objects/s recovering 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:46:11.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:11 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:46:11.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:11.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:11.592 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:11.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:11.690 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:11.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:11.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:11.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:11.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:11.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:12.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:12.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:12.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.231 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:12.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:12.278 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:12.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:12.350 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:12.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: 4.1a scrub starts 2024-04-25T05:46:12.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: 4.1a scrub ok 2024-04-25T05:46:12.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: osdmap e370: 8 total, 8 up, 5 in 2024-04-25T05:46:12.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: 4.1a scrub starts 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: 4.1a scrub ok 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: osdmap e370: 8 total, 8 up, 5 in 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:46:12.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:46:12.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.362 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:12.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:12.393 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:46:12.395 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:12.427 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:12.447 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:12.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:12.468 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:12.484 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:12.489 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: 4.1a scrub starts 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: 4.1a scrub ok 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: osdmap e370: 8 total, 8 up, 5 in 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:46:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:46:12.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:46:12.510 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:12.529 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:12.530 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:12.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:12.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:12.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:12.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:12.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.855 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:12.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:12.956 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:12.978 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:12.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:13.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:13.080 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:46:13.080 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:46:13.080 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:46:13.081 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:46:13.081 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:46:13.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:13.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:13.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.233 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:13.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:13.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:13.356 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.3 scrub starts 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.4 scrub starts 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.3 scrub ok 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.4 scrub ok 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.8 scrub starts 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: pgmap v645: 168 pgs: 10 unknown, 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7.9 MiB/s, 126 objects/s recovering 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 9.8 scrub ok 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1713027375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1505486672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:13.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: osdmap e371: 8 total, 8 up, 5 in 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:46:13.385 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:46:13.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.3 scrub starts 2024-04-25T05:46:13.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.4 scrub starts 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.3 scrub ok 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.4 scrub ok 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.8 scrub starts 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: pgmap v645: 168 pgs: 10 unknown, 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7.9 MiB/s, 126 objects/s recovering 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 9.8 scrub ok 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1713027375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1505486672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: osdmap e371: 8 total, 8 up, 5 in 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:46:13.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.3 scrub starts 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.4 scrub starts 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.3 scrub ok 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.4 scrub ok 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.8 scrub starts 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: pgmap v645: 168 pgs: 10 unknown, 158 active+clean; 798 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7.9 MiB/s, 126 objects/s recovering 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 9.8 scrub ok 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1713027375' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:13.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1505486672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: osdmap e371: 8 total, 8 up, 5 in 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:46:13.396 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:13 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:46:13.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:13.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:13.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:13.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:13.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:13.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:13.831 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:13.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:13.975 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:13.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:14.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:14.102 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.9 scrub starts 2024-04-25T05:46:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.9 scrub ok 2024-04-25T05:46:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.0 scrub starts 2024-04-25T05:46:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.0 scrub ok 2024-04-25T05:46:14.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.2 scrub starts 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 9.2 scrub ok 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: osdmap e372: 8 total, 8 up, 5 in 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.9 scrub starts 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.9 scrub ok 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.0 scrub starts 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.0 scrub ok 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.2 scrub starts 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 9.2 scrub ok 2024-04-25T05:46:14.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:14.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: osdmap e372: 8 total, 8 up, 5 in 2024-04-25T05:46:14.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:46:14.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:14 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:46:14.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:14.228 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:14.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:14.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:14.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:14.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.9 scrub starts 2024-04-25T05:46:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.9 scrub ok 2024-04-25T05:46:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.0 scrub starts 2024-04-25T05:46:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.0 scrub ok 2024-04-25T05:46:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.2 scrub starts 2024-04-25T05:46:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 9.2 scrub ok 2024-04-25T05:46:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: osdmap e372: 8 total, 8 up, 5 in 2024-04-25T05:46:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:46:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:14 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:46:14.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:14.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:14.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:14.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:14.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:14.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:14.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:14.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:15.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:15.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:15.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.7 scrub starts 2024-04-25T05:46:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.7 scrub ok 2024-04-25T05:46:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.6 scrub starts 2024-04-25T05:46:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.6 scrub ok 2024-04-25T05:46:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.1 scrub starts 2024-04-25T05:46:15.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: 9.1 scrub ok 2024-04-25T05:46:15.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: pgmap v648: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:46:15.143 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[27010]: osdmap e373: 8 total, 8 up, 5 in 2024-04-25T05:46:15.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.7 scrub starts 2024-04-25T05:46:15.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.7 scrub ok 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.6 scrub starts 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.6 scrub ok 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.1 scrub starts 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: 9.1 scrub ok 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: pgmap v648: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:46:15.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:15 smithi043 ceph-mon[37492]: osdmap e373: 8 total, 8 up, 5 in 2024-04-25T05:46:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:15.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:46:15.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:15.306 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.7 scrub starts 2024-04-25T05:46:15.306 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.7 scrub ok 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.6 scrub starts 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.6 scrub ok 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.1 scrub starts 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: 9.1 scrub ok 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: pgmap v648: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:46:15.307 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:15 smithi060 ceph-mon[30898]: osdmap e373: 8 total, 8 up, 5 in 2024-04-25T05:46:15.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:15.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:15.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:15.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:15.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:15.692 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:15.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:15.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:15.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:46:15.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:15.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:16.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:16.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 370'14458 2024-04-25T05:46:16.082 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14545 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 370'14458 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14545 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14477 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.19 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14613 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14457 2024-04-25T05:46:16.083 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 9.17 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14284 2024-04-25T05:46:16.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:16.143 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6] out_osds: [2, 5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:46:16.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:46:16.143 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-25T05:46:16.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd in 2 2024-04-25T05:46:16.164 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:16.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:16.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:16.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:46:16.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14477 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.19 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14613 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14457 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 9.17 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14284 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:46:16.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:16 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:174867 7.e 7:79e6cc5c:::benchmark_data_smithi043_74356_object57998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:03.066924+0000", 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.280689494, 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13658604399999999, 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.351 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 174867 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066924+0000", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066924+0000", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066926+0000", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3090000000000001e-06 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066932+0000", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1530000000000002e-06 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066945+0000", 2024-04-25T05:46:16.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3259e-05 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.066957+0000", 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1484999999999999e-05 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.067014+0000", 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7228999999999997e-05 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.067132+0000", 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000118548 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.067253+0000", 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121131 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.070168+0000", 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002914916 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.070211+0000", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2341e-05 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.203295+0000", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13308408399999999 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.203460+0000", 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000165021 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:03.203510+0000", 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0568000000000001e-05 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196426 7.e 7:7c1a9912:::benchmark_data_smithi043_74356_object79557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.897007+0000", 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 464.45060716299997, 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.179707375, 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.357 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196426 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897007+0000", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897007+0000", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897008+0000", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4279999999999999e-06 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897013+0000", 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3519999999999997e-06 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897025+0000", 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1182000000000001e-05 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897033+0000", 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3080000000000008e-06 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897150+0000", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117421 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897235+0000", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4318000000000001e-05 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.897359+0000", 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012426399999999999 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907256+0000", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098967010000000008 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907308+0000", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2605999999999997e-05 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907453+0000", 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000144843 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076514+0000", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16906104599999999 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076575+0000", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0554999999999999e-05 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.076714+0000", 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013935099999999999 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:196429 7.e 7:7fcfc889:::benchmark_data_smithi043_74356_object79560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:31.898441+0000", 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 464.44917313100001, 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.187306206, 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.364 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196429 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898441+0000", 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898441+0000", 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898442+0000", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.92e-07 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898446+0000", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.374e-06 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898456+0000", 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0239e-05 2024-04-25T05:46:16.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.898464+0000", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6890000000000004e-06 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902312+0000", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038482960000000002 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902443+0000", 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013121699999999999 2024-04-25T05:46:16.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.902585+0000", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014143399999999999 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907433+0000", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0048484950000000004 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907529+0000", 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5874999999999999e-05 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:31.907546+0000", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7091999999999998e-05 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.085306+0000", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17775971700000001 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.085382+0000", 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6472999999999998e-05 2024-04-25T05:46:16.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:32.085747+0000", 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036451300000000001 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198444 7.e 7:78abec1d:::benchmark_data_smithi043_74356_object81575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.794077+0000", 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 460.55353703100002, 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99911008099999998, 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198444 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794077+0000", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794077+0000", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794078+0000", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5409999999999999e-06 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794085+0000", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9879999999999998e-06 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794100+0000", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4712000000000001e-05 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794109+0000", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4050000000000006e-06 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794180+0000", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0209999999999994e-05 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:46:16.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.794190+0000", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9850000000000001e-06 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.780963+0000", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98677301299999998 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.781104+0000", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014153500000000001 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:16.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.781231+0000", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012691199999999999 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.787386+0000", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0061549869999999998 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.789874+0000", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024879559999999999 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793016+0000", 2024-04-25T05:46:16.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031423530000000001 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793124+0000", 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107588 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793187+0000", 2024-04-25T05:46:16.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2896000000000002e-05 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198460 7.e 7:709cce6b:::benchmark_data_smithi043_74356_object81591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.804688+0000", 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 460.542925529, 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98900476400000004, 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.377 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.378 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.378 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.378 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.378 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198460 2024-04-25T05:46:16.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804688+0000", 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804688+0000", 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.379 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804690+0000", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9599999999999999e-06 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804696+0000", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1789999999999996e-06 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804709+0000", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2148e-05 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804717+0000", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0250000000000002e-06 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804777+0000", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0288000000000003e-05 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.804785+0000", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2900000000000002e-06 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.784889+0000", 2024-04-25T05:46:16.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98010375400000005 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.785019+0000", 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013029 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.785148+0000", 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129114 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.382 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.787425+0000", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022764489999999998 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.792909+0000", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054843909999999999 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793371+0000", 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000462044 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.383 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793642+0000", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027039399999999997 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793693+0000", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1437999999999998e-05 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198463 7.e 7:7283c994:::benchmark_data_smithi043_74356_object81594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.805618+0000", 2024-04-25T05:46:16.384 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 460.54199608099998, 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98817368400000005, 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198463 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805618+0000", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805618+0000", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805619+0000", 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6099999999999999e-07 2024-04-25T05:46:16.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805624+0000", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9339999999999997e-06 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805633+0000", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3139999999999993e-06 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805641+0000", 2024-04-25T05:46:16.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7989999999999994e-06 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805696+0000", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4830000000000002e-05 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.805704+0000", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1410000000000005e-06 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.787453+0000", 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98174883700000004 2024-04-25T05:46:16.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.787574+0000", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121013 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.787678+0000", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010478400000000001 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.789927+0000", 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022483429999999999 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.792954+0000", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030277110000000002 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793702+0000", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00074747799999999999 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793752+0000", 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9781000000000003e-05 2024-04-25T05:46:16.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793791+0000", 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9758000000000001e-05 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:198478 7.e 7:7fba3efe:::benchmark_data_smithi043_74356_object81609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:38:35.823107+0000", 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 460.52450726699999, 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97447132000000003, 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 198478 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823107+0000", 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823107+0000", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823108+0000", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.314e-06 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823112+0000", 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3819999999999997e-06 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823124+0000", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1772e-05 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823131+0000", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6880000000000002e-06 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823184+0000", 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3202999999999999e-05 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:35.823192+0000", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6159999999999996e-06 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.789970+0000", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.966778523 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.790114+0000", 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014368900000000001 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.790240+0000", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000126625 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.792994+0000", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027537799999999999 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.793798+0000", 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080362000000000001 2024-04-25T05:46:16.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.797452+0000", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036543209999999999 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.797521+0000", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8399999999999996e-05 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:38:36.797578+0000", 2024-04-25T05:46:16.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7386999999999998e-05 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:123963 8.e 8:7c4c26a7:::benchmark_data_smithi043_104001_object123962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:54.946699+0000", 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 321.40091533399999, 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0544992120000001, 2024-04-25T05:46:16.398 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 123963 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946699+0000", 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946699+0000", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946700+0000", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6160000000000001e-06 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946727+0000", 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6619e-05 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946744+0000", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7618e-05 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946754+0000", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9669999999999996e-06 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946825+0000", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0970000000000007e-05 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:46:16.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:54.946837+0000", 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1342000000000001e-05 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.985010+0000", 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0381736539999999 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.985160+0000", 2024-04-25T05:46:16.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014998100000000001 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4,7", 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.985338+0000", 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000177553 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.998630+0000", 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013292625000000001 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.403 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.999150+0000", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00051931899999999997 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:55.999362+0000", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00021205700000000001 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:56.000787+0000", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001425335 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:56.001031+0000", 2024-04-25T05:46:16.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024400800000000001 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:56.001198+0000", 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016654800000000001 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(8.6 243/233 [PushReplyOp(8:64e6b5ba:::benchmark_data_smithi043_104001_object21505:head)])", 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:40:58.254594+0000", 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 318.09302014299999, 2024-04-25T05:46:16.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.172476457, 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254594+0000", 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254594+0000", 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254594+0000", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9700000000000002e-07 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254597+0000", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3759999999999999e-06 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254605+0000", 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3860000000000001e-06 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.254613+0000", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6629999999999993e-06 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.426762+0000", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17214921999999999 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:40:58.427070+0000", 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030841499999999998 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(8.6 244/233 [PushReplyOp(8:659be34b:::benchmark_data_smithi043_104001_object57702:head)])", 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:01.027481+0000", 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 315.32013298200002, 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13537011700000001, 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027481+0000", 2024-04-25T05:46:16.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027481+0000", 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027481+0000", 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8000000000000002e-07 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027485+0000", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5219999999999999e-06 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027492+0000", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1470000000000004e-06 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.027499+0000", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8009999999999998e-06 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.162614+0000", 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13511519699999999 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:01.162851+0000", 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023717000000000001 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:127829 8.e 8:7366e375:::benchmark_data_smithi043_104001_object3705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.821080+0000", 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.52653424099998, 2024-04-25T05:46:16.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15293510099999999, 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 127829 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821080+0000", 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821080+0000", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821081+0000", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4470000000000001e-06 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821085+0000", 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2560000000000004e-06 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821095+0000", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4639999999999995e-06 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821102+0000", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0720000000000002e-06 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821203+0000", 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100865 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821320+0000", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011730099999999999 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4,7", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.821449+0000", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012899600000000001 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828104+0000", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066550630000000001 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829006+0000", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00090220600000000001 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829080+0000", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.339e-05 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973828+0000", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14474809 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973902+0000", 2024-04-25T05:46:16.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4362000000000004e-05 2024-04-25T05:46:16.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:46:16.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.974015+0000", 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112589 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:127832 8.1 e245/241 8:8c291c95:::benchmark_data_smithi043_104001_object3708:head v 245'8143, mlcod=245'8141)", 2024-04-25T05:46:16.420 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.828403+0000", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.51921068500002, 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14527123, 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828403+0000", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828403+0000", 2024-04-25T05:46:16.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828404+0000", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3799999999999996e-07 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828411+0000", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0960000000000003e-06 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828420+0000", 2024-04-25T05:46:16.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2590000000000006e-06 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828426+0000", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3589999999999999e-06 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828480+0000", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4185999999999999e-05 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.828503+0000", 2024-04-25T05:46:16.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3549000000000001e-05 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973623+0000", 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.145119251 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973674+0000", 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1792000000000002e-05 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:127833 8.d e245/241 8:b40552da:::benchmark_data_smithi043_104001_object3709:head v 245'8049, mlcod=245'8046)", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.829543+0000", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.51807058899999, 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15249069600000001, 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829543+0000", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829543+0000", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829544+0000", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4000000000000002e-07 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829547+0000", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2540000000000001e-06 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829552+0000", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.181e-06 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829556+0000", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1670000000000001e-06 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829595+0000", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8216999999999997e-05 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.829618+0000", 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3362000000000001e-05 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.981990+0000", 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.15237231000000001 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.982034+0000", 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3764999999999999e-05 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:127837 8.1 e245/241 8:8d4e407e:::benchmark_data_smithi043_104001_object3713:head v 245'8144, mlcod=245'8141)", 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.830440+0000", 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.51717403499998, 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143271027, 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830440+0000", 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830440+0000", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830441+0000", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2099999999999996e-07 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830447+0000", 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8159999999999996e-06 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830456+0000", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6810000000000004e-06 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830463+0000", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0330000000000003e-06 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830643+0000", 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017955900000000001 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.830665+0000", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2104e-05 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973681+0000", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14301608399999999 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973711+0000", 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0029e-05 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(8.6 245/233 [PushReplyOp(8:660ecd01:::benchmark_data_smithi043_104001_object113081:head)])", 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.831033+0000", 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.51658068, 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14344849400000001, 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831033+0000", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831033+0000", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831033+0000", 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.41e-07 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831036+0000", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0750000000000002e-06 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831043+0000", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5039999999999999e-06 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831051+0000", 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5739999999999998e-06 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831096+0000", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5012999999999997e-05 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.974482+0000", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.143386087 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:127834 8.d e245/241 8:b2f67481:::benchmark_data_smithi043_104001_object3710:head v 245'8050, mlcod=245'8046)", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.831580+0000", 2024-04-25T05:46:16.436 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.516034139, 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.156366334, 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831580+0000", 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831580+0000", 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831580+0000", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0900000000000001e-07 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831585+0000", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9640000000000002e-06 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973303+0000", 2024-04-25T05:46:16.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14171729999999999 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973321+0000", 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8553e-05 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973682+0000", 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000361314 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973709+0000", 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6089e-05 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.987907+0000", 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014198936000000001 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.987946+0000", 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8569000000000003e-05 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop_reply(client.25115.0:127829 8.e e245/240 ondisk, result = 0)", 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:02.831891+0000", 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 313.51572283299998, 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14212941400000001, 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831891+0000", 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831891+0000", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831892+0000", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7400000000000003e-07 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.831896+0000", 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4050000000000002e-06 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973563+0000", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14166645 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973575+0000", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1969e-05 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973811+0000", 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023697599999999999 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.973821+0000", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0969999999999995e-06 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:02.974020+0000", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000199843 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:128369 8.e 8:73189daf:::benchmark_data_smithi043_104001_object4245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:03.533347+0000", 2024-04-25T05:46:16.444 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 312.81426730999999, 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14888737899999999, 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 128369 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533347+0000", 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533347+0000", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533348+0000", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533354+0000", 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8320000000000002e-06 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533366+0000", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1979e-05 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.533374+0000", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2700000000000004e-06 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.547972+0000", 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.014597933 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.548099+0000", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012693000000000001 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,4,7", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.548220+0000", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012168299999999999 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.681530+0000", 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.13330930499999999 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.681646+0000", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116259 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.681963+0000", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00031691 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.682003+0000", 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0417000000000003e-05 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.682097+0000", 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3949999999999993e-05 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.682234+0000", 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013663600000000001 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.25115.0:128384 8.4 e245/241 8:24dba3fe:::benchmark_data_smithi043_104001_object4260:head v 245'8148, mlcod=245'8146)", 2024-04-25T05:46:16.450 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:03.552467+0000", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 312.79514690500002, 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.140191645, 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.552467+0000", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.552467+0000", 2024-04-25T05:46:16.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.552468+0000", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6199999999999997e-07 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.552473+0000", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8880000000000002e-06 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.678969+0000", 2024-04-25T05:46:16.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12649527199999999 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.678982+0000", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3451e-05 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.679054+0000", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1329999999999996e-05 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.679083+0000", 2024-04-25T05:46:16.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9028000000000001e-05 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.692614+0000", 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013531911000000001 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:03.692659+0000", 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4103000000000001e-05 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(8.6 253/233 [PushReplyOp(8:6abae469:::benchmark_data_smithi043_104001_object52790:head)])", 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:41:22.191769+0000", 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 294.15584454499998, 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14323160500000001, 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191769+0000", 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191769+0000", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191769+0000", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8900000000000001e-07 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191773+0000", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2279999999999998e-06 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:46:16.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191778+0000", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7030000000000003e-06 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.191784+0000", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0599999999999996e-06 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.334521+0000", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14273676499999999 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:46:16.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:41:22.335001+0000", 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00047966000000000003 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:16.458 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:46:16.459 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 370'14458 2024-04-25T05:46:16.459 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14545 2024-04-25T05:46:16.459 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14477 2024-04-25T05:46:16.459 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.19 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14613 2024-04-25T05:46:16.460 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 370'14457 2024-04-25T05:46:16.460 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 9.17 starting backfill to osd.4 from (0'0,0'0] MAX to 370'14284 2024-04-25T05:46:16.460 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:46:16.460 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:16 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:46:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.461 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:16.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:16.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:16.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:16.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:16.707 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:16.761 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:16.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:16.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:16.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:16.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:16.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:17.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:17.089 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:17.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 9.19 scrub starts 2024-04-25T05:46:17.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 9.19 scrub ok 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: pgmap v650: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: osdmap e374: 8 total, 8 up, 5 in 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 6.6 scrub starts 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[27010]: 6.6 scrub ok 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 9.19 scrub starts 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 9.19 scrub ok 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: pgmap v650: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: osdmap e374: 8 total, 8 up, 5 in 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:46:17.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:46:17.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 6.6 scrub starts 2024-04-25T05:46:17.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:17 smithi043 ceph-mon[37492]: 6.6 scrub ok 2024-04-25T05:46:17.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.225 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:17.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:17.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:17.344 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:17.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:17.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:46:17.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 9.19 scrub starts 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 9.19 scrub ok 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: pgmap v650: 168 pgs: 4 remapped+peering, 1 peering, 2 unknown, 161 active+clean; 522 MiB data, 4.5 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: osdmap e374: 8 total, 8 up, 5 in 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 6.6 scrub starts 2024-04-25T05:46:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:17 smithi060 ceph-mon[30898]: 6.6 scrub ok 2024-04-25T05:46:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:17.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:17.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:17.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:17.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:17.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:17.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:17.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:18.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:18.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.191 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:18.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: osdmap e375: 8 total, 8 up, 5 in 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: 9.16 scrub starts 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: 9.16 scrub ok 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: 6.13 scrub starts 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: 6.13 scrub ok 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: osdmap e375: 8 total, 8 up, 5 in 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: 9.16 scrub starts 2024-04-25T05:46:18.284 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: 9.16 scrub ok 2024-04-25T05:46:18.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: 6.13 scrub starts 2024-04-25T05:46:18.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: 6.13 scrub ok 2024-04-25T05:46:18.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:18.285 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:18.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:18.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:18.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:18.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: osdmap e375: 8 total, 8 up, 5 in 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: 9.16 scrub starts 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: 9.16 scrub ok 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: 6.13 scrub starts 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: 6.13 scrub ok 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:18.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:18.589 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:18.655 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:18.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:18.717 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:18.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:18.763 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:18.813 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:18.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:18.853 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:18.895 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:18.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:18.896 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:18.937 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:18.980 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:18.984 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:18.985 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:18.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:19.022 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:19.107 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:18 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:46:18.807+0000 7f173ec68700 -1 mon.a@0(leader).osd e375 definitely_dead 0 2024-04-25T05:46:19.114 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.2. 2024-04-25T05:46:19.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:19.150 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:19.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:19.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:19.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: 9.10 scrub starts 2024-04-25T05:46:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: 9.10 scrub ok 2024-04-25T05:46:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: pgmap v653: 168 pgs: 3 remapped+peering, 1 peering, 2 unknown, 162 active+clean; 513 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1628661735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1008921791' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: 9.10 scrub starts 2024-04-25T05:46:19.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: 9.10 scrub ok 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: pgmap v653: 168 pgs: 3 remapped+peering, 1 peering, 2 unknown, 162 active+clean; 513 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1628661735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1008921791' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:19.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:19.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:19.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: 9.10 scrub starts 2024-04-25T05:46:19.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: 9.10 scrub ok 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: pgmap v653: 168 pgs: 3 remapped+peering, 1 peering, 2 unknown, 162 active+clean; 513 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1628661735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1008921791' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-04-25T05:46:19.524 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:19.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:19.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:19.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:19.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:19.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:19.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:19.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:19.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:20.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:20.068 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-25T05:46:20.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.154 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:20.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:20.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:20.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: 9.13 scrub starts 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: 9.13 scrub ok 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: osdmap e376: 8 total, 8 up, 6 in 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:46:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[27010]: osdmap e377: 8 total, 8 up, 6 in 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: 9.13 scrub starts 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: 9.13 scrub ok 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: osdmap e376: 8 total, 8 up, 6 in 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:46:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:20 smithi043 ceph-mon[37492]: osdmap e377: 8 total, 8 up, 6 in 2024-04-25T05:46:20.400 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: 9.13 scrub starts 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: 9.13 scrub ok 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: osdmap e376: 8 total, 8 up, 6 in 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:46:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:20 smithi060 ceph-mon[30898]: osdmap e377: 8 total, 8 up, 6 in 2024-04-25T05:46:20.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:20.522 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:20.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:20.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:20.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:20.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:20.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:20.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:20.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:21.021 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:21.050 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:21.050 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:21.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:21.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.15 scrub starts 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.15 scrub ok 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: pgmap v655: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 926 B/s rd, 0 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14521 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.14 scrub starts 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.14 scrub ok 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 4.16 scrub starts 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 4.16 scrub ok 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.0 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14666 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15040 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: osdmap e378: 8 total, 8 up, 6 in 2024-04-25T05:46:21.219 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15028 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14683 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[27010]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 375'15126 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.15 scrub starts 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.15 scrub ok 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: pgmap v655: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 926 B/s rd, 0 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14521 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.14 scrub starts 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.14 scrub ok 2024-04-25T05:46:21.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 4.16 scrub starts 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 4.16 scrub ok 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.0 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14666 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15040 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: osdmap e378: 8 total, 8 up, 6 in 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15028 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14683 2024-04-25T05:46:21.221 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:21 smithi043 ceph-mon[37492]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 375'15126 2024-04-25T05:46:21.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:21.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:21.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.15 scrub starts 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.15 scrub ok 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: pgmap v655: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 926 B/s rd, 0 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.17 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14521 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.14 scrub starts 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.14 scrub ok 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 4.16 scrub starts 2024-04-25T05:46:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 4.16 scrub ok 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.0 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14666 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.12 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15040 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: osdmap e378: 8 total, 8 up, 6 in 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.2 starting backfill to osd.2 from (0'0,0'0] MAX to 375'15028 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.6 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14683 2024-04-25T05:46:21.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:21 smithi060 ceph-mon[30898]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 375'15126 2024-04-25T05:46:21.522 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:21.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:21.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:21.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:21.831 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:21.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:21.969 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:21.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:22.033 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 121311 objects 2024-04-25T05:46:22.033 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :96.8073 2024-04-25T05:46:22.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:46:22.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:22.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:22.130 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:46:22.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:22.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:22.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.5 scrub starts 2024-04-25T05:46:22.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.5 scrub ok 2024-04-25T05:46:22.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14702 2024-04-25T05:46:22.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14685 2024-04-25T05:46:22.319 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 375'14917 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14520 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1255521570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.5 scrub starts 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.5 scrub ok 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14702 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14685 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 375'14917 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14520 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1255521570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:22 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:22.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:22.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:22.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.5 scrub starts 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.5 scrub ok 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.16 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14702 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.10 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14685 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 375'14917 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 375'14520 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1255521570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:22 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:22.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:22.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:22.732 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:22.834 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:22.856 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:22.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:22.983 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:22.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:23.084 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:23.109 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:23.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: 9.11 scrub starts 2024-04-25T05:46:23.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: 9.11 scrub ok 2024-04-25T05:46:23.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: pgmap v658: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:23.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[37492]: osdmap e379: 8 total, 8 up, 6 in 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: 9.11 scrub starts 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: 9.11 scrub ok 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: pgmap v658: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:23.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:23 smithi043 ceph-mon[27010]: osdmap e379: 8 total, 8 up, 6 in 2024-04-25T05:46:23.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:23.235 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:23.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:23.359 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: 9.11 scrub starts 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: 9.11 scrub ok 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: pgmap v658: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 164 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 172/8526 objects misplaced (2.017%) 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:23.391 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-mon[30898]: osdmap e379: 8 total, 8 up, 6 in 2024-04-25T05:46:23.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:23.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:23.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:23.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:23.730 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:23.748 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:23.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:23.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:23.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:23.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:23.981 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:23.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:24.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:24.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:24.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:24.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:24.256 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-25T05:46:24.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-25T05:46:24.274 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:24.274 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:24.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:24 smithi043 ceph-mon[37492]: osdmap e380: 8 total, 8 up, 6 in 2024-04-25T05:46:24.296 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:24.296 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:24 smithi043 ceph-mon[27010]: osdmap e380: 8 total, 8 up, 6 in 2024-04-25T05:46:24.296 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:24.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:24.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:24.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:24.482 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:24 smithi060 ceph-mon[30898]: osdmap e380: 8 total, 8 up, 6 in 2024-04-25T05:46:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:24.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:24.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:24.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:24.733 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:24.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.861 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:24.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:24.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:46:24.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:24.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:24.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:24.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:46:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:25.068 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:46:25.068 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:46:25.069 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-25T05:46:25.069 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T05:46:25.069 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T05:46:25.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-25T05:46:25.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:25.137 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:25.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:25.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[37492]: pgmap v661: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[37492]: osdmap e381: 8 total, 8 up, 6 in 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[27010]: pgmap v661: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:25.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:25 smithi043 ceph-mon[27010]: osdmap e381: 8 total, 8 up, 6 in 2024-04-25T05:46:25.335 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:25.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:25.384 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:25.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:25 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:25.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:25 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:25.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:25 smithi060 ceph-mon[30898]: pgmap v661: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:25.457 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:25 smithi060 ceph-mon[30898]: osdmap e381: 8 total, 8 up, 6 in 2024-04-25T05:46:25.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:25.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:25.563 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:25.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:25.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:25.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:25.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:46:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.759 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:25.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:25.852 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T05:46:25.852 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:25.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:25.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:25.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:25.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:26.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:26.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:26.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_2' removed 2024-04-25T05:46:26.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:26.250 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:26.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:26.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:46:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:46:26.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:26.490 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:46:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:26.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:26.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:26.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:26.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:26.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:26.878 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:26.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:26.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.000 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:27.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:27.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:27.124 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:27.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:27.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:27.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[27010]: pgmap v663: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:46:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[27010]: osdmap e382: 8 total, 8 up, 6 in 2024-04-25T05:46:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[37492]: pgmap v663: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:46:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:27 smithi043 ceph-mon[37492]: osdmap e382: 8 total, 8 up, 6 in 2024-04-25T05:46:27.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:27.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:27 smithi060 ceph-mon[30898]: pgmap v663: 168 pgs: 1 activating, 6 peering, 6 active+remapped+backfill_toofull, 155 active+clean; 1.8 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 20/750 objects misplaced (2.667%) 2024-04-25T05:46:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/872213298' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:46:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:27 smithi060 ceph-mon[30898]: osdmap e382: 8 total, 8 up, 6 in 2024-04-25T05:46:27.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:27.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:27.724 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:27.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:27.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:27.872 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:27.974 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:27.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:27.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:28.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.127 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:28.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:28.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:28.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:28.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:28.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:28.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:28.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:28 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-04-25T05:46:28.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:28.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:28.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:28 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-04-25T05:46:28.611 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:28 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull) 2024-04-25T05:46:28.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:28.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:28.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:28.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:28.884 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:28.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:28.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:29.011 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:29.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:29.138 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:29.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:29.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:29.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.393 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:29.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:29.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:29 smithi060 ceph-mon[30898]: pgmap v665: 142 pgs: 1 activating, 2 peering, 139 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-04-25T05:46:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:29 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:29 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:29.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:29.619 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[27010]: pgmap v665: 142 pgs: 1 activating, 2 peering, 139 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-04-25T05:46:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[37492]: pgmap v665: 142 pgs: 1 activating, 2 peering, 139 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-04-25T05:46:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:29.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.643 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:29.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:29.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:29.894 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:29.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:29.992 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:29.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:30.018 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:30.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:30.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:30.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:30.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:30 smithi043 ceph-mon[27010]: osdmap e383: 8 total, 8 up, 6 in 2024-04-25T05:46:30.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:30 smithi043 ceph-mon[37492]: osdmap e383: 8 total, 8 up, 6 in 2024-04-25T05:46:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:30.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:30 smithi060 ceph-mon[30898]: osdmap e383: 8 total, 8 up, 6 in 2024-04-25T05:46:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:30.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:30.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:30.641 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:30.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:30.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:30.871 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:30.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:30.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:30.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:31.024 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:31.099 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:31.099 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:31.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:31.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:31.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:31.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:31.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:31 smithi043 ceph-mon[27010]: pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:46:31.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:31 smithi043 ceph-mon[37492]: pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:46:31.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:31 smithi060 ceph-mon[30898]: pgmap v667: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:46:31.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:31.529 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2] out_osds: [5, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:46:31.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:46:31.529 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-25T05:46:31.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd in 5 2024-04-25T05:46:31.534 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:31.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:31.661 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:31.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:31.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:31.917 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:31.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:32.019 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.047 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:32.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:32.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:32.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:32.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:46:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:32.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:32.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:32.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:32.426 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:32.463 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:32.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:32.548 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:32.650 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:32.671 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:32.676 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-25T05:46:32.677 INFO:tasks.radosbench:Beginning radosbench... 2024-04-25T05:46:32.677 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-25T05:46:32.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-04-25T05:46:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:32.717 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:32.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:32.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:32.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:32.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:32.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:32.956 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:32.956 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:33.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:33.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:33.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:33.270 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[37492]: pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 525 B/s rd, 0 op/s 2024-04-25T05:46:33.270 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:33.270 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:33.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[27010]: pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 525 B/s rd, 0 op/s 2024-04-25T05:46:33.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:33.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:33.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:33.285 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.298 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:33.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:33.305 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:33.326 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:33.345 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:33.366 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:33.386 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:33.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:33.406 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:33.420 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:33.427 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:33.428 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:33.429 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:33 smithi060 ceph-mon[30898]: pgmap v668: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 525 B/s rd, 0 op/s 2024-04-25T05:46:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:33.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:33.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.545 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:33.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:33.645 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.670 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:33.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:33.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:46:33.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:33.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:33.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:33.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:33.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:33.971 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:34.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.043 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:34.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:34.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:34.167 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.5. 2024-04-25T05:46:34.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:34.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:34.350 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2984526003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:34.350 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:34 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:34.350 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-25T05:46:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2984526003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:34.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-25T05:46:34.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:46:34.196+0000 7f173ec68700 -1 mon.a@0(leader).osd e383 definitely_dead 0 2024-04-25T05:46:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2984526003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-25T05:46:34.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:34.413 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:34.479 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:34.480 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:46:34.480 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:46:34.480 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:46:34.480 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:34.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:34.536 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:34.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:34.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:34.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:34.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:34.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:34.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:34.954 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-25T05:46:35.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:35.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:35.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:35.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:46:35.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:35.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:35.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:35.493 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: osdmap e384: 8 total, 8 up, 7 in 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: osdmap e385: 8 total, 8 up, 7 in 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: osdmap e384: 8 total, 8 up, 7 in 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: osdmap e385: 8 total, 8 up, 7 in 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: pgmap v669: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4113356716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: osdmap e384: 8 total, 8 up, 7 in 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: osdmap e385: 8 total, 8 up, 7 in 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.514 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-04-25T05:46:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:35.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:35.750 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:35.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:35.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:35.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:35.961 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:46:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:129506 7.1 7:85e13d17:::benchmark_data_smithi043_74356_object12637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 573.27766185200005, 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.149986696, 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129506 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682201+0000", 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682202+0000", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.017e-06 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682208+0000", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8930000000000002e-06 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682218+0000", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0373e-05 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682226+0000", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2579999999999995e-06 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682278+0000", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2231999999999997e-05 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682392+0000", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011332000000000001 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:46:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.682493+0000", 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010118 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.684382+0000", 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018894020000000001 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.684645+0000", 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026231699999999998 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.741914+0000", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.057269792 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832066+0000", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.090151063000000004 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832106+0000", 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0522000000000003e-05 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:37:02.832187+0000", 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1327000000000002e-05 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:174866 7.1 7:88ee4521:::benchmark_data_smithi043_74356_object57997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 512.89356874600003, 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12223832499999999, 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 174866 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066294+0000", 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066295+0000", 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3e-06 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066302+0000", 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2300000000000002e-06 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066314+0000", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1701e-05 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066323+0000", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.524e-06 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066368+0000", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5392999999999997e-05 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066449+0000", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0623000000000004e-05 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.066576+0000", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012777700000000001 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.068628+0000", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020514819999999999 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.069247+0000", 2024-04-25T05:46:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000619153 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188455+0000", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.119208436 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188498+0000", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2179999999999999e-05 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:03.188532+0000", 2024-04-25T05:46:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4526000000000001e-05 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196432 7.4 7:201f4639:::benchmark_data_smithi043_74356_object79563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 484.05832023300002, 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.18528801, 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:46:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196432 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901544+0000", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3030000000000001e-06 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901549+0000", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901564+0000", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5019e-05 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901572+0000", 2024-04-25T05:46:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2009999999999999e-06 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901623+0000", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1400000000000003e-05 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901753+0000", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012954700000000001 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901869+0000", 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011555199999999999 2024-04-25T05:46:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904486+0000", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002616967 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904523+0000", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7156000000000001e-05 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086719+0000", 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.182196424 2024-04-25T05:46:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086782+0000", 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2833e-05 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086830+0000", 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8378000000000003e-05 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196436 7.1 7:86601c62:::benchmark_data_smithi043_74356_object79567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:46:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 484.057821945, 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.177590574, 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196436 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8000000000000005e-07 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902048+0000", 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9419999999999998e-06 2024-04-25T05:46:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902058+0000", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.008e-05 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902066+0000", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2790000000000001e-06 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902249+0000", 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018350800000000001 2024-04-25T05:46:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902354+0000", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105227 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902460+0000", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105164 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904687+0000", 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022273990000000001 2024-04-25T05:46:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907211+0000", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002523941 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079433+0000", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17222240799999999 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079565+0000", 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013155999999999999 2024-04-25T05:46:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079631+0000", 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6285999999999995e-05 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124112 8.2 8:4cdac3fe:::benchmark_data_smithi043_104001_object124111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 338.84796330199998, 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16378072499999999, 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124112 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:46:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111905+0000", 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1059999999999996e-06 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111939+0000", 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3980999999999999e-05 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111951+0000", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1635e-05 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111959+0000", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1689999999999992e-06 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112075+0000", 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011596500000000001 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112203+0000", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012803900000000001 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112363+0000", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000160039 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116878+0000", 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045148760000000001 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116924+0000", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5593999999999997e-05 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.119845+0000", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002921589 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275525+0000", 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15567968900000001 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275567+0000", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1882999999999999e-05 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275680+0000", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011315999999999999 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125664 8.2 8:4acd28bf:::benchmark_data_smithi043_104001_object1540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.67360638600002, 2024-04-25T05:46:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11907625199999999, 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125664 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286258+0000", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.415e-06 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286264+0000", 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-04-25T05:46:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286279+0000", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5085e-05 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286287+0000", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9650000000000008e-06 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287683+0000", 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001395269 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287830+0000", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147777 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287984+0000", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015359799999999999 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291040+0000", 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030556089999999999 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291225+0000", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018544300000000001 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291399+0000", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017389099999999999 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405138+0000", 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11373876199999999 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405204+0000", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6484999999999998e-05 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405332+0000", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012813700000000001 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:126959 8.2 8:4e62c023:::benchmark_data_smithi043_104001_object2835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.10816684899999, 2024-04-25T05:46:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14632261699999999, 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 126959 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851697+0000", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.237e-06 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851705+0000", 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8320000000000006e-06 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851714+0000", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.064e-06 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851720+0000", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0859999999999998e-06 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851962+0000", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024216799999999999 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852126+0000", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016425399999999999 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852306+0000", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000180082 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.854938+0000", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026320139999999998 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855570+0000", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00063133700000000004 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855614+0000", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4480000000000001e-05 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997822+0000", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.142207745 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997883+0000", 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1085000000000002e-05 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.998018+0000", 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013523300000000001 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129538 8.2 8:4c65616f:::benchmark_data_smithi043_104001_object5414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:46:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.94883912300003, 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12767905500000001, 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129538 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011025+0000", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1650000000000001e-06 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011031+0000", 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2639999999999997e-06 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011042+0000", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1385e-05 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011051+0000", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3149999999999992e-06 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.012950+0000", 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018998400000000001 2024-04-25T05:46:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013084+0000", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013384000000000001 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013219+0000", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013510700000000001 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018871+0000", 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056513270000000003 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018951+0000", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0836999999999999e-05 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022366+0000", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034148400000000001 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023969+0000", 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001602526 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.024068+0000", 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9500999999999995e-05 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138702+0000", 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.114634108 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129549 8.2 8:40741811:::benchmark_data_smithi043_104001_object5425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:46:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.94143041000001, 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13464008199999999, 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129549 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018434+0000", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.528e-06 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018439+0000", 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3040000000000004e-06 2024-04-25T05:46:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018450+0000", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0788000000000001e-05 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018458+0000", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.405e-06 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018994+0000", 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053616199999999997 2024-04-25T05:46:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019116+0000", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121228 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019243+0000", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012719599999999999 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022328+0000", 2024-04-25T05:46:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030850640000000002 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138716+0000", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11638823700000001 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138755+0000", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8992999999999997e-05 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152783+0000", 2024-04-25T05:46:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014027773 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152919+0000", 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013651599999999999 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153072+0000", 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152888 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129543 8.e e246/240 8:7455de0e:::benchmark_data_smithi043_104001_object5419:head v 246'7976, mlcod=245'7972)", 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.93981869499999, 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12636956799999999, 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.82e-07 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020049+0000", 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2490000000000003e-06 2024-04-25T05:46:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020060+0000", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1070000000000001e-05 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020065+0000", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2310000000000004e-06 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020859+0000", 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000793862 2024-04-25T05:46:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020882+0000", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3170999999999999e-05 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146330+0000", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12544777300000001 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146413+0000", 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3629999999999997e-05 2024-04-25T05:46:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:129549 8.2 e246/233 ondisk, result = 0)", 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.93672525199997, 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1156257, 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023138+0000", 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8999999999999996e-07 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023142+0000", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2039999999999999e-06 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023149+0000", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5019999999999998e-06 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023155+0000", 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0229999999999998e-06 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138742+0000", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115587254 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138750+0000", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8089999999999993e-06 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138763+0000", 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3318e-05 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129546 8.e e246/240 8:758a41b0:::benchmark_data_smithi043_104001_object5422:head v 246'7977, mlcod=245'7972)", 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.936009759, 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12274840099999999, 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:46:36.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9200000000000001e-07 2024-04-25T05:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023859+0000", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023867+0000", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0909999999999993e-06 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023874+0000", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8240000000000002e-06 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139217+0000", 2024-04-25T05:46:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115343705 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139245+0000", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7317999999999998e-05 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146417+0000", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0071721129999999999 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146601+0000", 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018442799999999999 2024-04-25T05:46:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129556 8.2 8:484428a6:::benchmark_data_smithi043_104001_object5432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 330.93180117100002, 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.125238935, 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129556 2024-04-25T05:46:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028062+0000", 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.091e-06 2024-04-25T05:46:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028067+0000", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8489999999999997e-06 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137388+0000", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.109320815 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137405+0000", 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6677999999999999e-05 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138822+0000", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001417614 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138949+0000", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012701800000000001 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139092+0000", 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014223099999999999 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.141864+0000", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027722189999999998 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143235+0000", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001370778 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143353+0000", 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118484 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153140+0000", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0097873479999999995 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153195+0000", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5000999999999997e-05 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153300+0000", 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010480900000000001 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.2 8.3a1a8b12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 236.08922415000001, 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12946094299999999, 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870640+0000", 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7379999999999999e-06 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870647+0000", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0330000000000003e-06 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870659+0000", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1748e-05 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870667+0000", 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3100000000000001e-06 2024-04-25T05:46:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000085+0000", 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12941729699999999 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000099+0000", 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4817000000000001e-05 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.2 8.516c972 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:46:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 236.08137651999999, 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122103355, 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878488+0000", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0209999999999998e-06 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878500+0000", 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1629e-05 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878519+0000", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9522999999999999e-05 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878530+0000", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0815000000000001e-05 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000576+0000", 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122046021 2024-04-25T05:46:36.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000589+0000", 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3346000000000001e-05 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.12 8:48d1585c:::benchmark_data_smithi043_104001_object102215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 236.06634540900001, 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1294363890000001, 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.043 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893519+0000", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1519999999999999e-06 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893531+0000", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1849e-05 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893551+0000", 2024-04-25T05:46:36.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9638000000000001e-05 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893564+0000", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.323e-05 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000742+0000", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10717832200000001 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000752+0000", 2024-04-25T05:46:36.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9930000000000007e-06 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006085+0000", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0053329209999999 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006212+0000", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126873 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006455+0000", 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024327100000000001 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011490+0000", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050344350000000003 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.012956+0000", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001465872 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022845+0000", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0098890200000000001 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022904+0000", 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9239e-05 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022954+0000", 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9573999999999997e-05 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.12 8:4e9368a0:::benchmark_data_smithi043_104001_object102219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 236.06627076800001, 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1317489059999999, 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4000000000000002e-07 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893595+0000", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8600000000000001e-06 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893603+0000", 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9000000000000006e-06 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893609+0000", 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2500000000000003e-06 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000774+0000", 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10716527300000001 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000780+0000", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7119999999999997e-06 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009444+0000", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086639690000001 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009577+0000", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013284399999999999 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:46:36.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009825+0000", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024752200000000002 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.013047+0000", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003222042 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020080+0000", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070331810000000003 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025198+0000", 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051180369999999998 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025295+0000", 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6797000000000002e-05 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025341+0000", 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6078999999999997e-05 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228033 8.14 8:2d5c3048:::benchmark_data_smithi043_104001_object103909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 233.97170898900001, 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0985097700000002, 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228033 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988156+0000", 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9879999999999999e-06 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988162+0000", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7870000000000004e-06 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988180+0000", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7886000000000001e-05 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988190+0000", 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9729999999999992e-06 2024-04-25T05:46:36.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988272+0000", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1977999999999993e-05 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988285+0000", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3176000000000001e-05 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980591+0000", 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99230595399999999 2024-04-25T05:46:36.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980607+0000", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5548000000000001e-05 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081723+0000", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1011164630000001 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081910+0000", 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018638699999999999 2024-04-25T05:46:36.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.082205+0000", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00029534400000000001 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085239+0000", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030338330000000001 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085283+0000", 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3695000000000001e-05 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085372+0000", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.954e-05 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086212+0000", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084000600000000002 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086435+0000", 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022244300000000001 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086501+0000", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6000000000000005e-05 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086663+0000", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162769 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228057 8.14 8:29afc790:::benchmark_data_smithi043_104001_object103933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:46:36.063 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 233.94989678499999, 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0771824429999999, 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228057 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009967+0000", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.418e-06 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009982+0000", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5211000000000001e-05 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009997+0000", 2024-04-25T05:46:36.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4542000000000001e-05 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010006+0000", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3619999999999995e-06 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010067+0000", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0965000000000003e-05 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010076+0000", 2024-04-25T05:46:36.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1249999999999999e-06 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980702+0000", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97062569499999996 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980712+0000", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0081999999999999e-05 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083607+0000", 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1028946399999999 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083747+0000", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013999600000000001 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.084002+0000", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025568299999999998 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085297+0000", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001294033 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085711+0000", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041416199999999998 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086266+0000", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055577100000000004 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086728+0000", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000461335 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086920+0000", 2024-04-25T05:46:36.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019192499999999999 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086999+0000", 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9742999999999999e-05 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.087148+0000", 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014875500000000001 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237380 9.16 9:6e797660:::benchmark_data_smithi043_129820_object116066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.791863327000002, 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94758949999999997, 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237380 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.071 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168000+0000", 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8400000000000002e-07 2024-04-25T05:46:36.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168004+0000", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7050000000000001e-06 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168011+0000", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4739999999999999e-06 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168017+0000", 2024-04-25T05:46:36.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5160000000000002e-06 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168065+0000", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8109000000000003e-05 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168072+0000", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2139999999999999e-06 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.111902+0000", 2024-04-25T05:46:36.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94382929999999998 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.112059+0000", 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000157455 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.112290+0000", 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023091 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113696+0000", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014061670000000001 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113711+0000", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5133000000000001e-05 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115483+0000", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017713340000000001 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:36.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115543+0000", 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0755999999999999e-05 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115589+0000", 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5442999999999998e-05 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:36.077 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:36.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:36.224 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:36.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:36.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:36.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:36.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:36.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[37492]: osdmap e386: 8 total, 8 up, 7 in 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-25T05:46:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:36 smithi043 ceph-mon[27010]: osdmap e386: 8 total, 8 up, 7 in 2024-04-25T05:46:36.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:36.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/400439783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:36 smithi060 ceph-mon[30898]: osdmap e386: 8 total, 8 up, 7 in 2024-04-25T05:46:36.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:36.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:36.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:36.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:46:36.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:36.875 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:36.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:36.975 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_3' created 2024-04-25T05:46:36.977 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:37.068 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:37.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:37.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:37.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:37.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:37.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[27010]: pgmap v672: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 760 B/s rd, 0 op/s 2024-04-25T05:46:37.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-25T05:46:37.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-25T05:46:37.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[27010]: osdmap e387: 8 total, 8 up, 7 in 2024-04-25T05:46:37.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[37492]: pgmap v672: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 760 B/s rd, 0 op/s 2024-04-25T05:46:37.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-25T05:46:37.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-25T05:46:37.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:37 smithi043 ceph-mon[37492]: osdmap e387: 8 total, 8 up, 7 in 2024-04-25T05:46:37.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:37.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:37.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:46:37.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:37 smithi060 ceph-mon[30898]: pgmap v672: 142 pgs: 142 active+clean; 583 KiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 760 B/s rd, 0 op/s 2024-04-25T05:46:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-25T05:46:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1283284741' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-25T05:46:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:37 smithi060 ceph-mon[30898]: osdmap e387: 8 total, 8 up, 7 in 2024-04-25T05:46:37.502 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:37.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.627 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:37.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:37.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:37.749 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:37.772 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:37.772 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:46:37.772 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:46:37.772 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:37.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-25T05:46:37.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:37.872 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:37.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:37.890 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:37.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:38.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:38.121 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:38.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.245 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:38.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:38.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:38.368 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:38.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:46:38.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:38.474 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:38.491 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:38.494 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:38.513 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:38.534 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:38.553 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:38.581 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:38.592 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:38.602 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:38.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:38.624 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:38.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:38.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:38.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:46:38.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:38.862 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:38.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:38.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:38.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:39.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:39.093 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: pgmap v675: 158 pgs: 2 creating+peering, 3 peering, 14 unknown, 139 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 64 KiB/s, 7 objects/s recovering 2024-04-25T05:46:39.093 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: osdmap e388: 8 total, 8 up, 7 in 2024-04-25T05:46:39.093 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:46:39.094 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:39.094 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:39.094 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/518205769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.113 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:39.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:39.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:39.221 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:39.221 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:39.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: pgmap v675: 158 pgs: 2 creating+peering, 3 peering, 14 unknown, 139 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 64 KiB/s, 7 objects/s recovering 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: osdmap e388: 8 total, 8 up, 7 in 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/518205769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:39.254 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: pgmap v675: 158 pgs: 2 creating+peering, 3 peering, 14 unknown, 139 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 64 KiB/s, 7 objects/s recovering 2024-04-25T05:46:39.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: osdmap e388: 8 total, 8 up, 7 in 2024-04-25T05:46:39.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:46:39.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:39.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:39.255 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/518205769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:39.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:39.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:39.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:39.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:39.608 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:39.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.733 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:39.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:39.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:39.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:39.955 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:46:39.956 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:46:39.956 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-25T05:46:39.956 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T05:46:39.956 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T05:46:39.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-04-25T05:46:39.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:39.980 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:39.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:39.989 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-25T05:46:40.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:46:40.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:40.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:40.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:40.213 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:40.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:39 smithi060 ceph-mon[30898]: osdmap e389: 8 total, 8 up, 7 in 2024-04-25T05:46:40.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:46:40.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:40.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:40.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:39 smithi043 ceph-mon[27010]: osdmap e389: 8 total, 8 up, 7 in 2024-04-25T05:46:40.360 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:46:40.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:39 smithi043 ceph-mon[37492]: osdmap e389: 8 total, 8 up, 7 in 2024-04-25T05:46:40.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:46:40.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.474 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:40.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:40.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:40.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:40.661 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:40.661 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T05:46:40.661 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:40.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:40.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:40.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:40.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:40.965 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:40.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:41.055 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[37492]: pgmap v678: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 19 objects/s recovering 2024-04-25T05:46:41.055 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:46:41.055 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[37492]: osdmap e390: 8 total, 8 up, 7 in 2024-04-25T05:46:41.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[27010]: pgmap v678: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 19 objects/s recovering 2024-04-25T05:46:41.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:46:41.056 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:40 smithi043 ceph-mon[27010]: osdmap e390: 8 total, 8 up, 7 in 2024-04-25T05:46:41.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:41.088 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:41.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:41.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:40 smithi060 ceph-mon[30898]: pgmap v678: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 145 KiB/s, 19 objects/s recovering 2024-04-25T05:46:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/208704505' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:46:41.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:40 smithi060 ceph-mon[30898]: osdmap e390: 8 total, 8 up, 7 in 2024-04-25T05:46:41.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:41.340 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:41.419 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:41.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:41.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:41.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:41.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:41.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:41.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:41.781 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-25T05:46:41.783 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-25T05:46:41.783 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-25T05:46:41.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:41.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:41.921 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-04-25T05:46:41.921 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-25T05:46:41.921 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_152558 2024-04-25T05:46:41.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:46:41.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-04-25T05:46:41.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:41.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:41.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:46:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:41.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:42.010 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:46:42.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:42.110 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:42.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:42.165 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:46:42.166 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:46:42.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:41 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:42.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:41 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:42.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:41 smithi060 ceph-mon[30898]: osdmap e391: 8 total, 8 up, 7 in 2024-04-25T05:46:42.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/378016149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.189 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:41 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.212 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:42.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:42.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:42.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:42.384 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:42.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:42.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[27010]: osdmap e391: 8 total, 8 up, 7 in 2024-04-25T05:46:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/378016149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[37492]: osdmap e391: 8 total, 8 up, 7 in 2024-04-25T05:46:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/378016149' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:41 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:46:42.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:42.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:42.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:42.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:43.110 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 3123 3107 194.134 194.188 0.00702253 0.00513596 2024-04-25T05:46:43.110 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:43.110 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: pgmap v681: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 11 objects/s recovering 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: osdmap e392: 8 total, 8 up, 7 in 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: pgmap v681: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 11 objects/s recovering 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:46:43.111 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:43.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:43.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: osdmap e392: 8 total, 8 up, 7 in 2024-04-25T05:46:43.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:46:43.112 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:42 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:46:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:42 smithi060 ceph-mon[30898]: pgmap v681: 158 pgs: 9 creating+peering, 3 peering, 146 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 11 objects/s recovering 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:42 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:42 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:43 smithi060 ceph-mon[30898]: osdmap e392: 8 total, 8 up, 7 in 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:43 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:46:43.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:43 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:46:43.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:46:43.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:43.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:43.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:43.468 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:43.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:43.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:43.705 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:43.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:43.727 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:43.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:43.762 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:43.804 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:43.844 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:43.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:43.880 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:43.898 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:43.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:43.919 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:43.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 6245 6229 194.621 195.125 0.0051237 0.00512508 2024-04-25T05:46:43.961 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:43.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:44.002 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.040 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:44.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:44.139 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:44 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:44.139 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3395565465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:44.139 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:44 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:46:44.140 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:44 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:46:44.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:44.180 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:44.282 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:44.310 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:44.310 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3395565465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:44.310 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:46:44.310 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:46:44.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:44.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3395565465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:44.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:46:44.311 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:44 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:46:44.321 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:44.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:44.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:44.562 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:44.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7365 10.a 10:550b3adc:::benchmark_data_smithi043_152558_object7364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.366208+0000", 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0054576340000000003, 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0054911680000000003, 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7365 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366208+0000", 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.634 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366208+0000", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366209+0000", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2600000000000001e-07 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366245+0000", 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5457999999999998e-05 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.635 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366255+0000", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0175e-05 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366263+0000", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2949999999999994e-06 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366523+0000", 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025977299999999998 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.636 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366650+0000", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012767 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366789+0000", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013907200000000001 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369523+0000", 2024-04-25T05:46:44.637 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027337559999999999 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369536+0000", 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.308e-05 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7367 10.7 10:e552da94:::benchmark_data_smithi043_152558_object7366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.638 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.366834+0000", 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0048316230000000002, 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049654870000000002, 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7367 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.639 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366834+0000", 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366834+0000", 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366835+0000", 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.37e-07 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.640 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366881+0000", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6255999999999999e-05 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366889+0000", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2020000000000004e-06 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366896+0000", 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1450000000000002e-06 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.641 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369584+0000", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026880799999999998 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369871+0000", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028656100000000001 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370995+0000", 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011237409999999999 2024-04-25T05:46:44.642 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7368 10.a 10:5771f9b5:::benchmark_data_smithi043_152558_object7367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.366908+0000", 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0047577299999999999, 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049716730000000002, 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7368 2024-04-25T05:46:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366908+0000", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366908+0000", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366909+0000", 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8500000000000002e-07 2024-04-25T05:46:44.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367521+0000", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061295499999999999 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367530+0000", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8680000000000005e-06 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367538+0000", 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6569999999999997e-06 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.645 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367724+0000", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018618899999999999 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367852+0000", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012767900000000001 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368058+0000", 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000206147 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369541+0000", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014832140000000001 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370813+0000", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001271933 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7369 10.a 10:5eb19acf:::benchmark_data_smithi043_152558_object7368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.367576+0000", 2024-04-25T05:46:44.647 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0040894249999999998, 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044163850000000001, 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7369 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367576+0000", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367576+0000", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367577+0000", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3600000000000003e-07 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367613+0000", 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6303000000000001e-05 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367623+0000", 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5300000000000002e-06 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367631+0000", 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8040000000000002e-06 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368807+0000", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011761009999999999 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368937+0000", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013035400000000001 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369094+0000", 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000157095 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.651 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369632+0000", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053730800000000001 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370906+0000", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0012745740000000001 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7370 10.b 10:d80ab2e8:::benchmark_data_smithi043_152558_object7369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.367646+0000", 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.004019534, 2024-04-25T05:46:44.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044535659999999999, 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7370 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367646+0000", 2024-04-25T05:46:44.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367646+0000", 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367647+0000", 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7800000000000001e-07 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367852+0000", 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000205059 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367861+0000", 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6000000000000007e-06 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367868+0000", 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3220000000000004e-06 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.367927+0000", 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8689e-05 2024-04-25T05:46:44.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368057+0000", 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000130588 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368196+0000", 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013842300000000001 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369074+0000", 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00087810000000000004 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.371022+0000", 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001947856 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.658 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.15975.0:7358 10.7 e392/387)", 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.369561+0000", 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0021044599999999998, 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002650435, 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369561+0000", 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369561+0000", 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369562+0000", 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1399999999999997e-07 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369566+0000", 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3080000000000001e-06 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369573+0000", 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3420000000000002e-06 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369580+0000", 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3679999999999998e-06 2024-04-25T05:46:44.661 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.15975.0:7359 10.7 e392/387)", 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.369599+0000", 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.002067073, 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027014439999999999, 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369599+0000", 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369599+0000", 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369599+0000", 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.91e-07 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369602+0000", 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7389999999999999e-06 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369608+0000", 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6209999999999998e-06 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369614+0000", 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5600000000000001e-06 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.15975.0:7363 10.7 e392/387)", 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.369622+0000", 2024-04-25T05:46:44.666 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.002043287, 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027272339999999998, 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369622+0000", 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369622+0000", 2024-04-25T05:46:44.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369623+0000", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9900000000000002e-07 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369626+0000", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7549999999999999e-06 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369631+0000", 2024-04-25T05:46:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1139999999999997e-06 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369635+0000", 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7759999999999997e-06 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7372 10.f 10.3e30a71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.369662+0000", 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0020040190000000001, 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027378480000000002, 2024-04-25T05:46:44.669 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7372 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369662+0000", 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369662+0000", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369663+0000", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5519999999999999e-06 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369697+0000", 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3306000000000003e-05 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369707+0000", 2024-04-25T05:46:44.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0678e-05 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369715+0000", 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5689999999999999e-06 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7374 10.0 10:002c3688:::benchmark_data_smithi043_152558_object7373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.369860+0000", 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.001805954, 2024-04-25T05:46:44.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026031119999999999, 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7374 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369860+0000", 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369860+0000", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369861+0000", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3499999999999995e-07 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370008+0000", 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014741600000000001 2024-04-25T05:46:44.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370017+0000", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1009999999999998e-06 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370025+0000", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7980000000000006e-06 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370096+0000", 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1450000000000002e-05 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370224+0000", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012749199999999999 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.370361+0000", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013736299999999999 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7358 10.7 10:e3f09a3f:::benchmark_data_smithi043_152558_object7357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.363526+0000", 2024-04-25T05:46:44.677 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0081398200000000007, 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090179149999999996, 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7358 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363526+0000", 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363526+0000", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363527+0000", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4300000000000003e-07 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363565+0000", 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8850000000000002e-05 2024-04-25T05:46:44.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363574+0000", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5969999999999992e-06 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363582+0000", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3250000000000008e-06 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.364344+0000", 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00076166999999999999 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.364460+0000", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011609500000000001 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.364597+0000", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013651900000000001 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368952+0000", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043549030000000002 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369138+0000", 2024-04-25T05:46:44.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018638500000000001 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7359 10.7 10:ebed4823:::benchmark_data_smithi043_152558_object7358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.363595+0000", 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0080709370000000002, 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0090477119999999994, 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.682 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7359 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363595+0000", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363595+0000", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363595+0000", 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6199999999999998e-07 2024-04-25T05:46:44.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363628+0000", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2438999999999999e-05 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363635+0000", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1470000000000004e-06 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.363642+0000", 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6220000000000003e-06 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365229+0000", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015873009999999999 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365349+0000", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119719 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365491+0000", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014211000000000001 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.368956+0000", 2024-04-25T05:46:44.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034657500000000001 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369182+0000", 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022509400000000001 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.15975.0:7369 10.a e392/387 ondisk, result = 0)", 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.372024+0000", 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4294967295.9996419, 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00072978099999999998, 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.686 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372024+0000", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372024+0000", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372024+0000", 2024-04-25T05:46:44.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3900000000000001e-07 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372026+0000", 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1160000000000002e-06 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372031+0000", 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8359999999999999e-06 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372036+0000", 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6650000000000002e-06 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372522+0000", 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048646699999999998 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372529+0000", 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9999999999999999e-06 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.15975.0:7367 10.7 e392/387)", 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.372270+0000", 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 4294967295.9993954, 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055974499999999995, 2024-04-25T05:46:44.690 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372270+0000", 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372270+0000", 2024-04-25T05:46:44.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372271+0000", 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9299999999999997e-07 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372279+0000", 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.303e-06 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372290+0000", 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0305e-05 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.372297+0000", 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3889999999999999e-06 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.693 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:7363 10.7 10:e8f5b549:::benchmark_data_smithi043_152558_object7362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:44.365230+0000", 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0064360540000000001, 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0076808859999999996, 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:44.694 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 7363 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365230+0000", 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365230+0000", 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365231+0000", 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1999999999999999e-06 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365261+0000", 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.021e-05 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365271+0000", 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0199e-05 2024-04-25T05:46:44.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.365279+0000", 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8390000000000007e-06 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366084+0000", 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00080516300000000002 2024-04-25T05:46:44.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366546+0000", 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00046146099999999998 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.366686+0000", 2024-04-25T05:46:44.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139994 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369197+0000", 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025113330000000001 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:44.369241+0000", 2024-04-25T05:46:44.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3510000000000002e-05 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 15 2024-04-25T05:46:44.701 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:44.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:44.737 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:44.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:44.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:44.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:44.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 8901 8885 185.076 166 0.00335189 0.00539173 2024-04-25T05:46:44.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:45.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:45.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:46:45.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:45.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: pgmap v683: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 740 op/s 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:46:45.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:45 smithi060 ceph-mon[30898]: osdmap e393: 8 total, 8 up, 7 in 2024-04-25T05:46:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:45.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: pgmap v683: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 740 op/s 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:46:45.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:46:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[37492]: osdmap e393: 8 total, 8 up, 7 in 2024-04-25T05:46:45.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: pgmap v683: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 740 op/s 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:46:45.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:45 smithi043 ceph-mon[27010]: osdmap e393: 8 total, 8 up, 7 in 2024-04-25T05:46:45.416 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:45.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:45.520 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.556 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:45.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:45.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:45.689 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:45.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:45.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:45.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:46:45.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:45.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 11504 11488 179.475 162.688 0.00341656 0.00555527 2024-04-25T05:46:45.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:45.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:46.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:46.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:46.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:46.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:46.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:46:46.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:46 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:46:46.290 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:46.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:46.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:46:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:46:46.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:46:46.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:46 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:46:46.396 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:46:46.396 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:46:46.396 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T05:46:46.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:46:46.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:46.516 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:46.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:46.656 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:46.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:46.799 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:46.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:46:46.809 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:46.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:46.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 13769 13753 171.89 141.562 0.0104118 0.00580824 2024-04-25T05:46:46.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:46.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:46.994 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:46.994 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:46:46.994 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:46:46.994 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:46:46.994 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:47.033 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:46:47.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:47.072 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:47.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 4.2 scrub starts 2024-04-25T05:46:47.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 4.2 scrub ok 2024-04-25T05:46:47.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:46:47.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:46:47.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: pgmap v685: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 861 B/s rd, 39 MiB/s wr, 617 op/s 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1157798618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 4.2 scrub starts 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 4.2 scrub ok 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: pgmap v685: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 861 B/s rd, 39 MiB/s wr, 617 op/s 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1157798618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:46:47.110 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:47 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:46:47.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:47.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 4.2 scrub starts 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 4.2 scrub ok 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: pgmap v685: 158 pgs: 158 active+clean; 230 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 861 B/s rd, 39 MiB/s wr, 617 op/s 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1157798618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:46:47.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:47 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:46:47.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:47.338 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:47.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:47.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:47.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:47.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:47.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:47.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:47.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:47.885 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:47.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:47.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 16404 16388 170.687 164.688 0.00403936 0.00585091 2024-04-25T05:46:47.986 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:47.990 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:46:48.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:48.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:48.167 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 6.1b scrub starts 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 6.1b scrub ok 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:46:48.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: osdmap e394: 8 total, 8 up, 7 in 2024-04-25T05:46:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:46:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:46:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:48.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:48.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:48.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 6.1b scrub starts 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 6.1b scrub ok 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: osdmap e394: 8 total, 8 up, 7 in 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:48.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:48.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 6.1b scrub starts 2024-04-25T05:46:48.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:46:48.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:46:48.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 6.1b scrub ok 2024-04-25T05:46:48.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: osdmap e394: 8 total, 8 up, 7 in 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:46:48.278 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:46:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:48.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:48.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:48.424 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:48.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:48.548 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:48.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:48.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 394, 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:46:47.017360+0000", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:46:34.198958+0000", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5799936, 2024-04-25T05:46:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 10, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:46:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 11, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:46:48.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 11, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "174", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "171", 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.712 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.713 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.714 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.715 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.1700000762939453, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.1700000762939453, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:46:48.716 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.717 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.718 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.719 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.720 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.721 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.722 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6700000762939453, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6700000762939453, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.75, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.723 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.724 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.725 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.727 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.727 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:46:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:46:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.7400000095367432, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.7400000095367432, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.4100000858306885, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.4100000858306885, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:46:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.4900000095367432, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.4900000095367432, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.309999942779541, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.309999942779541, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:46:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.2400000095367432, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.2400000095367432, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.9700000286102295, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.9700000286102295, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 10, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_3", 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:46:36.375039+0000", 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:46:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:46:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "390", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:46:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:46:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:46:48.756 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 3.4800000190734863, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 3.4800000190734863, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 3.059999942779541, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 3.059999942779541, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.95134353637695312, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 0.94999998807907104, 2024-04-25T05:46:48.757 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 0, 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-25T05:46:48.758 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 304, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.759 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:46:48.762 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.763 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:46:48.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:46:48.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:46:48.767 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.768 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.770 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:46:48.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:46:48.772 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.773 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:46:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0.610748291015625, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 386, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:46:48.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:46:48.781 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.783 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:48.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:48.784 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.787 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:46:48.788 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 387, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:46:48.791 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:46:48.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:46:48.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.795 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 65536, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:46:40.371345+0000", 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:46:28.659382+0000", 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.797 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:45:38.197299+0000", 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:46:48.798 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:45:53.175414+0000", 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.799 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:45:57.115334+0000", 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.800 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:45:58.986301+0000", 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.801 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:46:38.253169+0000", 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.802 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:46:23.222016+0000", 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:46:48.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1c", 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: 5 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.3", 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.805 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.d", 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.807 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 7 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1f", 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.0", 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 6 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1a", 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 4, 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 3 2024-04-25T05:46:48.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 7, 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.9", 2024-04-25T05:46:48.810 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 3, 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [], 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.811 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:46:48.812 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:46:48.813 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:46:48.814 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:46:48.814 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:46:48.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:46:48.814 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:46:48.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:48.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:48.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:48.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 18593 18577 165.846 136.812 0.00503349 0.00602347 2024-04-25T05:46:49.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:49.054 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:49.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:46:49.174 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: pgmap v687: 158 pgs: 158 active+clean; 335 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-25T05:46:49.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/358355910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:46:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:49.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:49.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:49.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:46:49.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: pgmap v687: 158 pgs: 158 active+clean; 335 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-25T05:46:49.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/358355910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: pgmap v687: 158 pgs: 158 active+clean; 335 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-25T05:46:49.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/358355910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:46:49.402 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6421 8.5 8.f1be6a5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.11895765100002, 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.497201589, 2024-04-25T05:46:49.403 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6421 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277266+0000", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277267+0000", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1319999999999999e-06 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277387+0000", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011962 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277399+0000", 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2403e-05 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277406+0000", 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7109999999999998e-06 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277467+0000", 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1427000000000007e-05 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.277479+0000", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1579999999999999e-05 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.774455+0000", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.496976258 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.774468+0000", 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2458e-05 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6434 8.9 8.bb656db9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.11589990300001, 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.49508851199999998, 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6434 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280324+0000", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280325+0000", 2024-04-25T05:46:49.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0580000000000001e-06 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280387+0000", 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2152999999999999e-05 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280404+0000", 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.658e-05 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280412+0000", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3950000000000001e-06 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280479+0000", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6492e-05 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.280487+0000", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7269999999999996e-06 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775403+0000", 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.494915463 2024-04-25T05:46:49.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775412+0000", 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6439999999999995e-06 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6438 8.9 8.e7f58319 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.10962160899999, 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48884452099999998, 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.412 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6438 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286602+0000", 2024-04-25T05:46:49.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286603+0000", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.139e-06 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286692+0000", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8694000000000002e-05 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286705+0000", 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3135999999999999e-05 2024-04-25T05:46:49.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286714+0000", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7439999999999996e-06 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286763+0000", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8854999999999999e-05 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.286768+0000", 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3709999999999999e-06 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.415 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775438+0000", 2024-04-25T05:46:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.488669992 2024-04-25T05:46:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.417 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775447+0000", 2024-04-25T05:46:49.417 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5900000000000008e-06 2024-04-25T05:46:49.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.418 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6446 8.6 8.62fadf16 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.106903354, 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48884683800000001, 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6446 2024-04-25T05:46:49.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289320+0000", 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289321+0000", 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5300000000000004e-07 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289390+0000", 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8848000000000002e-05 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289400+0000", 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0473e-05 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289406+0000", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8510000000000004e-06 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289473+0000", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6346000000000005e-05 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.289491+0000", 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8439000000000001e-05 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778157+0000", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48866554099999998 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778167+0000", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0686999999999999e-05 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6450 8.9 8.8eb80129 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:46:49.424 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.104376285, 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48362897700000002, 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6450 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:46:49.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291848+0000", 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4499999999999996e-07 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291910+0000", 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1204999999999994e-05 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291920+0000", 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0862e-05 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291927+0000", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5629999999999997e-06 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291992+0000", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4504000000000004e-05 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.291999+0000", 2024-04-25T05:46:49.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8159999999999994e-06 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775469+0000", 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48346932799999998 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775477+0000", 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8539999999999997e-06 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6453 8.6 8.63dba756 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:46:49.430 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.10316551199998, 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48515036299999997, 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6453 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293058+0000", 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293059+0000", 2024-04-25T05:46:49.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5899999999999995e-07 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293122+0000", 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2519000000000002e-05 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293132+0000", 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0234000000000001e-05 2024-04-25T05:46:49.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293138+0000", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7590000000000003e-06 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293189+0000", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1654999999999999e-05 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.293198+0000", 2024-04-25T05:46:49.435 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5429999999999994e-06 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778200+0000", 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48500220399999999 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.778209+0000", 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5900000000000008e-06 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.436 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:6455 8.9 8.8d6e43a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 440.101980819, 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48138075499999999, 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6455 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.437 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294243+0000", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294244+0000", 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.99e-07 2024-04-25T05:46:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294314+0000", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0374000000000001e-05 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294326+0000", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2265e-05 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294336+0000", 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4110000000000002e-06 2024-04-25T05:46:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294382+0000", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5765e-05 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.294386+0000", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6709999999999998e-06 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775604+0000", 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48121744700000002 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:39:29.775624+0000", 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0123e-05 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124108 8.9 8:9546d480:::benchmark_data_smithi043_104001_object124107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 352.28513167199998, 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16467942699999999, 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124108 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111092+0000", 2024-04-25T05:46:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111094+0000", 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4950000000000001e-06 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111129+0000", 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5827000000000001e-05 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111141+0000", 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1758e-05 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111150+0000", 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0480000000000004e-06 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111533+0000", 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038226200000000002 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.444 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111676+0000", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014314800000000001 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111863+0000", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018720400000000001 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116324+0000", 2024-04-25T05:46:49.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044614750000000003 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116341+0000", 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6549999999999999e-05 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.118495+0000", 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021544089999999999 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275636+0000", 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15714107499999999 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275702+0000", 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5479000000000001e-05 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275772+0000", 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9696999999999994e-05 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125653 8.9 8:998fdd37:::benchmark_data_smithi043_104001_object1529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 349.11739686200002, 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11526359899999999, 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125653 2024-04-25T05:46:49.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278827+0000", 2024-04-25T05:46:49.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:49.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278828+0000", 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2589999999999999e-06 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278834+0000", 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1149999999999999e-06 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278847+0000", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2585000000000001e-05 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.278855+0000", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0150000000000003e-06 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280624+0000", 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001768635 2024-04-25T05:46:49.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280741+0000", 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117799 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.280865+0000", 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000123301 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287412+0000", 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.006547766 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287422+0000", 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4779999999999997e-06 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294464+0000", 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070417780000000003 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.393906+0000", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099442738000000003 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.393962+0000", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5572000000000003e-05 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.394091+0000", 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128558 2024-04-25T05:46:49.455 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125666 8.9 8:9242a807:::benchmark_data_smithi043_104001_object1542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:46:49.456 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 349.10875315099997, 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.112053468, 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125666 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:46:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287471+0000", 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287474+0000", 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3299999999999999e-06 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287480+0000", 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2430000000000002e-06 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287492+0000", 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2150999999999999e-05 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287510+0000", 2024-04-25T05:46:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7833999999999999e-05 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287630+0000", 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119732 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287867+0000", 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000237235 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288022+0000", 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015471199999999999 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291108+0000", 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003086082 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294515+0000", 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034066919999999998 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296045+0000", 2024-04-25T05:46:49.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015305239999999999 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399316+0000", 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.103270836 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399392+0000", 2024-04-25T05:46:49.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5893e-05 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399524+0000", 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132204 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125667 8.9 8:928428f8:::benchmark_data_smithi043_104001_object1543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 349.10758120100002, 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.111196555, 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.466 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125667 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288643+0000", 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288644+0000", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6899999999999996e-07 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288660+0000", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6252e-05 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288677+0000", 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7002000000000001e-05 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.469 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.288686+0000", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8459999999999997e-06 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291154+0000", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024682900000000002 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291315+0000", 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016116399999999999 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291477+0000", 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016224300000000001 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.294566+0000", 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030882739999999998 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296052+0000", 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001486076 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.296138+0000", 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6753000000000001e-05 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399659+0000", 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10352016999999999 2024-04-25T05:46:49.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399721+0000", 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2592e-05 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.399839+0000", 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117924 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125909 8.9 8:94a2b73d:::benchmark_data_smithi043_104001_object1785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 348.79106691300001, 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115848672, 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.474 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125909 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605157+0000", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1399999999999997e-07 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605162+0000", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7859999999999996e-06 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605168+0000", 2024-04-25T05:46:49.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1430000000000003e-06 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.605174+0000", 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4759999999999997e-06 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606391+0000", 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001217079 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606512+0000", 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012084899999999999 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.606639+0000", 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012696499999999999 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.611089+0000", 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044506930000000004 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.611123+0000", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3827e-05 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.612135+0000", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001011458 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.720612+0000", 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.108477643 2024-04-25T05:46:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.720676+0000", 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3501000000000001e-05 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.721006+0000", 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00032973799999999999 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129544 8.9 8:983c06aa:::benchmark_data_smithi043_104001_object5420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 344.38275159800003, 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13394251400000001, 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129544 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.481 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013472+0000", 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.482 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013473+0000", 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.249e-06 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013477+0000", 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7290000000000002e-06 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013485+0000", 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8800000000000008e-06 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013491+0000", 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0599999999999996e-06 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013565+0000", 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4268000000000001e-05 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013690+0000", 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012419699999999999 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013817+0000", 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127875 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.016786+0000", 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029689080000000001 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.017604+0000", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00081734600000000004 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018660+0000", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010562309999999999 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147216+0000", 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.128556437 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147282+0000", 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5395000000000007e-05 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.147415+0000", 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000132939 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129548 8.9 8:90c26391:::benchmark_data_smithi043_104001_object5424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 344.37752285900001, 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12964294300000001, 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129548 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.488 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018701+0000", 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018702+0000", 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2160000000000001e-06 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018711+0000", 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4090000000000003e-06 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018722+0000", 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1080000000000001e-05 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018729+0000", 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4460000000000004e-06 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.021688+0000", 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029587459999999999 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.021861+0000", 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000173306 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022027+0000", 2024-04-25T05:46:49.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016550899999999999 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025055+0000", 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030286250000000001 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025839+0000", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00078327500000000005 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.025862+0000", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3841000000000001e-05 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148136+0000", 2024-04-25T05:46:49.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122273643 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148203+0000", 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7297000000000003e-05 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.148344+0000", 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014055 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "MOSDPGPushReply(8.9 246/233 [PushReplyOp(8:97a75abd:::benchmark_data_smithi043_104001_object42380:head)])", 2024-04-25T05:46:49.496 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 344.36976625199998, 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11376488899999999, 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026458+0000", 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.48e-07 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026461+0000", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3280000000000002e-06 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026468+0000", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2470000000000002e-06 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026476+0000", 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8369999999999997e-06 2024-04-25T05:46:49.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.026525+0000", 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8912000000000002e-05 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.140222+0000", 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11369731700000001 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:149154 8.9 8:9744f1fd:::benchmark_data_smithi043_104001_object25030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 324.31492762400001, 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11038581, 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.502 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 149154 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081296+0000", 2024-04-25T05:46:49.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081298+0000", 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8810000000000001e-06 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081305+0000", 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9020000000000002e-06 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081327+0000", 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1769000000000001e-05 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081334+0000", 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6939999999999995e-06 2024-04-25T05:46:49.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081396+0000", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1356e-05 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081519+0000", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012302799999999999 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.081645+0000", 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012661399999999999 2024-04-25T05:46:49.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.084389+0000", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002743161 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.087947+0000", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003558748 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.089766+0000", 2024-04-25T05:46:49.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001818587 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191451+0000", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.101684582 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191552+0000", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101734 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:25.191682+0000", 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012975399999999999 2024-04-25T05:46:49.509 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226342 8.5 8:a7a71f38:::benchmark_data_smithi043_104001_object102218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 249.52259367600001, 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.208876007, 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.510 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226342 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873630+0000", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873632+0000", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.784e-06 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873639+0000", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1029999999999996e-06 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873654+0000", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5387999999999999e-05 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873663+0000", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9679999999999995e-06 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873759+0000", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6012000000000002e-05 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.873897+0000", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000137479 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:46:49.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.874034+0000", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013683900000000001 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881894+0000", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0078598769999999995 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881921+0000", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6939000000000001e-05 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082491+0000", 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.20057040500000001 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082506+0000", 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5213e-05 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226342 8.5 8:a7a71f38:::benchmark_data_smithi043_104001_object102218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:46:49.515 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 249.50265714599999, 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1555693469999999, 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226342 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893569+0000", 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2000000000000001e-06 2024-04-25T05:46:49.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893577+0000", 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5590000000000006e-06 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893597+0000", 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9826e-05 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893608+0000", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1221e-05 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893676+0000", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7723000000000002e-05 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893681+0000", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6539999999999998e-06 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082813+0000", 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.189132576 2024-04-25T05:46:49.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082823+0000", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6500000000000008e-06 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015505+0000", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.93268209199999996 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015634+0000", 2024-04-25T05:46:49.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012852800000000001 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048918+0000", 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033284728999999999 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049136+0000", 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021758899999999999 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:226352 8.7 e282/209)", 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 249.489181051, 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.207367144, 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.522 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907043+0000", 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.34e-07 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907069+0000", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5814999999999999e-05 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907079+0000", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0471e-05 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.907099+0000", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9745000000000002e-05 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.114390+0000", 2024-04-25T05:46:49.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.207291169 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.114410+0000", 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.961e-05 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226356 8.5 8:a631803d:::benchmark_data_smithi043_104001_object102232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 249.266523095, 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92008493099999999, 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:49.525 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226356 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:49.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129703+0000", 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6369999999999999e-06 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129718+0000", 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4413000000000001e-05 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:49.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129742+0000", 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.472e-05 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129757+0000", 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4751e-05 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130011+0000", 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025401500000000001 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130048+0000", 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6837e-05 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:49.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016167+0000", 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88611901299999996 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016334+0000", 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016642 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016642+0000", 2024-04-25T05:46:49.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00030824900000000002 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048561+0000", 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.031919212000000002 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049494+0000", 2024-04-25T05:46:49.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093273599999999998 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049612+0000", 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118149 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049689+0000", 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7195999999999997e-05 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:49.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049786+0000", 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6582999999999993e-05 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:49.534 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:49.535 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.537 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:49.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:49.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:49.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:49.770 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:49.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:49.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:49.886 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:49.887 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:49.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:49.922 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 20752 20736 161.981 134.938 0.00280386 0.00614551 2024-04-25T05:46:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:49.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:49.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:46:50.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:50.090 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:50.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:50.224 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:50.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:50.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: 4.19 scrub starts 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: 4.19 scrub ok 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: 4.d scrub starts 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: 4.d scrub ok 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3449560991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[37492]: osdmap e395: 8 total, 8 up, 7 in 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: 4.19 scrub starts 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: 4.19 scrub ok 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: 4.d scrub starts 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: 4.d scrub ok 2024-04-25T05:46:50.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3449560991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:50.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:50 smithi043 ceph-mon[27010]: osdmap e395: 8 total, 8 up, 7 in 2024-04-25T05:46:50.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:50.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:50.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: 4.19 scrub starts 2024-04-25T05:46:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: 4.19 scrub ok 2024-04-25T05:46:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: 4.d scrub starts 2024-04-25T05:46:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: 4.d scrub ok 2024-04-25T05:46:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3449560991' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:50.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:50 smithi060 ceph-mon[30898]: osdmap e395: 8 total, 8 up, 7 in 2024-04-25T05:46:50.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.618 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:50.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:50.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:50.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:50.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:50.891 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:50.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:50.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 22402 22386 155.441 103.125 0.00380699 0.00642716 2024-04-25T05:46:50.992 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:51.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:51.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:51.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 6.1f scrub starts 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: pgmap v688: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 6.1f scrub ok 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:46:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:51 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:46:51.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:51.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:51.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 6.1f scrub starts 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: pgmap v688: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 6.1f scrub ok 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 6.1f scrub starts 2024-04-25T05:46:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:46:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:46:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: pgmap v688: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:46:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 6.1f scrub ok 2024-04-25T05:46:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:46:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:51 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:46:51.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:51.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:51.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:51.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:51.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:51.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:51.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:51.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:51.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 24252 24236 151.458 115.625 0.00637076 0.0065965 2024-04-25T05:46:51.953 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:51.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:52.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:52.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:52.212 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:52.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:52.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:52 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:46:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:52 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:46:52.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:52 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:46:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:52 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:46:52.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:52.473 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:52.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:52 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:46:52.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:52 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:46:52.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:52.601 INFO:tasks.thrashosds.thrasher:j is {'epoch': 394, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:46:47.017360+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:46:34.198958+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '171', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.740000009536743, 'score_stable': 2.740000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.4100000858306885, 'raw_score_stable': 2.4100000858306885, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-04-25T05:46:36.375039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '390', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.4800000190734863, 'score_stable': 3.4800000190734863, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.9513435363769531, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 304, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.610748291015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 387, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T05:46:40.371345+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:46:28.659382+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:45:38.197299+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:45:53.175414+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:45:57.115334+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:45:58.986301+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:46:38.253169+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:46:23.222016+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}, {'pgid': '5.1c', 'osds': [6, 0, 5]}], 'pg_upmap_items': [{'pgid': '4.3', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.0', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1a', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.3', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '6.9', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:46:52.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:46:52.606 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:52.667 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:52.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:52.732 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:52.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:52.868 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:52.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:52.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 16 25731 25715 146.092 92.4375 0.00604124 0.00683939 2024-04-25T05:46:52.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:53.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:53.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:53.141 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:53.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:53.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:53.335 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:53.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 6.c scrub starts 2024-04-25T05:46:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 6.c scrub ok 2024-04-25T05:46:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: pgmap v690: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 733 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:46:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:46:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 6.c scrub starts 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 6.c scrub ok 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: pgmap v690: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 733 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:46:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:53 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:46:53.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 6.c scrub starts 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 6.c scrub ok 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: pgmap v690: 158 pgs: 158 active+clean; 1.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 733 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:46:53.423 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:46:53.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:46:53.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.526 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:53.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:53.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:53.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:46:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:46:53.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.838 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:53.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 27012 26996 140.588 80.0625 0.00189263 0.00707672 2024-04-25T05:46:53.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:53.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:53.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:53.992 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:54.089 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:54.122 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:46:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.126 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:54.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:54.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:54.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:54.356 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 4.4 scrub starts 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 4.4 scrub ok 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 6.b scrub starts 2024-04-25T05:46:54.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 6.b scrub ok 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: osdmap e396: 8 total, 8 up, 7 in 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3815037585' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 4.4 scrub starts 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 4.4 scrub ok 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 6.b scrub starts 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 6.b scrub ok 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: osdmap e396: 8 total, 8 up, 7 in 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3815037585' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:54 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.364 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:46:54.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:54.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:54.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 4.4 scrub starts 2024-04-25T05:46:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 4.4 scrub ok 2024-04-25T05:46:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:46:54.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 6.b scrub starts 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 6.b scrub ok 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: osdmap e396: 8 total, 8 up, 7 in 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3815037585' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:54 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:46:54.520 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:54.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:54.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:54.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:54.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:54.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:54.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:54.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:54.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 28912 28896 138.908 118.75 0.00850652 0.00719313 2024-04-25T05:46:55.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:55.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:55.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: pgmap v692: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 202 MiB/s wr, 3.23k op/s 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: osdmap e397: 8 total, 8 up, 7 in 2024-04-25T05:46:55.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[27010]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: pgmap v692: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 202 MiB/s wr, 3.23k op/s 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: osdmap e397: 8 total, 8 up, 7 in 2024-04-25T05:46:55.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:55 smithi043 ceph-mon[37492]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:55.176 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:55.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:55.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:55.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:55.424 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:46:55.424 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:46:55.425 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: pgmap v692: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 202 MiB/s wr, 3.23k op/s 2024-04-25T05:46:55.425 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:46:55.425 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:46:55.425 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: osdmap e397: 8 total, 8 up, 7 in 2024-04-25T05:46:55.425 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:55 smithi060 ceph-mon[30898]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:46:55.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:55.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:55.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:55.582 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:46:55.621 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:46:55.657 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:46:55.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:55.694 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:46:55.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:55.734 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:46:55.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:46:55.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:46:55.836 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:55.864 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:46:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:46:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:46:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:46:55.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 15 30775 30760 137.307 116.5 0.00551907 0.00727755 2024-04-25T05:46:55.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:55.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:55.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:46:56.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:56.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:56.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:56.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2830114944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.258 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:56.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:56.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:56.390 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2830114944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:56.390 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2830114944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:46:56.435 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2024-04-25T05:46:54.104058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1702894107,"num_objects":26205,"num_object_clones":0,"num_object_copies":78615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26205,"num_whiteouts":0,"num_read":2167,"num_read_kb":1887,"num_write":26889,"num_write_kb":1662445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":1196524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28455,"ondisk_log_size":28455,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8999752,"kb_used_data":4940104,"kb_used_omap":10,"kb_used_meta":4059573,"kb_avail":647216312,"statfs":{"total":671965249536,"available":662749503488,"internally_reserved":0,"allocated":5058666496,"data_stored":5054743001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4157002859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1352007680,"num_objects":20630,"num_object_clones":0,"num_object_copies":61890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20630,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":20630,"num_write_kb":1320320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.180437"},"pg_stats":[{"pgid":"10.2","version":"396'1684","reported_seq":1712,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.871271+0000","last_change":"2024-04-25T05:46:37.979403+0000","last_active":"2024-04-25T05:46:53.871271+0000","last_peered":"2024-04-25T05:46:53.871271+0000","last_clean":"2024-04-25T05:46:53.871271+0000","last_became_active":"2024-04-25T05:46:37.979097+0000","last_became_peered":"2024-04-25T05:46:37.979097+0000","last_unstale":"2024-04-25T05:46:53.871271+0000","last_undegraded":"2024-04-25T05:46:53.871271+0000","last_fullsized":"2024-04-25T05:46:53.871271+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1684,"log_dups_size":0,"ondisk_log_size":1684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.041548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1684,"num_write_kb":107776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":424,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068631+0000","last_change":"2024-04-25T05:46:09.856702+0000","last_active":"2024-04-25T05:46:53.068631+0000","last_peered":"2024-04-25T05:46:53.068631+0000","last_clean":"2024-04-25T05:46:53.068631+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:46:53.068631+0000","last_undegraded":"2024-04-25T05:46:53.068631+0000","last_fullsized":"2024-04-25T05:46:53.068631+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:46:09.856439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.856439+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:35.321244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":715,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067928+0000","last_change":"2024-04-25T05:46:03.753399+0000","last_active":"2024-04-25T05:46:53.067928+0000","last_peered":"2024-04-25T05:46:53.067928+0000","last_clean":"2024-04-25T05:46:53.067928+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:46:53.067928+0000","last_undegraded":"2024-04-25T05:46:53.067928+0000","last_fullsized":"2024-04-25T05:46:53.067928+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:03.753274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:03.753274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:20.423586+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":68,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.083145+0000","last_change":"2024-04-25T05:46:35.013019+0000","last_active":"2024-04-25T05:46:53.083145+0000","last_peered":"2024-04-25T05:46:53.083145+0000","last_clean":"2024-04-25T05:46:53.083145+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:46:53.083145+0000","last_undegraded":"2024-04-25T05:46:53.083145+0000","last_fullsized":"2024-04-25T05:46:53.083145+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:45:36.284279+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.284279+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:02.997357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":385,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065296+0000","last_change":"2024-04-25T05:46:19.993548+0000","last_active":"2024-04-25T05:46:53.065296+0000","last_peered":"2024-04-25T05:46:53.065296+0000","last_clean":"2024-04-25T05:46:53.065296+0000","last_became_active":"2024-04-25T05:46:19.990605+0000","last_became_peered":"2024-04-25T05:46:19.990605+0000","last_unstale":"2024-04-25T05:46:53.065296+0000","last_undegraded":"2024-04-25T05:46:53.065296+0000","last_fullsized":"2024-04-25T05:46:53.065296+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1028,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965756+0000","last_change":"2024-04-25T05:46:23.149014+0000","last_active":"2024-04-25T05:46:49.965756+0000","last_peered":"2024-04-25T05:46:49.965756+0000","last_clean":"2024-04-25T05:46:49.965756+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:46:49.965756+0000","last_undegraded":"2024-04-25T05:46:49.965756+0000","last_fullsized":"2024-04-25T05:46:49.965756+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:52.225304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"396'1674","reported_seq":1700,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.126318+0000","last_change":"2024-04-25T05:46:37.983434+0000","last_active":"2024-04-25T05:46:53.126318+0000","last_peered":"2024-04-25T05:46:53.126318+0000","last_clean":"2024-04-25T05:46:53.126318+0000","last_became_active":"2024-04-25T05:46:37.983078+0000","last_became_peered":"2024-04-25T05:46:37.983078+0000","last_unstale":"2024-04-25T05:46:53.126318+0000","last_undegraded":"2024-04-25T05:46:53.126318+0000","last_fullsized":"2024-04-25T05:46:53.126318+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1674,"log_dups_size":0,"ondisk_log_size":1674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.882443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109707264,"num_objects":1674,"num_object_clones":0,"num_object_copies":5022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1674,"num_write_kb":107136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":98,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.083199+0000","last_change":"2024-04-25T05:46:48.976027+0000","last_active":"2024-04-25T05:46:53.083199+0000","last_peered":"2024-04-25T05:46:53.083199+0000","last_clean":"2024-04-25T05:46:53.083199+0000","last_became_active":"2024-04-25T05:46:36.984563+0000","last_became_peered":"2024-04-25T05:46:36.984563+0000","last_unstale":"2024-04-25T05:46:53.083199+0000","last_undegraded":"2024-04-25T05:46:53.083199+0000","last_fullsized":"2024-04-25T05:46:53.083199+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:46:48.975826+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.975826+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:11.159170+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":427,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966212+0000","last_change":"2024-04-25T05:46:42.728489+0000","last_active":"2024-04-25T05:46:49.966212+0000","last_peered":"2024-04-25T05:46:49.966212+0000","last_clean":"2024-04-25T05:46:49.966212+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:46:49.966212+0000","last_undegraded":"2024-04-25T05:46:49.966212+0000","last_fullsized":"2024-04-25T05:46:49.966212+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:42.728367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:46:42.728367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:04.407596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":721,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065801+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:46:53.065801+0000","last_peered":"2024-04-25T05:46:53.065801+0000","last_clean":"2024-04-25T05:46:53.065801+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:46:53.065801+0000","last_undegraded":"2024-04-25T05:46:53.065801+0000","last_fullsized":"2024-04-25T05:46:53.065801+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":48,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.082666+0000","last_change":"2024-04-25T05:46:20.012603+0000","last_active":"2024-04-25T05:46:53.082666+0000","last_peered":"2024-04-25T05:46:53.082666+0000","last_clean":"2024-04-25T05:46:53.082666+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:46:53.082666+0000","last_undegraded":"2024-04-25T05:46:53.082666+0000","last_fullsized":"2024-04-25T05:46:53.082666+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":618,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964779+0000","last_change":"2024-04-25T05:46:37.981262+0000","last_active":"2024-04-25T05:46:49.964779+0000","last_peered":"2024-04-25T05:46:49.964779+0000","last_clean":"2024-04-25T05:46:49.964779+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:46:49.964779+0000","last_undegraded":"2024-04-25T05:46:49.964779+0000","last_fullsized":"2024-04-25T05:46:49.964779+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"395'1630","reported_seq":1656,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.928166+0000","last_change":"2024-04-25T05:46:37.984708+0000","last_active":"2024-04-25T05:46:52.928166+0000","last_peered":"2024-04-25T05:46:52.928166+0000","last_clean":"2024-04-25T05:46:52.928166+0000","last_became_active":"2024-04-25T05:46:37.984393+0000","last_became_peered":"2024-04-25T05:46:37.984393+0000","last_unstale":"2024-04-25T05:46:52.928166+0000","last_undegraded":"2024-04-25T05:46:52.928166+0000","last_fullsized":"2024-04-25T05:46:52.928166+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1630,"log_dups_size":0,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:41.038864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1630,"num_write_kb":104320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965876+0000","last_change":"2024-04-25T05:45:26.763116+0000","last_active":"2024-04-25T05:46:49.965876+0000","last_peered":"2024-04-25T05:46:49.965876+0000","last_clean":"2024-04-25T05:46:49.965876+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:46:49.965876+0000","last_undegraded":"2024-04-25T05:46:49.965876+0000","last_fullsized":"2024-04-25T05:46:49.965876+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:45:26.762873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:26.762873+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.956625+0000","last_change":"2024-04-25T05:46:19.963247+0000","last_active":"2024-04-25T05:46:49.956625+0000","last_peered":"2024-04-25T05:46:49.956625+0000","last_clean":"2024-04-25T05:46:49.956625+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:46:49.956625+0000","last_undegraded":"2024-04-25T05:46:49.956625+0000","last_fullsized":"2024-04-25T05:46:49.956625+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:44.222624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965923+0000","last_change":"2024-04-25T05:46:34.982029+0000","last_active":"2024-04-25T05:46:49.965923+0000","last_peered":"2024-04-25T05:46:49.965923+0000","last_clean":"2024-04-25T05:46:49.965923+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:46:49.965923+0000","last_undegraded":"2024-04-25T05:46:49.965923+0000","last_fullsized":"2024-04-25T05:46:49.965923+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:18.666459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:46:18.666459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.700514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":709,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068419+0000","last_change":"2024-04-25T05:46:51.568352+0000","last_active":"2024-04-25T05:46:53.068419+0000","last_peered":"2024-04-25T05:46:53.068419+0000","last_clean":"2024-04-25T05:46:53.068419+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:46:53.068419+0000","last_undegraded":"2024-04-25T05:46:53.068419+0000","last_fullsized":"2024-04-25T05:46:53.068419+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:51.568255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:51.568255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.506539+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"395'1691","reported_seq":1717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.011218+0000","last_change":"2024-04-25T05:46:37.983353+0000","last_active":"2024-04-25T05:46:53.011218+0000","last_peered":"2024-04-25T05:46:53.011218+0000","last_clean":"2024-04-25T05:46:53.011218+0000","last_became_active":"2024-04-25T05:46:37.983037+0000","last_became_peered":"2024-04-25T05:46:37.983037+0000","last_unstale":"2024-04-25T05:46:53.011218+0000","last_undegraded":"2024-04-25T05:46:53.011218+0000","last_fullsized":"2024-04-25T05:46:53.011218+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1691,"log_dups_size":0,"ondisk_log_size":1691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:51.005000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110821376,"num_objects":1691,"num_object_clones":0,"num_object_copies":5073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1691,"num_write_kb":108224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":743,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065290+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:46:53.065290+0000","last_peered":"2024-04-25T05:46:53.065290+0000","last_clean":"2024-04-25T05:46:53.065290+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:46:53.065290+0000","last_undegraded":"2024-04-25T05:46:53.065290+0000","last_fullsized":"2024-04-25T05:46:53.065290+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":697,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067391+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:46:53.067391+0000","last_peered":"2024-04-25T05:46:53.067391+0000","last_clean":"2024-04-25T05:46:53.067391+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:46:53.067391+0000","last_undegraded":"2024-04-25T05:46:53.067391+0000","last_fullsized":"2024-04-25T05:46:53.067391+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'15","reported_seq":37,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964518+0000","last_change":"2024-04-25T05:46:35.014060+0000","last_active":"2024-04-25T05:46:49.964518+0000","last_peered":"2024-04-25T05:46:49.964518+0000","last_clean":"2024-04-25T05:46:49.964518+0000","last_became_active":"2024-04-25T05:46:34.989992+0000","last_became_peered":"2024-04-25T05:46:34.989992+0000","last_unstale":"2024-04-25T05:46:49.964518+0000","last_undegraded":"2024-04-25T05:46:49.964518+0000","last_fullsized":"2024-04-25T05:46:49.964518+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964164+0000","last_change":"2024-04-25T05:46:34.990045+0000","last_active":"2024-04-25T05:46:49.964164+0000","last_peered":"2024-04-25T05:46:49.964164+0000","last_clean":"2024-04-25T05:46:49.964164+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:46:49.964164+0000","last_undegraded":"2024-04-25T05:46:49.964164+0000","last_fullsized":"2024-04-25T05:46:49.964164+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.239865+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":583,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965100+0000","last_change":"2024-04-25T05:46:37.982298+0000","last_active":"2024-04-25T05:46:49.965100+0000","last_peered":"2024-04-25T05:46:49.965100+0000","last_clean":"2024-04-25T05:46:49.965100+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:46:49.965100+0000","last_undegraded":"2024-04-25T05:46:49.965100+0000","last_fullsized":"2024-04-25T05:46:49.965100+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"395'1542","reported_seq":1568,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.931922+0000","last_change":"2024-04-25T05:46:37.983920+0000","last_active":"2024-04-25T05:46:52.931922+0000","last_peered":"2024-04-25T05:46:52.931922+0000","last_clean":"2024-04-25T05:46:52.931922+0000","last_became_active":"2024-04-25T05:46:37.983600+0000","last_became_peered":"2024-04-25T05:46:37.983600+0000","last_unstale":"2024-04-25T05:46:52.931922+0000","last_undegraded":"2024-04-25T05:46:52.931922+0000","last_fullsized":"2024-04-25T05:46:52.931922+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:37.012992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101056512,"num_objects":1542,"num_object_clones":0,"num_object_copies":4626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1542,"num_write_kb":98688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":704,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.956530+0000","last_change":"2024-04-25T05:45:29.415240+0000","last_active":"2024-04-25T05:46:49.956530+0000","last_peered":"2024-04-25T05:46:49.956530+0000","last_clean":"2024-04-25T05:46:49.956530+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:46:49.956530+0000","last_undegraded":"2024-04-25T05:46:49.956530+0000","last_fullsized":"2024-04-25T05:46:49.956530+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.415117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.415117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":852,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067493+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:46:53.067493+0000","last_peered":"2024-04-25T05:46:53.067493+0000","last_clean":"2024-04-25T05:46:53.067493+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:46:53.067493+0000","last_undegraded":"2024-04-25T05:46:53.067493+0000","last_fullsized":"2024-04-25T05:46:53.067493+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":729,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068509+0000","last_change":"2024-04-25T05:46:34.972300+0000","last_active":"2024-04-25T05:46:53.068509+0000","last_peered":"2024-04-25T05:46:53.068509+0000","last_clean":"2024-04-25T05:46:53.068509+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:46:53.068509+0000","last_undegraded":"2024-04-25T05:46:53.068509+0000","last_fullsized":"2024-04-25T05:46:53.068509+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:21.023940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:21.023940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":380,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068364+0000","last_change":"2024-04-25T05:46:19.998231+0000","last_active":"2024-04-25T05:46:53.068364+0000","last_peered":"2024-04-25T05:46:53.068364+0000","last_clean":"2024-04-25T05:46:53.068364+0000","last_became_active":"2024-04-25T05:46:19.997163+0000","last_became_peered":"2024-04-25T05:46:19.997163+0000","last_unstale":"2024-04-25T05:46:53.068364+0000","last_undegraded":"2024-04-25T05:46:53.068364+0000","last_fullsized":"2024-04-25T05:46:53.068364+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":932,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.957076+0000","last_change":"2024-04-25T05:46:23.130821+0000","last_active":"2024-04-25T05:46:49.957076+0000","last_peered":"2024-04-25T05:46:49.957076+0000","last_clean":"2024-04-25T05:46:49.957076+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:46:49.957076+0000","last_undegraded":"2024-04-25T05:46:49.957076+0000","last_fullsized":"2024-04-25T05:46:49.957076+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:36.587388+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.587388+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:01.606220+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"395'1618","reported_seq":1644,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.932338+0000","last_change":"2024-04-25T05:46:37.985638+0000","last_active":"2024-04-25T05:46:52.932338+0000","last_peered":"2024-04-25T05:46:52.932338+0000","last_clean":"2024-04-25T05:46:52.932338+0000","last_became_active":"2024-04-25T05:46:37.982701+0000","last_became_peered":"2024-04-25T05:46:37.982701+0000","last_unstale":"2024-04-25T05:46:52.932338+0000","last_undegraded":"2024-04-25T05:46:52.932338+0000","last_fullsized":"2024-04-25T05:46:52.932338+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1618,"log_dups_size":0,"ondisk_log_size":1618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:41.010962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":103552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":863,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067612+0000","last_change":"2024-04-25T05:45:36.207163+0000","last_active":"2024-04-25T05:46:53.067612+0000","last_peered":"2024-04-25T05:46:53.067612+0000","last_clean":"2024-04-25T05:46:53.067612+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:46:53.067612+0000","last_undegraded":"2024-04-25T05:46:53.067612+0000","last_fullsized":"2024-04-25T05:46:53.067612+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:45:34.969209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:34.969209+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.116597+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":48,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963142+0000","last_change":"2024-04-25T05:46:35.010369+0000","last_active":"2024-04-25T05:46:49.963142+0000","last_peered":"2024-04-25T05:46:49.963142+0000","last_clean":"2024-04-25T05:46:49.963142+0000","last_became_active":"2024-04-25T05:46:34.980031+0000","last_became_peered":"2024-04-25T05:46:34.980031+0000","last_unstale":"2024-04-25T05:46:49.963142+0000","last_undegraded":"2024-04-25T05:46:49.963142+0000","last_fullsized":"2024-04-25T05:46:49.963142+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:45:27.764695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.764695+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":402,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065652+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:46:53.065652+0000","last_peered":"2024-04-25T05:46:53.065652+0000","last_clean":"2024-04-25T05:46:53.065652+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:46:53.065652+0000","last_undegraded":"2024-04-25T05:46:53.065652+0000","last_fullsized":"2024-04-25T05:46:53.065652+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":80,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.066417+0000","last_change":"2024-04-25T05:46:19.992323+0000","last_active":"2024-04-25T05:46:53.066417+0000","last_peered":"2024-04-25T05:46:53.066417+0000","last_clean":"2024-04-25T05:46:53.066417+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:46:53.066417+0000","last_undegraded":"2024-04-25T05:46:53.066417+0000","last_fullsized":"2024-04-25T05:46:53.066417+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":898,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068255+0000","last_change":"2024-04-25T05:46:37.983775+0000","last_active":"2024-04-25T05:46:53.068255+0000","last_peered":"2024-04-25T05:46:53.068255+0000","last_clean":"2024-04-25T05:46:53.068255+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:46:53.068255+0000","last_undegraded":"2024-04-25T05:46:53.068255+0000","last_fullsized":"2024-04-25T05:46:53.068255+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.882669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.882669+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:00.446393+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"395'1643","reported_seq":1669,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.011378+0000","last_change":"2024-04-25T05:46:37.986037+0000","last_active":"2024-04-25T05:46:53.011378+0000","last_peered":"2024-04-25T05:46:53.011378+0000","last_clean":"2024-04-25T05:46:53.011378+0000","last_became_active":"2024-04-25T05:46:37.985670+0000","last_became_peered":"2024-04-25T05:46:37.985670+0000","last_unstale":"2024-04-25T05:46:53.011378+0000","last_undegraded":"2024-04-25T05:46:53.011378+0000","last_fullsized":"2024-04-25T05:46:53.011378+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1643,"log_dups_size":0,"ondisk_log_size":1643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:52.279419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":4929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1643,"num_write_kb":105152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":452,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965395+0000","last_change":"2024-04-25T05:46:46.694407+0000","last_active":"2024-04-25T05:46:49.965395+0000","last_peered":"2024-04-25T05:46:49.965395+0000","last_clean":"2024-04-25T05:46:49.965395+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:46:49.965395+0000","last_undegraded":"2024-04-25T05:46:49.965395+0000","last_fullsized":"2024-04-25T05:46:49.965395+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:46:46.694273+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.694273+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:04.268414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":447,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.066315+0000","last_change":"2024-04-25T05:46:45.354890+0000","last_active":"2024-04-25T05:46:53.066315+0000","last_peered":"2024-04-25T05:46:53.066315+0000","last_clean":"2024-04-25T05:46:53.066315+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:46:53.066315+0000","last_undegraded":"2024-04-25T05:46:53.066315+0000","last_fullsized":"2024-04-25T05:46:53.066315+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:46:45.354712+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.354712+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:50.813205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964254+0000","last_change":"2024-04-25T05:46:34.988645+0000","last_active":"2024-04-25T05:46:49.964254+0000","last_peered":"2024-04-25T05:46:49.964254+0000","last_clean":"2024-04-25T05:46:49.964254+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:46:49.964254+0000","last_undegraded":"2024-04-25T05:46:49.964254+0000","last_fullsized":"2024-04-25T05:46:49.964254+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":344,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067111+0000","last_change":"2024-04-25T05:46:44.669771+0000","last_active":"2024-04-25T05:46:53.067111+0000","last_peered":"2024-04-25T05:46:53.067111+0000","last_clean":"2024-04-25T05:46:53.067111+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:46:53.067111+0000","last_undegraded":"2024-04-25T05:46:53.067111+0000","last_fullsized":"2024-04-25T05:46:53.067111+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.669615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.669615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.090431+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":882,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.968029+0000","last_change":"2024-04-25T05:45:40.844494+0000","last_active":"2024-04-25T05:46:49.968029+0000","last_peered":"2024-04-25T05:46:49.968029+0000","last_clean":"2024-04-25T05:46:49.968029+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:46:49.968029+0000","last_undegraded":"2024-04-25T05:46:49.968029+0000","last_fullsized":"2024-04-25T05:46:49.968029+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:40.844354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:40.844354+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"395'1577","reported_seq":1603,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.936477+0000","last_change":"2024-04-25T05:46:37.986967+0000","last_active":"2024-04-25T05:46:52.936477+0000","last_peered":"2024-04-25T05:46:52.936477+0000","last_clean":"2024-04-25T05:46:52.936477+0000","last_became_active":"2024-04-25T05:46:37.986694+0000","last_became_peered":"2024-04-25T05:46:37.986694+0000","last_unstale":"2024-04-25T05:46:52.936477+0000","last_undegraded":"2024-04-25T05:46:52.936477+0000","last_fullsized":"2024-04-25T05:46:52.936477+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.428344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":4731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1577,"num_write_kb":100928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":457,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966307+0000","last_change":"2024-04-25T05:46:34.966272+0000","last_active":"2024-04-25T05:46:49.966307+0000","last_peered":"2024-04-25T05:46:49.966307+0000","last_clean":"2024-04-25T05:46:49.966307+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:46:49.966307+0000","last_undegraded":"2024-04-25T05:46:49.966307+0000","last_fullsized":"2024-04-25T05:46:49.966307+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:12.712241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:46:12.712241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.000979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":76,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965266+0000","last_change":"2024-04-25T05:46:36.996561+0000","last_active":"2024-04-25T05:46:49.965266+0000","last_peered":"2024-04-25T05:46:49.965266+0000","last_clean":"2024-04-25T05:46:49.965266+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:46:49.965266+0000","last_undegraded":"2024-04-25T05:46:49.965266+0000","last_fullsized":"2024-04-25T05:46:49.965266+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:46:07.457049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.457049+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:13.362844+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":736,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065569+0000","last_change":"2024-04-25T05:46:34.974950+0000","last_active":"2024-04-25T05:46:53.065569+0000","last_peered":"2024-04-25T05:46:53.065569+0000","last_clean":"2024-04-25T05:46:53.065569+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:46:53.065569+0000","last_undegraded":"2024-04-25T05:46:53.065569+0000","last_fullsized":"2024-04-25T05:46:53.065569+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":718,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065560+0000","last_change":"2024-04-25T05:46:47.388550+0000","last_active":"2024-04-25T05:46:53.065560+0000","last_peered":"2024-04-25T05:46:53.065560+0000","last_clean":"2024-04-25T05:46:53.065560+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:46:53.065560+0000","last_undegraded":"2024-04-25T05:46:53.065560+0000","last_fullsized":"2024-04-25T05:46:53.065560+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.388440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.388440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:09.383423+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":908,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.649935+0000","last_change":"2024-04-25T05:46:52.649991+0000","last_active":"2024-04-25T05:46:52.649935+0000","last_peered":"2024-04-25T05:46:52.649935+0000","last_clean":"2024-04-25T05:46:52.649935+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:46:52.649935+0000","last_undegraded":"2024-04-25T05:46:52.649935+0000","last_fullsized":"2024-04-25T05:46:52.649935+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:52.649850+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.649850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.101172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"396'1638","reported_seq":1664,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.123799+0000","last_change":"2024-04-25T05:46:37.986465+0000","last_active":"2024-04-25T05:46:53.123799+0000","last_peered":"2024-04-25T05:46:53.123799+0000","last_clean":"2024-04-25T05:46:53.123799+0000","last_became_active":"2024-04-25T05:46:37.986157+0000","last_became_peered":"2024-04-25T05:46:37.986157+0000","last_unstale":"2024-04-25T05:46:53.123799+0000","last_undegraded":"2024-04-25T05:46:53.123799+0000","last_fullsized":"2024-04-25T05:46:53.123799+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1638,"log_dups_size":0,"ondisk_log_size":1638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:03.585583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107347968,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1638,"num_write_kb":104832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966107+0000","last_change":"2024-04-25T05:46:34.986045+0000","last_active":"2024-04-25T05:46:49.966107+0000","last_peered":"2024-04-25T05:46:49.966107+0000","last_clean":"2024-04-25T05:46:49.966107+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:46:49.966107+0000","last_undegraded":"2024-04-25T05:46:49.966107+0000","last_fullsized":"2024-04-25T05:46:49.966107+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":44,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964697+0000","last_change":"2024-04-25T05:46:35.019569+0000","last_active":"2024-04-25T05:46:49.964697+0000","last_peered":"2024-04-25T05:46:49.964697+0000","last_clean":"2024-04-25T05:46:49.964697+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:46:49.964697+0000","last_undegraded":"2024-04-25T05:46:49.964697+0000","last_fullsized":"2024-04-25T05:46:49.964697+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1511,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:51.485356+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:46:51.485356+0000","last_peered":"2024-04-25T05:46:51.485356+0000","last_clean":"2024-04-25T05:46:51.485356+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:46:51.485356+0000","last_undegraded":"2024-04-25T05:46:51.485356+0000","last_fullsized":"2024-04-25T05:46:51.485356+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":756,"num_read_kb":751,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":329,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.956714+0000","last_change":"2024-04-25T05:46:34.967478+0000","last_active":"2024-04-25T05:46:49.956714+0000","last_peered":"2024-04-25T05:46:49.956714+0000","last_clean":"2024-04-25T05:46:49.956714+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:46:49.956714+0000","last_undegraded":"2024-04-25T05:46:49.956714+0000","last_fullsized":"2024-04-25T05:46:49.956714+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:30.434101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.434101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:52.955132+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":706,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065125+0000","last_change":"2024-04-25T05:46:34.975022+0000","last_active":"2024-04-25T05:46:53.065125+0000","last_peered":"2024-04-25T05:46:53.065125+0000","last_clean":"2024-04-25T05:46:53.065125+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:46:53.065125+0000","last_undegraded":"2024-04-25T05:46:53.065125+0000","last_fullsized":"2024-04-25T05:46:53.065125+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:36.702085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.702085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.632142+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":888,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.066055+0000","last_change":"2024-04-25T05:46:43.418452+0000","last_active":"2024-04-25T05:46:53.066055+0000","last_peered":"2024-04-25T05:46:53.066055+0000","last_clean":"2024-04-25T05:46:53.066055+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:46:53.066055+0000","last_undegraded":"2024-04-25T05:46:53.066055+0000","last_fullsized":"2024-04-25T05:46:53.066055+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:43.418024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.418024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.516553+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"395'1625","reported_seq":1650,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.928579+0000","last_change":"2024-04-25T05:46:37.985455+0000","last_active":"2024-04-25T05:46:52.928579+0000","last_peered":"2024-04-25T05:46:52.928579+0000","last_clean":"2024-04-25T05:46:52.928579+0000","last_became_active":"2024-04-25T05:46:37.985141+0000","last_became_peered":"2024-04-25T05:46:37.985141+0000","last_unstale":"2024-04-25T05:46:52.928579+0000","last_undegraded":"2024-04-25T05:46:52.928579+0000","last_fullsized":"2024-04-25T05:46:52.928579+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1625,"log_dups_size":0,"ondisk_log_size":1625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.374506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":4875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":104000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":726,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964583+0000","last_change":"2024-04-25T05:46:34.982419+0000","last_active":"2024-04-25T05:46:49.964583+0000","last_peered":"2024-04-25T05:46:49.964583+0000","last_clean":"2024-04-25T05:46:49.964583+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:46:49.964583+0000","last_undegraded":"2024-04-25T05:46:49.964583+0000","last_fullsized":"2024-04-25T05:46:49.964583+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":543,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965956+0000","last_change":"2024-04-25T05:45:38.816178+0000","last_active":"2024-04-25T05:46:49.965956+0000","last_peered":"2024-04-25T05:46:49.965956+0000","last_clean":"2024-04-25T05:46:49.965956+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:46:49.965956+0000","last_undegraded":"2024-04-25T05:46:49.965956+0000","last_fullsized":"2024-04-25T05:46:49.965956+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:45:38.815913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.815913+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:02.678590+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":722,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068228+0000","last_change":"2024-04-25T05:46:34.970673+0000","last_active":"2024-04-25T05:46:53.068228+0000","last_peered":"2024-04-25T05:46:53.068228+0000","last_clean":"2024-04-25T05:46:53.068228+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:46:53.068228+0000","last_undegraded":"2024-04-25T05:46:53.068228+0000","last_fullsized":"2024-04-25T05:46:53.068228+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.811914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.811914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:10.415552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1021,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963868+0000","last_change":"2024-04-25T05:46:06.483237+0000","last_active":"2024-04-25T05:46:49.963868+0000","last_peered":"2024-04-25T05:46:49.963868+0000","last_clean":"2024-04-25T05:46:49.963868+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:46:49.963868+0000","last_undegraded":"2024-04-25T05:46:49.963868+0000","last_fullsized":"2024-04-25T05:46:49.963868+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:06.483108+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.483108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:33.443264+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966019+0000","last_change":"2024-04-25T05:46:34.981187+0000","last_active":"2024-04-25T05:46:49.966019+0000","last_peered":"2024-04-25T05:46:49.966019+0000","last_clean":"2024-04-25T05:46:49.966019+0000","last_became_active":"2024-04-25T05:46:34.980440+0000","last_became_peered":"2024-04-25T05:46:34.980440+0000","last_unstale":"2024-04-25T05:46:49.966019+0000","last_undegraded":"2024-04-25T05:46:49.966019+0000","last_fullsized":"2024-04-25T05:46:49.966019+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:37.037435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.037435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"395'1602","reported_seq":1626,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.900452+0000","last_change":"2024-04-25T05:46:37.984020+0000","last_active":"2024-04-25T05:46:52.900452+0000","last_peered":"2024-04-25T05:46:52.900452+0000","last_clean":"2024-04-25T05:46:52.900452+0000","last_became_active":"2024-04-25T05:46:37.983253+0000","last_became_peered":"2024-04-25T05:46:37.983253+0000","last_unstale":"2024-04-25T05:46:52.900452+0000","last_undegraded":"2024-04-25T05:46:52.900452+0000","last_fullsized":"2024-04-25T05:46:52.900452+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1602,"log_dups_size":0,"ondisk_log_size":1602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:53.170282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":4806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1602,"num_write_kb":102528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.082462+0000","last_change":"2024-04-25T05:46:34.971368+0000","last_active":"2024-04-25T05:46:53.082462+0000","last_peered":"2024-04-25T05:46:53.082462+0000","last_clean":"2024-04-25T05:46:53.082462+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:46:53.082462+0000","last_undegraded":"2024-04-25T05:46:53.082462+0000","last_fullsized":"2024-04-25T05:46:53.082462+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":844,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065189+0000","last_change":"2024-04-25T05:46:35.010157+0000","last_active":"2024-04-25T05:46:53.065189+0000","last_peered":"2024-04-25T05:46:53.065189+0000","last_clean":"2024-04-25T05:46:53.065189+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:46:53.065189+0000","last_undegraded":"2024-04-25T05:46:53.065189+0000","last_fullsized":"2024-04-25T05:46:53.065189+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081559+0000","last_change":"2024-04-25T05:46:50.947463+0000","last_active":"2024-04-25T05:46:53.081559+0000","last_peered":"2024-04-25T05:46:53.081559+0000","last_clean":"2024-04-25T05:46:53.081559+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:46:53.081559+0000","last_undegraded":"2024-04-25T05:46:53.081559+0000","last_fullsized":"2024-04-25T05:46:53.081559+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.947345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.947345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:59.498121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":742,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.064939+0000","last_change":"2024-04-25T05:46:19.992163+0000","last_active":"2024-04-25T05:46:53.064939+0000","last_peered":"2024-04-25T05:46:53.064939+0000","last_clean":"2024-04-25T05:46:53.064939+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:46:53.064939+0000","last_undegraded":"2024-04-25T05:46:53.064939+0000","last_fullsized":"2024-04-25T05:46:53.064939+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":704,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964872+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:46:49.964872+0000","last_peered":"2024-04-25T05:46:49.964872+0000","last_clean":"2024-04-25T05:46:49.964872+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:46:49.964872+0000","last_undegraded":"2024-04-25T05:46:49.964872+0000","last_fullsized":"2024-04-25T05:46:49.964872+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1029,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964253+0000","last_change":"2024-04-25T05:46:23.137608+0000","last_active":"2024-04-25T05:46:49.964253+0000","last_peered":"2024-04-25T05:46:49.964253+0000","last_clean":"2024-04-25T05:46:49.964253+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:46:49.964253+0000","last_undegraded":"2024-04-25T05:46:49.964253+0000","last_fullsized":"2024-04-25T05:46:49.964253+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"395'1554","reported_seq":1580,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.913945+0000","last_change":"2024-04-25T05:46:37.985580+0000","last_active":"2024-04-25T05:46:52.913945+0000","last_peered":"2024-04-25T05:46:52.913945+0000","last_clean":"2024-04-25T05:46:52.913945+0000","last_became_active":"2024-04-25T05:46:37.985031+0000","last_became_peered":"2024-04-25T05:46:37.985031+0000","last_unstale":"2024-04-25T05:46:52.913945+0000","last_undegraded":"2024-04-25T05:46:52.913945+0000","last_fullsized":"2024-04-25T05:46:52.913945+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:45.128788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101842944,"num_objects":1554,"num_object_clones":0,"num_object_copies":4662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":99456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":331,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065551+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:46:53.065551+0000","last_peered":"2024-04-25T05:46:53.065551+0000","last_clean":"2024-04-25T05:46:53.065551+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:46:53.065551+0000","last_undegraded":"2024-04-25T05:46:53.065551+0000","last_fullsized":"2024-04-25T05:46:53.065551+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1006,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065677+0000","last_change":"2024-04-25T05:46:52.415241+0000","last_active":"2024-04-25T05:46:53.065677+0000","last_peered":"2024-04-25T05:46:53.065677+0000","last_clean":"2024-04-25T05:46:53.065677+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:46:53.065677+0000","last_undegraded":"2024-04-25T05:46:53.065677+0000","last_fullsized":"2024-04-25T05:46:53.065677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:46:52.415149+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.415149+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:12.643185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":499,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:50.698599+0000","last_change":"2024-04-25T05:46:50.698635+0000","last_active":"2024-04-25T05:46:50.698599+0000","last_peered":"2024-04-25T05:46:50.698599+0000","last_clean":"2024-04-25T05:46:50.698599+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:46:50.698599+0000","last_undegraded":"2024-04-25T05:46:50.698599+0000","last_fullsized":"2024-04-25T05:46:50.698599+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.698507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.698507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:59.959496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":42,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964997+0000","last_change":"2024-04-25T05:46:49.868732+0000","last_active":"2024-04-25T05:46:49.964997+0000","last_peered":"2024-04-25T05:46:49.964997+0000","last_clean":"2024-04-25T05:46:49.964997+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:46:49.964997+0000","last_undegraded":"2024-04-25T05:46:49.964997+0000","last_fullsized":"2024-04-25T05:46:49.964997+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:46:49.868648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.868648+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.090373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":685,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067573+0000","last_change":"2024-04-25T05:46:16.748067+0000","last_active":"2024-04-25T05:46:53.067573+0000","last_peered":"2024-04-25T05:46:53.067573+0000","last_clean":"2024-04-25T05:46:53.067573+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:46:53.067573+0000","last_undegraded":"2024-04-25T05:46:53.067573+0000","last_fullsized":"2024-04-25T05:46:53.067573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.747963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.747963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.649984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963253+0000","last_change":"2024-04-25T05:46:37.982952+0000","last_active":"2024-04-25T05:46:49.963253+0000","last_peered":"2024-04-25T05:46:49.963253+0000","last_clean":"2024-04-25T05:46:49.963253+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:46:49.963253+0000","last_undegraded":"2024-04-25T05:46:49.963253+0000","last_fullsized":"2024-04-25T05:46:49.963253+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:11.595125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:11.595125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:14.254842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"396'1659","reported_seq":1685,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.134348+0000","last_change":"2024-04-25T05:46:37.986363+0000","last_active":"2024-04-25T05:46:53.134348+0000","last_peered":"2024-04-25T05:46:53.134348+0000","last_clean":"2024-04-25T05:46:53.134348+0000","last_became_active":"2024-04-25T05:46:37.986037+0000","last_became_peered":"2024-04-25T05:46:37.986037+0000","last_unstale":"2024-04-25T05:46:53.134348+0000","last_undegraded":"2024-04-25T05:46:53.134348+0000","last_fullsized":"2024-04-25T05:46:53.134348+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1659,"log_dups_size":0,"ondisk_log_size":1659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:06.423561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1659,"num_write_kb":106176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":416,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065749+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:46:53.065749+0000","last_peered":"2024-04-25T05:46:53.065749+0000","last_clean":"2024-04-25T05:46:53.065749+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:46:53.065749+0000","last_undegraded":"2024-04-25T05:46:53.065749+0000","last_fullsized":"2024-04-25T05:46:53.065749+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963413+0000","last_change":"2024-04-25T05:46:34.977874+0000","last_active":"2024-04-25T05:46:49.963413+0000","last_peered":"2024-04-25T05:46:49.963413+0000","last_clean":"2024-04-25T05:46:49.963413+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:46:49.963413+0000","last_undegraded":"2024-04-25T05:46:49.963413+0000","last_fullsized":"2024-04-25T05:46:49.963413+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":48,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081245+0000","last_change":"2024-04-25T05:46:19.996324+0000","last_active":"2024-04-25T05:46:53.081245+0000","last_peered":"2024-04-25T05:46:53.081245+0000","last_clean":"2024-04-25T05:46:53.081245+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:46:53.081245+0000","last_undegraded":"2024-04-25T05:46:53.081245+0000","last_fullsized":"2024-04-25T05:46:53.081245+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":321,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965661+0000","last_change":"2024-04-25T05:46:19.989240+0000","last_active":"2024-04-25T05:46:49.965661+0000","last_peered":"2024-04-25T05:46:49.965661+0000","last_clean":"2024-04-25T05:46:49.965661+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:46:49.965661+0000","last_undegraded":"2024-04-25T05:46:49.965661+0000","last_fullsized":"2024-04-25T05:46:49.965661+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"396'1647","reported_seq":1675,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.780237+0000","last_change":"2024-04-25T05:46:37.979066+0000","last_active":"2024-04-25T05:46:53.780237+0000","last_peered":"2024-04-25T05:46:53.780237+0000","last_clean":"2024-04-25T05:46:53.780237+0000","last_became_active":"2024-04-25T05:46:37.978316+0000","last_became_peered":"2024-04-25T05:46:37.978316+0000","last_unstale":"2024-04-25T05:46:53.780237+0000","last_undegraded":"2024-04-25T05:46:53.780237+0000","last_fullsized":"2024-04-25T05:46:53.780237+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1647,"log_dups_size":0,"ondisk_log_size":1647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:49.031349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1647,"num_write_kb":105408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":473,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966103+0000","last_change":"2024-04-25T05:46:06.689035+0000","last_active":"2024-04-25T05:46:49.966103+0000","last_peered":"2024-04-25T05:46:49.966103+0000","last_clean":"2024-04-25T05:46:49.966103+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:46:49.966103+0000","last_undegraded":"2024-04-25T05:46:49.966103+0000","last_fullsized":"2024-04-25T05:46:49.966103+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'41","last_scrub_stamp":"2024-04-25T05:46:06.688883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.688883+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:19.872066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":757,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964691+0000","last_change":"2024-04-25T05:46:45.682570+0000","last_active":"2024-04-25T05:46:49.964691+0000","last_peered":"2024-04-25T05:46:49.964691+0000","last_clean":"2024-04-25T05:46:49.964691+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:46:49.964691+0000","last_undegraded":"2024-04-25T05:46:49.964691+0000","last_fullsized":"2024-04-25T05:46:49.964691+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:46:45.682441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.682441+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:12.440746+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":54,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081485+0000","last_change":"2024-04-25T05:46:34.972681+0000","last_active":"2024-04-25T05:46:53.081485+0000","last_peered":"2024-04-25T05:46:53.081485+0000","last_clean":"2024-04-25T05:46:53.081485+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:46:53.081485+0000","last_undegraded":"2024-04-25T05:46:53.081485+0000","last_fullsized":"2024-04-25T05:46:53.081485+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":703,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963554+0000","last_change":"2024-04-25T05:46:19.987220+0000","last_active":"2024-04-25T05:46:49.963554+0000","last_peered":"2024-04-25T05:46:49.963554+0000","last_clean":"2024-04-25T05:46:49.963554+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:46:49.963554+0000","last_undegraded":"2024-04-25T05:46:49.963554+0000","last_fullsized":"2024-04-25T05:46:49.963554+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"395'1584","reported_seq":1610,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.927934+0000","last_change":"2024-04-25T05:46:37.986817+0000","last_active":"2024-04-25T05:46:52.927934+0000","last_peered":"2024-04-25T05:46:52.927934+0000","last_clean":"2024-04-25T05:46:52.927934+0000","last_became_active":"2024-04-25T05:46:37.983489+0000","last_became_peered":"2024-04-25T05:46:37.983489+0000","last_unstale":"2024-04-25T05:46:52.927934+0000","last_undegraded":"2024-04-25T05:46:52.927934+0000","last_fullsized":"2024-04-25T05:46:52.927934+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.028685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1584,"num_write_kb":101376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081008+0000","last_change":"2024-04-25T05:46:34.971470+0000","last_active":"2024-04-25T05:46:53.081008+0000","last_peered":"2024-04-25T05:46:53.081008+0000","last_clean":"2024-04-25T05:46:53.081008+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:46:53.081008+0000","last_undegraded":"2024-04-25T05:46:53.081008+0000","last_fullsized":"2024-04-25T05:46:53.081008+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":355,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065136+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:46:53.065136+0000","last_peered":"2024-04-25T05:46:53.065136+0000","last_clean":"2024-04-25T05:46:53.065136+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:46:53.065136+0000","last_undegraded":"2024-04-25T05:46:53.065136+0000","last_fullsized":"2024-04-25T05:46:53.065136+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963960+0000","last_change":"2024-04-25T05:46:46.685480+0000","last_active":"2024-04-25T05:46:49.963960+0000","last_peered":"2024-04-25T05:46:49.963960+0000","last_clean":"2024-04-25T05:46:49.963960+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:46:49.963960+0000","last_undegraded":"2024-04-25T05:46:49.963960+0000","last_fullsized":"2024-04-25T05:46:49.963960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.685365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.685365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:57.792441+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":697,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068619+0000","last_change":"2024-04-25T05:46:52.779088+0000","last_active":"2024-04-25T05:46:53.068619+0000","last_peered":"2024-04-25T05:46:53.068619+0000","last_clean":"2024-04-25T05:46:53.068619+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:46:53.068619+0000","last_undegraded":"2024-04-25T05:46:53.068619+0000","last_fullsized":"2024-04-25T05:46:53.068619+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:52.779013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.779013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.137766+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"395'1607","reported_seq":1631,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.920502+0000","last_change":"2024-04-25T05:46:37.981240+0000","last_active":"2024-04-25T05:46:52.920502+0000","last_peered":"2024-04-25T05:46:52.920502+0000","last_clean":"2024-04-25T05:46:52.920502+0000","last_became_active":"2024-04-25T05:46:37.980913+0000","last_became_peered":"2024-04-25T05:46:37.980913+0000","last_unstale":"2024-04-25T05:46:52.920502+0000","last_undegraded":"2024-04-25T05:46:52.920502+0000","last_fullsized":"2024-04-25T05:46:52.920502+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":1607,"log_dups_size":0,"ondisk_log_size":1607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:43.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105316352,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1607,"num_write_kb":102848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":890,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965284+0000","last_change":"2024-04-25T05:46:47.674917+0000","last_active":"2024-04-25T05:46:49.965284+0000","last_peered":"2024-04-25T05:46:49.965284+0000","last_clean":"2024-04-25T05:46:49.965284+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:46:49.965284+0000","last_undegraded":"2024-04-25T05:46:49.965284+0000","last_fullsized":"2024-04-25T05:46:49.965284+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:46:47.674789+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.674789+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.201644+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":50,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964880+0000","last_change":"2024-04-25T05:46:35.009698+0000","last_active":"2024-04-25T05:46:49.964880+0000","last_peered":"2024-04-25T05:46:49.964880+0000","last_clean":"2024-04-25T05:46:49.964880+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:46:49.964880+0000","last_undegraded":"2024-04-25T05:46:49.964880+0000","last_fullsized":"2024-04-25T05:46:49.964880+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":751,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964238+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:46:49.964238+0000","last_peered":"2024-04-25T05:46:49.964238+0000","last_clean":"2024-04-25T05:46:49.964238+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:46:49.964238+0000","last_undegraded":"2024-04-25T05:46:49.964238+0000","last_fullsized":"2024-04-25T05:46:49.964238+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965067+0000","last_change":"2024-04-25T05:46:34.973794+0000","last_active":"2024-04-25T05:46:49.965067+0000","last_peered":"2024-04-25T05:46:49.965067+0000","last_clean":"2024-04-25T05:46:49.965067+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:46:49.965067+0000","last_undegraded":"2024-04-25T05:46:49.965067+0000","last_fullsized":"2024-04-25T05:46:49.965067+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:10.692888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.692888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:24.236919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":723,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068829+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:46:53.068829+0000","last_peered":"2024-04-25T05:46:53.068829+0000","last_clean":"2024-04-25T05:46:53.068829+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:46:53.068829+0000","last_undegraded":"2024-04-25T05:46:53.068829+0000","last_fullsized":"2024-04-25T05:46:53.068829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":457,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.962437+0000","last_change":"2024-04-25T05:46:47.655135+0000","last_active":"2024-04-25T05:46:49.962437+0000","last_peered":"2024-04-25T05:46:49.962437+0000","last_clean":"2024-04-25T05:46:49.962437+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:46:49.962437+0000","last_undegraded":"2024-04-25T05:46:49.962437+0000","last_fullsized":"2024-04-25T05:46:49.962437+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.655013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.655013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:48.222095+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964611+0000","last_change":"2024-04-25T05:46:34.988515+0000","last_active":"2024-04-25T05:46:49.964611+0000","last_peered":"2024-04-25T05:46:49.964611+0000","last_clean":"2024-04-25T05:46:49.964611+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:46:49.964611+0000","last_undegraded":"2024-04-25T05:46:49.964611+0000","last_fullsized":"2024-04-25T05:46:49.964611+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":703,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964365+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:46:49.964365+0000","last_peered":"2024-04-25T05:46:49.964365+0000","last_clean":"2024-04-25T05:46:49.964365+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:46:49.964365+0000","last_undegraded":"2024-04-25T05:46:49.964365+0000","last_fullsized":"2024-04-25T05:46:49.964365+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":412,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067582+0000","last_change":"2024-04-25T05:46:20.806968+0000","last_active":"2024-04-25T05:46:53.067582+0000","last_peered":"2024-04-25T05:46:53.067582+0000","last_clean":"2024-04-25T05:46:53.067582+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:46:53.067582+0000","last_undegraded":"2024-04-25T05:46:53.067582+0000","last_fullsized":"2024-04-25T05:46:53.067582+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:46:20.806869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:20.806869+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.735644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":721,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.966660+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:46:49.966660+0000","last_peered":"2024-04-25T05:46:49.966660+0000","last_clean":"2024-04-25T05:46:49.966660+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:46:49.966660+0000","last_undegraded":"2024-04-25T05:46:49.966660+0000","last_fullsized":"2024-04-25T05:46:49.966660+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":681,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068694+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:46:53.068694+0000","last_peered":"2024-04-25T05:46:53.068694+0000","last_clean":"2024-04-25T05:46:53.068694+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:46:53.068694+0000","last_undegraded":"2024-04-25T05:46:53.068694+0000","last_fullsized":"2024-04-25T05:46:53.068694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":48,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081961+0000","last_change":"2024-04-25T05:46:19.984536+0000","last_active":"2024-04-25T05:46:53.081961+0000","last_peered":"2024-04-25T05:46:53.081961+0000","last_clean":"2024-04-25T05:46:53.081961+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:46:53.081961+0000","last_undegraded":"2024-04-25T05:46:53.081961+0000","last_fullsized":"2024-04-25T05:46:53.081961+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.695090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.695090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:28.011559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":42,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963875+0000","last_change":"2024-04-25T05:46:35.006236+0000","last_active":"2024-04-25T05:46:49.963875+0000","last_peered":"2024-04-25T05:46:49.963875+0000","last_clean":"2024-04-25T05:46:49.963875+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:46:49.963875+0000","last_undegraded":"2024-04-25T05:46:49.963875+0000","last_fullsized":"2024-04-25T05:46:49.963875+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:37.317498+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.317498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:53.371921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":315,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963536+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:46:49.963536+0000","last_peered":"2024-04-25T05:46:49.963536+0000","last_clean":"2024-04-25T05:46:49.963536+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:46:49.963536+0000","last_undegraded":"2024-04-25T05:46:49.963536+0000","last_fullsized":"2024-04-25T05:46:49.963536+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":864,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068764+0000","last_change":"2024-04-25T05:46:20.022186+0000","last_active":"2024-04-25T05:46:53.068764+0000","last_peered":"2024-04-25T05:46:53.068764+0000","last_clean":"2024-04-25T05:46:53.068764+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:46:53.068764+0000","last_undegraded":"2024-04-25T05:46:53.068764+0000","last_fullsized":"2024-04-25T05:46:53.068764+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":54,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.082580+0000","last_change":"2024-04-25T05:46:34.972368+0000","last_active":"2024-04-25T05:46:53.082580+0000","last_peered":"2024-04-25T05:46:53.082580+0000","last_clean":"2024-04-25T05:46:53.082580+0000","last_became_active":"2024-04-25T05:46:34.971901+0000","last_became_peered":"2024-04-25T05:46:34.971901+0000","last_unstale":"2024-04-25T05:46:53.082580+0000","last_undegraded":"2024-04-25T05:46:53.082580+0000","last_fullsized":"2024-04-25T05:46:53.082580+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":747,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068513+0000","last_change":"2024-04-25T05:45:32.015466+0000","last_active":"2024-04-25T05:46:53.068513+0000","last_peered":"2024-04-25T05:46:53.068513+0000","last_clean":"2024-04-25T05:46:53.068513+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:46:53.068513+0000","last_undegraded":"2024-04-25T05:46:53.068513+0000","last_fullsized":"2024-04-25T05:46:53.068513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:45:32.015242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.015242+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":715,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964616+0000","last_change":"2024-04-25T05:46:19.985933+0000","last_active":"2024-04-25T05:46:49.964616+0000","last_peered":"2024-04-25T05:46:49.964616+0000","last_clean":"2024-04-25T05:46:49.964616+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:46:49.964616+0000","last_undegraded":"2024-04-25T05:46:49.964616+0000","last_fullsized":"2024-04-25T05:46:49.964616+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":37,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964065+0000","last_change":"2024-04-25T05:46:34.997195+0000","last_active":"2024-04-25T05:46:49.964065+0000","last_peered":"2024-04-25T05:46:49.964065+0000","last_clean":"2024-04-25T05:46:49.964065+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:46:49.964065+0000","last_undegraded":"2024-04-25T05:46:49.964065+0000","last_fullsized":"2024-04-25T05:46:49.964065+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:46:19.516621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:19.516621+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:31.276451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":315,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964884+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:46:49.964884+0000","last_peered":"2024-04-25T05:46:49.964884+0000","last_clean":"2024-04-25T05:46:49.964884+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:46:49.964884+0000","last_undegraded":"2024-04-25T05:46:49.964884+0000","last_fullsized":"2024-04-25T05:46:49.964884+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":841,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.956953+0000","last_change":"2024-04-25T05:46:36.988922+0000","last_active":"2024-04-25T05:46:49.956953+0000","last_peered":"2024-04-25T05:46:49.956953+0000","last_clean":"2024-04-25T05:46:49.956953+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:46:49.956953+0000","last_undegraded":"2024-04-25T05:46:49.956953+0000","last_fullsized":"2024-04-25T05:46:49.956953+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:46:13.591210+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:46:13.591210+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.574846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":382,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964227+0000","last_change":"2024-04-25T05:45:33.294954+0000","last_active":"2024-04-25T05:46:49.964227+0000","last_peered":"2024-04-25T05:46:49.964227+0000","last_clean":"2024-04-25T05:46:49.964227+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:46:49.964227+0000","last_undegraded":"2024-04-25T05:46:49.964227+0000","last_fullsized":"2024-04-25T05:46:49.964227+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:03.065653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":704,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.066233+0000","last_change":"2024-04-25T05:46:34.974010+0000","last_active":"2024-04-25T05:46:53.066233+0000","last_peered":"2024-04-25T05:46:53.066233+0000","last_clean":"2024-04-25T05:46:53.066233+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:46:53.066233+0000","last_undegraded":"2024-04-25T05:46:53.066233+0000","last_fullsized":"2024-04-25T05:46:53.066233+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":729,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.519326+0000","last_change":"2024-04-25T05:46:53.519364+0000","last_active":"2024-04-25T05:46:53.519326+0000","last_peered":"2024-04-25T05:46:53.519326+0000","last_clean":"2024-04-25T05:46:53.519326+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:46:53.519326+0000","last_undegraded":"2024-04-25T05:46:53.519326+0000","last_fullsized":"2024-04-25T05:46:53.519326+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.519223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.519223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:57.533661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":259,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.066745+0000","last_change":"2024-04-25T05:46:20.039638+0000","last_active":"2024-04-25T05:46:53.066745+0000","last_peered":"2024-04-25T05:46:53.066745+0000","last_clean":"2024-04-25T05:46:53.066745+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:46:53.066745+0000","last_undegraded":"2024-04-25T05:46:53.066745+0000","last_fullsized":"2024-04-25T05:46:53.066745+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:46:07.665439+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.665439+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.578540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964466+0000","last_change":"2024-04-25T05:46:44.682768+0000","last_active":"2024-04-25T05:46:49.964466+0000","last_peered":"2024-04-25T05:46:49.964466+0000","last_clean":"2024-04-25T05:46:49.964466+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:46:49.964466+0000","last_undegraded":"2024-04-25T05:46:49.964466+0000","last_fullsized":"2024-04-25T05:46:49.964466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.682645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.682645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:10.110610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":705,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068423+0000","last_change":"2024-04-25T05:46:34.970569+0000","last_active":"2024-04-25T05:46:53.068423+0000","last_peered":"2024-04-25T05:46:53.068423+0000","last_clean":"2024-04-25T05:46:53.068423+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:46:53.068423+0000","last_undegraded":"2024-04-25T05:46:53.068423+0000","last_fullsized":"2024-04-25T05:46:53.068423+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964684+0000","last_change":"2024-04-25T05:46:34.974440+0000","last_active":"2024-04-25T05:46:49.964684+0000","last_peered":"2024-04-25T05:46:49.964684+0000","last_clean":"2024-04-25T05:46:49.964684+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:46:49.964684+0000","last_undegraded":"2024-04-25T05:46:49.964684+0000","last_fullsized":"2024-04-25T05:46:49.964684+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:51.491191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:51.491191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":776,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067556+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:46:53.067556+0000","last_peered":"2024-04-25T05:46:53.067556+0000","last_clean":"2024-04-25T05:46:53.067556+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:46:53.067556+0000","last_undegraded":"2024-04-25T05:46:53.067556+0000","last_fullsized":"2024-04-25T05:46:53.067556+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964433+0000","last_change":"2024-04-25T05:46:34.970402+0000","last_active":"2024-04-25T05:46:49.964433+0000","last_peered":"2024-04-25T05:46:49.964433+0000","last_clean":"2024-04-25T05:46:49.964433+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:46:49.964433+0000","last_undegraded":"2024-04-25T05:46:49.964433+0000","last_fullsized":"2024-04-25T05:46:49.964433+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":713,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964187+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:46:49.964187+0000","last_peered":"2024-04-25T05:46:49.964187+0000","last_clean":"2024-04-25T05:46:49.964187+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:46:49.964187+0000","last_undegraded":"2024-04-25T05:46:49.964187+0000","last_fullsized":"2024-04-25T05:46:49.964187+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":721,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068730+0000","last_change":"2024-04-25T05:46:19.986880+0000","last_active":"2024-04-25T05:46:53.068730+0000","last_peered":"2024-04-25T05:46:53.068730+0000","last_clean":"2024-04-25T05:46:53.068730+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:46:53.068730+0000","last_undegraded":"2024-04-25T05:46:53.068730+0000","last_fullsized":"2024-04-25T05:46:53.068730+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:17.841621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:46:17.841621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.909197+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":729,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067275+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:46:53.067275+0000","last_peered":"2024-04-25T05:46:53.067275+0000","last_clean":"2024-04-25T05:46:53.067275+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:46:53.067275+0000","last_undegraded":"2024-04-25T05:46:53.067275+0000","last_fullsized":"2024-04-25T05:46:53.067275+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965779+0000","last_change":"2024-04-25T05:46:34.978621+0000","last_active":"2024-04-25T05:46:49.965779+0000","last_peered":"2024-04-25T05:46:49.965779+0000","last_clean":"2024-04-25T05:46:49.965779+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:46:49.965779+0000","last_undegraded":"2024-04-25T05:46:49.965779+0000","last_fullsized":"2024-04-25T05:46:49.965779+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":419,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.961584+0000","last_change":"2024-04-25T05:45:29.823062+0000","last_active":"2024-04-25T05:46:49.961584+0000","last_peered":"2024-04-25T05:46:49.961584+0000","last_clean":"2024-04-25T05:46:49.961584+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:46:49.961584+0000","last_undegraded":"2024-04-25T05:46:49.961584+0000","last_fullsized":"2024-04-25T05:46:49.961584+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.822937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.473503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":712,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964318+0000","last_change":"2024-04-25T05:46:20.027611+0000","last_active":"2024-04-25T05:46:49.964318+0000","last_peered":"2024-04-25T05:46:49.964318+0000","last_clean":"2024-04-25T05:46:49.964318+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:46:49.964318+0000","last_undegraded":"2024-04-25T05:46:49.964318+0000","last_fullsized":"2024-04-25T05:46:49.964318+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":539,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.968331+0000","last_change":"2024-04-25T05:46:44.699964+0000","last_active":"2024-04-25T05:46:49.968331+0000","last_peered":"2024-04-25T05:46:49.968331+0000","last_clean":"2024-04-25T05:46:49.968331+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:46:49.968331+0000","last_undegraded":"2024-04-25T05:46:49.968331+0000","last_fullsized":"2024-04-25T05:46:49.968331+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:46:44.699808+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.699808+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:10.283576+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":721,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067114+0000","last_change":"2024-04-25T05:46:19.997617+0000","last_active":"2024-04-25T05:46:53.067114+0000","last_peered":"2024-04-25T05:46:53.067114+0000","last_clean":"2024-04-25T05:46:53.067114+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:46:53.067114+0000","last_undegraded":"2024-04-25T05:46:53.067114+0000","last_fullsized":"2024-04-25T05:46:53.067114+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":321,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964509+0000","last_change":"2024-04-25T05:46:19.988527+0000","last_active":"2024-04-25T05:46:49.964509+0000","last_peered":"2024-04-25T05:46:49.964509+0000","last_clean":"2024-04-25T05:46:49.964509+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:46:49.964509+0000","last_undegraded":"2024-04-25T05:46:49.964509+0000","last_fullsized":"2024-04-25T05:46:49.964509+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":703,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065205+0000","last_change":"2024-04-25T05:46:34.974816+0000","last_active":"2024-04-25T05:46:53.065205+0000","last_peered":"2024-04-25T05:46:53.065205+0000","last_clean":"2024-04-25T05:46:53.065205+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:46:53.065205+0000","last_undegraded":"2024-04-25T05:46:53.065205+0000","last_fullsized":"2024-04-25T05:46:53.065205+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":388,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964661+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:46:49.964661+0000","last_peered":"2024-04-25T05:46:49.964661+0000","last_clean":"2024-04-25T05:46:49.964661+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:46:49.964661+0000","last_undegraded":"2024-04-25T05:46:49.964661+0000","last_fullsized":"2024-04-25T05:46:49.964661+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":706,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065247+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:46:53.065247+0000","last_peered":"2024-04-25T05:46:53.065247+0000","last_clean":"2024-04-25T05:46:53.065247+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:46:53.065247+0000","last_undegraded":"2024-04-25T05:46:53.065247+0000","last_fullsized":"2024-04-25T05:46:53.065247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":519,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.967736+0000","last_change":"2024-04-25T05:46:15.701376+0000","last_active":"2024-04-25T05:46:49.967736+0000","last_peered":"2024-04-25T05:46:49.967736+0000","last_clean":"2024-04-25T05:46:49.967736+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:46:49.967736+0000","last_undegraded":"2024-04-25T05:46:49.967736+0000","last_fullsized":"2024-04-25T05:46:49.967736+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:46:15.701250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:15.701250+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:30.567412+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964477+0000","last_change":"2024-04-25T05:46:34.991626+0000","last_active":"2024-04-25T05:46:49.964477+0000","last_peered":"2024-04-25T05:46:49.964477+0000","last_clean":"2024-04-25T05:46:49.964477+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:46:49.964477+0000","last_undegraded":"2024-04-25T05:46:49.964477+0000","last_fullsized":"2024-04-25T05:46:49.964477+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:46:09.493093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.493093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:27.723926+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":729,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068214+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:46:53.068214+0000","last_peered":"2024-04-25T05:46:53.068214+0000","last_clean":"2024-04-25T05:46:53.068214+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:46:53.068214+0000","last_undegraded":"2024-04-25T05:46:53.068214+0000","last_fullsized":"2024-04-25T05:46:53.068214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":737,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.957686+0000","last_change":"2024-04-25T05:46:44.693255+0000","last_active":"2024-04-25T05:46:49.957686+0000","last_peered":"2024-04-25T05:46:49.957686+0000","last_clean":"2024-04-25T05:46:49.957686+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:46:49.957686+0000","last_undegraded":"2024-04-25T05:46:49.957686+0000","last_fullsized":"2024-04-25T05:46:49.957686+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.693144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.693144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.099880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":388,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068668+0000","last_change":"2024-04-25T05:46:47.610533+0000","last_active":"2024-04-25T05:46:53.068668+0000","last_peered":"2024-04-25T05:46:53.068668+0000","last_clean":"2024-04-25T05:46:53.068668+0000","last_became_active":"2024-04-25T05:46:20.009022+0000","last_became_peered":"2024-04-25T05:46:20.009022+0000","last_unstale":"2024-04-25T05:46:53.068668+0000","last_undegraded":"2024-04-25T05:46:53.068668+0000","last_fullsized":"2024-04-25T05:46:53.068668+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.610387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.610387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.887722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":710,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068773+0000","last_change":"2024-04-25T05:46:49.931810+0000","last_active":"2024-04-25T05:46:53.068773+0000","last_peered":"2024-04-25T05:46:53.068773+0000","last_clean":"2024-04-25T05:46:53.068773+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:46:53.068773+0000","last_undegraded":"2024-04-25T05:46:53.068773+0000","last_fullsized":"2024-04-25T05:46:53.068773+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:49.930039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.930039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.913953+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":64,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.081695+0000","last_change":"2024-04-25T05:46:20.037905+0000","last_active":"2024-04-25T05:46:53.081695+0000","last_peered":"2024-04-25T05:46:53.081695+0000","last_clean":"2024-04-25T05:46:53.081695+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:46:53.081695+0000","last_undegraded":"2024-04-25T05:46:53.081695+0000","last_fullsized":"2024-04-25T05:46:53.081695+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":190,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.964831+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:46:49.964831+0000","last_peered":"2024-04-25T05:46:49.964831+0000","last_clean":"2024-04-25T05:46:49.964831+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:46:49.964831+0000","last_undegraded":"2024-04-25T05:46:49.964831+0000","last_fullsized":"2024-04-25T05:46:49.964831+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":721,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065441+0000","last_change":"2024-04-25T05:46:34.974900+0000","last_active":"2024-04-25T05:46:53.065441+0000","last_peered":"2024-04-25T05:46:53.065441+0000","last_clean":"2024-04-25T05:46:53.065441+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:46:53.065441+0000","last_undegraded":"2024-04-25T05:46:53.065441+0000","last_fullsized":"2024-04-25T05:46:53.065441+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":732,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.960249+0000","last_change":"2024-04-25T05:46:43.757487+0000","last_active":"2024-04-25T05:46:49.960249+0000","last_peered":"2024-04-25T05:46:49.960249+0000","last_clean":"2024-04-25T05:46:49.960249+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:46:49.960249+0000","last_undegraded":"2024-04-25T05:46:49.960249+0000","last_fullsized":"2024-04-25T05:46:49.960249+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:43.757378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.757378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.378972+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":447,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068535+0000","last_change":"2024-04-25T05:46:05.752045+0000","last_active":"2024-04-25T05:46:53.068535+0000","last_peered":"2024-04-25T05:46:53.068535+0000","last_clean":"2024-04-25T05:46:53.068535+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:46:53.068535+0000","last_undegraded":"2024-04-25T05:46:53.068535+0000","last_fullsized":"2024-04-25T05:46:53.068535+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:46:05.751904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:05.751904+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.203255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963502+0000","last_change":"2024-04-25T05:46:34.989054+0000","last_active":"2024-04-25T05:46:49.963502+0000","last_peered":"2024-04-25T05:46:49.963502+0000","last_clean":"2024-04-25T05:46:49.963502+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:46:49.963502+0000","last_undegraded":"2024-04-25T05:46:49.963502+0000","last_fullsized":"2024-04-25T05:46:49.963502+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:00.435460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:45:00.435460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":701,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963705+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:46:49.963705+0000","last_peered":"2024-04-25T05:46:49.963705+0000","last_clean":"2024-04-25T05:46:49.963705+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:46:49.963705+0000","last_undegraded":"2024-04-25T05:46:49.963705+0000","last_fullsized":"2024-04-25T05:46:49.963705+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:54.988841+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963354+0000","last_change":"2024-04-25T05:46:34.985777+0000","last_active":"2024-04-25T05:46:49.963354+0000","last_peered":"2024-04-25T05:46:49.963354+0000","last_clean":"2024-04-25T05:46:49.963354+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:46:49.963354+0000","last_undegraded":"2024-04-25T05:46:49.963354+0000","last_fullsized":"2024-04-25T05:46:49.963354+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:38.767074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.767074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:46:58.308074+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":504,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068316+0000","last_change":"2024-04-25T05:46:20.046230+0000","last_active":"2024-04-25T05:46:53.068316+0000","last_peered":"2024-04-25T05:46:53.068316+0000","last_clean":"2024-04-25T05:46:53.068316+0000","last_became_active":"2024-04-25T05:46:20.009817+0000","last_became_peered":"2024-04-25T05:46:20.009817+0000","last_unstale":"2024-04-25T05:46:53.068316+0000","last_undegraded":"2024-04-25T05:46:53.068316+0000","last_fullsized":"2024-04-25T05:46:53.068316+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'28","last_scrub_stamp":"2024-04-25T05:46:10.858811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.858811+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:26.610333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.963967+0000","last_change":"2024-04-25T05:46:34.979633+0000","last_active":"2024-04-25T05:46:49.963967+0000","last_peered":"2024-04-25T05:46:49.963967+0000","last_clean":"2024-04-25T05:46:49.963967+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:46:49.963967+0000","last_undegraded":"2024-04-25T05:46:49.963967+0000","last_fullsized":"2024-04-25T05:46:49.963967+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965258+0000","last_change":"2024-04-25T05:46:34.979661+0000","last_active":"2024-04-25T05:46:49.965258+0000","last_peered":"2024-04-25T05:46:49.965258+0000","last_clean":"2024-04-25T05:46:49.965258+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:46:49.965258+0000","last_undegraded":"2024-04-25T05:46:49.965258+0000","last_fullsized":"2024-04-25T05:46:49.965258+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":323,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065988+0000","last_change":"2024-04-25T05:46:19.991930+0000","last_active":"2024-04-25T05:46:53.065988+0000","last_peered":"2024-04-25T05:46:53.065988+0000","last_clean":"2024-04-25T05:46:53.065988+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:46:53.065988+0000","last_undegraded":"2024-04-25T05:46:53.065988+0000","last_fullsized":"2024-04-25T05:46:53.065988+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:49.433748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:49.433748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":994,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067716+0000","last_change":"2024-04-25T05:46:48.610335+0000","last_active":"2024-04-25T05:46:53.067716+0000","last_peered":"2024-04-25T05:46:53.067716+0000","last_clean":"2024-04-25T05:46:53.067716+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:46:53.067716+0000","last_undegraded":"2024-04-25T05:46:53.067716+0000","last_fullsized":"2024-04-25T05:46:53.067716+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:46:48.610157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.610157+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.926651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":722,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.067381+0000","last_change":"2024-04-25T05:46:19.997604+0000","last_active":"2024-04-25T05:46:53.067381+0000","last_peered":"2024-04-25T05:46:53.067381+0000","last_clean":"2024-04-25T05:46:53.067381+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:46:53.067381+0000","last_undegraded":"2024-04-25T05:46:53.067381+0000","last_fullsized":"2024-04-25T05:46:53.067381+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":670,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.962711+0000","last_change":"2024-04-25T05:45:32.772720+0000","last_active":"2024-04-25T05:46:49.962711+0000","last_peered":"2024-04-25T05:46:49.962711+0000","last_clean":"2024-04-25T05:46:49.962711+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:46:49.962711+0000","last_undegraded":"2024-04-25T05:46:49.962711+0000","last_fullsized":"2024-04-25T05:46:49.962711+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.772481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.772481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":697,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.068512+0000","last_change":"2024-04-25T05:46:46.830806+0000","last_active":"2024-04-25T05:46:53.068512+0000","last_peered":"2024-04-25T05:46:53.068512+0000","last_clean":"2024-04-25T05:46:53.068512+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:46:53.068512+0000","last_undegraded":"2024-04-25T05:46:53.068512+0000","last_fullsized":"2024-04-25T05:46:53.068512+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.830676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.830676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.793071+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":793,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:52.599869+0000","last_change":"2024-04-25T05:46:52.599923+0000","last_active":"2024-04-25T05:46:52.599869+0000","last_peered":"2024-04-25T05:46:52.599869+0000","last_clean":"2024-04-25T05:46:52.599869+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:46:52.599869+0000","last_undegraded":"2024-04-25T05:46:52.599869+0000","last_fullsized":"2024-04-25T05:46:52.599869+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:46:52.599775+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.599775+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:52.650570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":38,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965627+0000","last_change":"2024-04-25T05:46:44.971898+0000","last_active":"2024-04-25T05:46:49.965627+0000","last_peered":"2024-04-25T05:46:49.965627+0000","last_clean":"2024-04-25T05:46:49.965627+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:46:49.965627+0000","last_undegraded":"2024-04-25T05:46:49.965627+0000","last_fullsized":"2024-04-25T05:46:49.965627+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.971755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.971755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:02.005304+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":715,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-04-25T05:46:53.065444+0000","last_change":"2024-04-25T05:46:34.974443+0000","last_active":"2024-04-25T05:46:53.065444+0000","last_peered":"2024-04-25T05:46:53.065444+0000","last_clean":"2024-04-25T05:46:53.065444+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:46:53.065444+0000","last_undegraded":"2024-04-25T05:46:53.065444+0000","last_fullsized":"2024-04-25T05:46:53.065444+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-04-25T05:46:49.965172+0000","last_change":"2024-04-25T05:46:34.988586+0000","last_active":"2024-04-25T05:46:49.965172+0000","last_peered":"2024-04-25T05:46:49.965172+0000","last_clean":"2024-04-25T05:46:49.965172+0000","last_became_active":"2024-04-25T05:46:34.986413+0000","last_became_peered":"2024-04-25T05:46:34.986413+0000","last_unstale":"2024-04-25T05:46:49.965172+0000","last_undegraded":"2024-04-25T05:46:49.965172+0000","last_fullsized":"2024-04-25T05:46:49.965172+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:39.734373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.734373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.578868+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1702297600,"num_objects":25975,"num_object_clones":0,"num_object_copies":77925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25975,"num_write_kb":1662400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5141954560,"data_stored":5141954560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25975,"ondisk_log_size":25975,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":756,"num_read_kb":751,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1394,"num_read_kb":1119,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":10814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":958,"ondisk_log_size":958,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070624,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157104,"kb_used_data":516456,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92588048,"statfs":{"total":95995035648,"available":94810161152,"internally_reserved":0,"allocated":528850944,"data_stored":528277902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364964,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207748,"kb_used_data":699004,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92537404,"statfs":{"total":95995035648,"available":94758301696,"internally_reserved":0,"allocated":715780096,"data_stored":715250526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382732,"kb_used_data":639360,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92362420,"statfs":{"total":95995035648,"available":94579118080,"internally_reserved":0,"allocated":654704640,"data_stored":654112812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738563,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411512,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124636,"kb_used_data":718612,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92620516,"statfs":{"total":95995035648,"available":94843408384,"internally_reserved":0,"allocated":735858688,"data_stored":735304588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1741116,"kb_used_data":849200,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92004036,"statfs":{"total":95995035648,"available":94212132864,"internally_reserved":0,"allocated":869580800,"data_stored":869005633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":913308103},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986349,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381664,"kb_used_data":904856,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":92363488,"statfs":{"total":95995035648,"available":94580211712,"internally_reserved":0,"allocated":926572544,"data_stored":926014484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692009,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004752,"kb_used_data":612616,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92740400,"statfs":{"total":95995035648,"available":94966169600,"internally_reserved":0,"allocated":627318784,"data_stored":626777056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":669450240,"data_stored":669450240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":745996288,"data_stored":745996288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":881721344,"data_stored":881721344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":945946624,"data_stored":945946624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":629866496,"data_stored":629866496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":729546752,"data_stored":729546752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":539426816,"data_stored":539426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:46:56.446 INFO:tasks.thrashosds.thrasher:Setting 6.7 to [1, 2, 4] 2024-04-25T05:46:56.446 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.7', '1', '2', '4'] 2024-04-25T05:46:56.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 6.7 1 2 4 2024-04-25T05:46:56.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:46:56.517 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.532 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:56.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:46:56.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:56.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:56.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:56.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 32576 32560 135.652 112.5 0.0148735 0.00736584 2024-04-25T05:46:56.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:56.967 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:56.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:57.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:57.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:33120 10.c 10:32f813c1:::benchmark_data_smithi043_152558_object33119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:57.214310+0000", 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.001348199, 2024-04-25T05:46:57.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001385746, 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 33120 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214310+0000", 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214310+0000", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214311+0000", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2240000000000001e-06 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214358+0000", 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6857999999999998e-05 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214369+0000", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0552e-05 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214377+0000", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1669999999999999e-06 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214425+0000", 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7766000000000003e-05 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214571+0000", 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014609000000000001 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,4", 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:57.214737+0000", 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016644300000000001 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 1 2024-04-25T05:46:57.220 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:57.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:46:57.327 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:46:57.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:57 smithi043 ceph-mon[27010]: pgmap v694: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-25T05:46:57.344 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:57 smithi043 ceph-mon[37492]: pgmap v694: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-25T05:46:57.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.395 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:57.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:57.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:57 smithi060 ceph-mon[30898]: pgmap v694: 158 pgs: 158 active+clean; 1.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-25T05:46:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:57.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:57.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:57.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:57.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:57.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:57.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:57.923 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 34225 34209 133.615 103.062 0.012076 0.00747849 2024-04-25T05:46:57.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:57.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:58.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:58.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:58.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:58.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:46:58.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:46:58.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:58.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:58.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:58.477 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:58.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:46:58.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:46:58.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:46:58.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:46:58.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:58.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:46:58.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 35860 35844 131.766 102.188 0.00357649 0.00758458 2024-04-25T05:46:58.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:59.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:59.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:59.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:46:59.242 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:46:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:59.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:46:59.382 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:46:59 smithi060 ceph-mon[30898]: pgmap v695: 158 pgs: 158 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 641 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T05:46:59.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:46:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:46:59 smithi043 ceph-mon[37492]: pgmap v695: 158 pgs: 158 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 641 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T05:46:59.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:46:59 smithi043 ceph-mon[27010]: pgmap v695: 158 pgs: 158 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 641 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-04-25T05:46:59.397 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:46:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:46:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:46:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:46:59.397 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:46:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:46:59.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:59.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:59.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:46:59.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:46:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:46:59.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:46:59.836 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:46:59.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 37525 37509 130.226 104.062 0.00560984 0.00767288 2024-04-25T05:46:59.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:46:59.943 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:46:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:46:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:46:59.976 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:46:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:46:59.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:00.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.114 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:00.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:00.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:47:00.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.289 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:00.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:00.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:00.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:00.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:00.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:00.608 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:47:00.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:00.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:00.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:00.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:00.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 39144 39128 128.697 101.188 0.00990719 0.00776361 2024-04-25T05:47:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:00.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:00.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:00.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:01.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:01.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:01.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[37492]: pgmap v696: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-04-25T05:47:01.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4213351952' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.129 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:01.149 INFO:teuthology.orchestra.run.smithi043.stderr:set 6.7 pg_upmap mapping to [1,2,4] 2024-04-25T05:47:01.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:01.269 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:01.286 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228018 8.17 8.878e0737 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 259.30733909600002, 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0050960120000001, 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.287 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228018 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975924+0000", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.711e-06 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975930+0000", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1469999999999998e-06 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975947+0000", 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7601999999999999e-05 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975956+0000", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9400000000000008e-06 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976889+0000", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093282800000000002 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976898+0000", 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8769999999999999e-06 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980999+0000", 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.00410116 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981018+0000", 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8746999999999999e-05 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228040 8.17 8.3c35da57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:47:01.291 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 259.28988480599998, 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98798208499999995, 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228040 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993377+0000", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.012e-06 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993383+0000", 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9769999999999997e-06 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993394+0000", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0370999999999999e-05 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993402+0000", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1729999999999995e-06 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993458+0000", 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6025e-05 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993465+0000", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2339999999999997e-06 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981341+0000", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98787640899999996 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981358+0000", 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6884e-05 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.295 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228041 8.17 8.7bebe17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 259.28972949799999, 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98792126800000002, 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228041 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.296 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993532+0000", 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8400000000000005e-07 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993535+0000", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6070000000000002e-06 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993542+0000", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9619999999999996e-06 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993548+0000", 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8190000000000003e-06 2024-04-25T05:47:01.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993585+0000", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6328000000000002e-05 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993588+0000", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4319999999999999e-06 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981443+0000", 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98785475499999997 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981453+0000", 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8810000000000008e-06 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:6 9.7 9:e9b4c7c6:::benchmark_data_smithi043_129820_object5:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 226.08271162099999, 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.006128809, 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200550+0000", 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3300000000000002e-07 2024-04-25T05:47:01.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200583+0000", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2659e-05 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200590+0000", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9890000000000003e-06 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200595+0000", 2024-04-25T05:47:01.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7899999999999996e-06 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200646+0000", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0822000000000002e-05 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200656+0000", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5109999999999992e-06 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192142+0000", 2024-04-25T05:47:01.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99148672699999996 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192281+0000", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139018 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192427+0000", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014579999999999999 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.194168+0000", 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017404429999999999 2024-04-25T05:47:01.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205031+0000", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010862966 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206577+0000", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015465780000000001 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206635+0000", 2024-04-25T05:47:01.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7998000000000003e-05 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206678+0000", 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2975000000000001e-05 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:12 9.d 9:b543e130:::benchmark_data_smithi043_129820_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 226.08255787100001, 2024-04-25T05:47:01.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0180756520000001, 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200704+0000", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2099999999999997e-07 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212291+0000", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011587573 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212312+0000", 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0627000000000001e-05 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212324+0000", 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.185e-05 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212430+0000", 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010610199999999999 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212444+0000", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4188999999999999e-05 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205046+0000", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99260219400000005 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205178+0000", 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013207299999999999 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205330+0000", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000151438 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.209820+0000", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044903390000000003 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210175+0000", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00035483800000000001 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.312 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218678+0000", 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0085029509999999999 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218737+0000", 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9036000000000002e-05 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218779+0000", 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1921000000000001e-05 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:20 9.d 9:b555abe5:::benchmark_data_smithi043_129820_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 226.068520237, 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0041661829999999, 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 20 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2600000000000002e-07 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214807+0000", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5287e-05 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214816+0000", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7630000000000006e-06 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214821+0000", 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214856+0000", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5147000000000001e-05 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214861+0000", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205852+0000", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99099027500000003 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.317 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206034+0000", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000182628 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206177+0000", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014306599999999999 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210096+0000", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039190930000000002 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210182+0000", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6057000000000006e-05 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218819+0000", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086365039999999997 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218868+0000", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9302000000000003e-05 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218907+0000", 2024-04-25T05:47:01.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8674999999999998e-05 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:34 9.7 9:e321d650:::benchmark_data_smithi043_129820_object33:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 226.06442023700001, 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.988632811, 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.320 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 34 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4099999999999998e-07 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218902+0000", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0786999999999999e-05 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218912+0000", 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4930000000000004e-06 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218920+0000", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1890000000000007e-06 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218967+0000", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6733000000000001e-05 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218984+0000", 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7153999999999999e-05 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.323 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192758+0000", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97377451299999995 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192893+0000", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013458900000000001 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.193067+0000", 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017409899999999999 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.195263+0000", 2024-04-25T05:47:01.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021959700000000002 2024-04-25T05:47:01.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.325 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:01 smithi060 ceph-mon[30898]: pgmap v696: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-04-25T05:47:01.325 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4213351952' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.325 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:01 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206451+0000", 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011187704 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207381+0000", 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093041 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207433+0000", 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2145000000000003e-05 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207474+0000", 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0284000000000003e-05 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122647 9.9 9:946f8a80:::benchmark_data_smithi043_129820_object1333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:47:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 135.01495453999999, 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98571332099999998, 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122647 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268307+0000", 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5600000000000004e-07 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.329 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268313+0000", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9560000000000002e-06 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268323+0000", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3479999999999993e-06 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268330+0000", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.861e-06 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.330 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268387+0000", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7014e-05 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268396+0000", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.842e-06 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249234+0000", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98083744799999995 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249334+0000", 2024-04-25T05:47:01.331 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9856000000000001e-05 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249425+0000", 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1697e-05 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253800+0000", 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043740819999999996 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.332 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253847+0000", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7231999999999998e-05 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253904+0000", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7262000000000002e-05 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253938+0000", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4394000000000003e-05 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254020+0000", 2024-04-25T05:47:01.333 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1372999999999995e-05 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122655 9.9 9:91969b48:::benchmark_data_smithi043_129820_object1341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 135.010628637, 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98173263399999999, 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.334 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122655 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.335 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272633+0000", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.39e-07 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272636+0000", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4369999999999999e-06 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272641+0000", 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0069999999999997e-06 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.336 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272644+0000", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1300000000000001e-06 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272681+0000", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7063000000000001e-05 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272687+0000", 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2059999999999998e-06 2024-04-25T05:47:01.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251540+0000", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97885334999999996 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251625+0000", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5487000000000003e-05 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251692+0000", 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6076999999999998e-05 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.338 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254167+0000", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002475929 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254218+0000", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1023999999999998e-05 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254276+0000", 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7348e-05 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.339 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254326+0000", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0139999999999998e-05 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254365+0000", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8996999999999999e-05 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122657 9.9 9:97065b42:::benchmark_data_smithi043_129820_object1343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:47:01.340 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 135.01015987299999, 2024-04-25T05:47:01.341 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98447252200000002, 2024-04-25T05:47:01.341 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.341 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.341 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.341 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122657 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:47:01.342 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273102+0000", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1399999999999995e-07 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273106+0000", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4240000000000003e-06 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273116+0000", 2024-04-25T05:47:01.343 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1109999999999997e-06 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273123+0000", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2089999999999999e-06 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273170+0000", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7475999999999997e-05 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273178+0000", 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5789999999999998e-06 2024-04-25T05:47:01.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252726+0000", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97954808900000001 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252818+0000", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2471000000000003e-05 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252938+0000", 2024-04-25T05:47:01.345 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119312 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254376+0000", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001438745 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254418+0000", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1127000000000001e-05 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257471+0000", 2024-04-25T05:47:01.346 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003053596 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257528+0000", 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7151000000000002e-05 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257574+0000", 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5318000000000002e-05 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234262 9.13 9:c8a64bff:::benchmark_data_smithi043_129820_object112948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 47.863738282, 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044775727, 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234262 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419524+0000", 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.114e-06 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.349 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419529+0000", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1320000000000002e-06 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419541+0000", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2517000000000001e-05 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419551+0000", 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.55e-06 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419626+0000", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5401999999999994e-05 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419639+0000", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3045e-05 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458481+0000", 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0388411980000001 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.351 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458596+0000", 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114869 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458847+0000", 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000251838 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.352 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463718+0000", 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048708349999999996 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463779+0000", 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0671000000000002e-05 2024-04-25T05:47:01.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464065+0000", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028592400000000001 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464124+0000", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8913000000000003e-05 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464180+0000", 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6740000000000002e-05 2024-04-25T05:47:01.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464298+0000", 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117979 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234266 9.13 9:cde636a1:::benchmark_data_smithi043_129820_object112952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 47.859977090000001, 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.043099716, 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.355 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234266 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:47:01.356 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423285+0000", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.47e-07 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423290+0000", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-06 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423301+0000", 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1249e-05 2024-04-25T05:47:01.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423309+0000", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423370+0000", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1496999999999998e-05 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423380+0000", 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:47:01.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.459982+0000", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.036602386 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460104+0000", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121814 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460347+0000", 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024315799999999999 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464108+0000", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003761347 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464306+0000", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019793999999999999 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466211+0000", 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001904836 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.360 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466254+0000", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2517e-05 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466302+0000", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8791000000000001e-05 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466384+0000", 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1110000000000001e-05 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237263 9.10 9:09fb57ed:::benchmark_data_smithi043_129820_object115949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.187069708000003, 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0207323559999999, 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237263 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096192+0000", 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8400000000000004e-07 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096195+0000", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1750000000000001e-06 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096202+0000", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8600000000000004e-06 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096206+0000", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096257+0000", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0695000000000002e-05 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096262+0000", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5300000000000004e-06 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113813+0000", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0175506910000001 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114018+0000", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000205124 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114193+0000", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000174981 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116636+0000", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024427059999999998 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116671+0000", 2024-04-25T05:47:01.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5380999999999999e-05 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116825+0000", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000153993 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116873+0000", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7308000000000002e-05 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116924+0000", 2024-04-25T05:47:01.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1087000000000001e-05 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237265 9.13 9:ced551e7:::benchmark_data_smithi043_129820_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.185846261000002, 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.029123689, 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.368 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237265 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3300000000000001e-07 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097420+0000", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0969999999999999e-06 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097429+0000", 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0380000000000005e-06 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097436+0000", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7449999999999995e-06 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097483+0000", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6343e-05 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097489+0000", 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6830000000000002e-06 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.371 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115159+0000", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0176693729999999 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115269+0000", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109979 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115548+0000", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027962100000000003 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118288+0000", 2024-04-25T05:47:01.372 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027397879999999999 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120356+0000", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020682280000000001 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126451+0000", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060945569999999996 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126495+0000", 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3942e-05 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126538+0000", 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3562000000000001e-05 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237270 9.13 9:ccb5a786:::benchmark_data_smithi043_129820_object115956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.182480681000001, 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.025924235, 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.374 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237270 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:47:01.375 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100781+0000", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1200000000000002e-07 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100789+0000", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7179999999999996e-06 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100798+0000", 2024-04-25T05:47:01.376 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8599999999999999e-06 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100804+0000", 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0009999999999998e-06 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100867+0000", 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3851999999999998e-05 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.377 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100877+0000", 2024-04-25T05:47:01.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1940000000000004e-06 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117090+0000", 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016213622 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117202+0000", 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111869 2024-04-25T05:47:01.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117405+0000", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020259799999999999 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126115+0000", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0087104629999999999 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126211+0000", 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5789000000000001e-05 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126619+0000", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00040798499999999998 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126663+0000", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3543e-05 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126705+0000", 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2029e-05 2024-04-25T05:47:01.381 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237273 9.10 9:0de19dcd:::benchmark_data_smithi043_129820_object115959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.180709397999998, 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0209831819999999, 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.382 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237273 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.383 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102553+0000", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.034e-06 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102559+0000", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9089999999999997e-06 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102570+0000", 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.112e-05 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.384 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102578+0000", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2709999999999992e-06 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102617+0000", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8695999999999997e-05 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102625+0000", 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9929999999999995e-06 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115511+0000", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0128864559999999 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115643+0000", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131746 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115827+0000", 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018383200000000001 2024-04-25T05:47:01.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118880+0000", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030529400000000001 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119838+0000", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00095799800000000005 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123433+0000", 2024-04-25T05:47:01.387 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035950230000000001 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123491+0000", 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8242000000000002e-05 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123535+0000", 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3921999999999997e-05 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.388 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237293 9.13 9:cf509854:::benchmark_data_smithi043_129820_object115979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.172590575000001, 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016106186, 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237293 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.389 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110671+0000", 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.390 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110676+0000", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6029999999999998e-06 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110685+0000", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6149999999999997e-06 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110691+0000", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8569999999999997e-06 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110748+0000", 2024-04-25T05:47:01.391 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6978999999999998e-05 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110757+0000", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7949999999999996e-06 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118323+0000", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007566237 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118419+0000", 2024-04-25T05:47:01.392 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5356e-05 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118567+0000", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014814099999999999 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126167+0000", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.007599522 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126302+0000", 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013507600000000001 2024-04-25T05:47:01.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126712+0000", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041020800000000001 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126745+0000", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3250999999999997e-05 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126777+0000", 2024-04-25T05:47:01.394 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1547000000000001e-05 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237310 9.13 9:cc1e0a4c:::benchmark_data_smithi043_129820_object115996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.163362524999997, 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007006198, 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.395 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237310 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119899+0000", 2024-04-25T05:47:01.396 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9100000000000003e-07 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119903+0000", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5090000000000002e-06 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119912+0000", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0440000000000001e-06 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119917+0000", 2024-04-25T05:47:01.397 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2959999999999998e-06 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119983+0000", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5779999999999997e-05 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119993+0000", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8179999999999999e-06 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119292+0000", 2024-04-25T05:47:01.398 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99929933400000004 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119417+0000", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125449 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119591+0000", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017395799999999999 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126261+0000", 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066693810000000003 2024-04-25T05:47:01.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126390+0000", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000129164 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126784+0000", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039376400000000001 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126863+0000", 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8963999999999998e-05 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.400 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126905+0000", 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2045999999999997e-05 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237325 9.13 9:cbb3ea78:::benchmark_data_smithi043_129820_object116011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.153765712999999, 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99763348100000004, 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.401 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237325 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129496+0000", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4600000000000004e-07 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129500+0000", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1790000000000001e-06 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129511+0000", 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0611000000000001e-05 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129518+0000", 2024-04-25T05:47:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6020000000000002e-06 2024-04-25T05:47:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.404 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.404 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[27010]: pgmap v696: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-04-25T05:47:01.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4213351952' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.405 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:01 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]: dispatch 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129565+0000", 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6158000000000001e-05 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129573+0000", 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6240000000000008e-06 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.405 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120395+0000", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99082194099999998 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120502+0000", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010669 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120678+0000", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175976 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.406 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126345+0000", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005667025 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126911+0000", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056641199999999997 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127019+0000", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107938 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.407 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127082+0000", 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2848999999999994e-05 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127129+0000", 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6730000000000002e-05 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237346 9.10 9:0cf8b6b3:::benchmark_data_smithi043_129820_object116032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 45.141976763999999, 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98237561100000004, 2024-04-25T05:47:01.408 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237346 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:47:01.409 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141285+0000", 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3699999999999999e-07 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141290+0000", 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6820000000000002e-06 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.410 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141300+0000", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9890000000000004e-06 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141309+0000", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9840000000000007e-06 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141364+0000", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4886999999999998e-05 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:47:01.411 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141373+0000", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2369999999999998e-06 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116981+0000", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97560839600000004 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117101+0000", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119911 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:47:01.412 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117270+0000", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016895300000000001 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119880+0000", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026103020000000001 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.121458+0000", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001577422 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:01.413 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123574+0000", 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021161970000000002 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123625+0000", 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0562000000000001e-05 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123660+0000", 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5151999999999998e-05 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.415 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:47:01.415 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:47:01.415 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:47:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.446 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:01.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:01.580 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:01.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:01.716 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:01.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:01.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:47:01.925025+0000 min lat: 0.0012506 max lat: 0.202544 avg lat: 0.0078344 2024-04-25T05:47:01.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:47:01.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 40838 40822 127.556 105.875 0.00648406 0.0078344 2024-04-25T05:47:01.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:01.990 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:01.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:02.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:02.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:02.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:02 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]': finished 2024-04-25T05:47:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:02 smithi060 ceph-mon[30898]: osdmap e398: 8 total, 8 up, 7 in 2024-04-25T05:47:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:02.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:02.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:02.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:47:02.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:02.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:02 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]': finished 2024-04-25T05:47:02.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:02 smithi043 ceph-mon[27010]: osdmap e398: 8 total, 8 up, 7 in 2024-04-25T05:47:02.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:02 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [1, 2, 4]}]': finished 2024-04-25T05:47:02.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:02 smithi043 ceph-mon[37492]: osdmap e398: 8 total, 8 up, 7 in 2024-04-25T05:47:02.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.519 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:02.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:02.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:02.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:02.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:02.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:02.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 42472 42456 126.345 102.125 0.00751439 0.00790947 2024-04-25T05:47:02.931 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:02.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:03.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:03.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:03.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:03.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: pgmap v698: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: osdmap e399: 8 total, 8 up, 7 in 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: pgmap v698: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: osdmap e399: 8 total, 8 up, 7 in 2024-04-25T05:47:03.256 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:03.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:03.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:03.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:03.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:03.329 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.347 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:03.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:03.368 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:03.408 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: pgmap v698: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: osdmap e399: 8 total, 8 up, 7 in 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:03.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:03.447 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:03.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:03.485 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:03.487 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:03.528 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:03.568 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:03.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:03.606 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:03.621 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:03.655 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:03.660 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:03.660 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:03.660 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:03.660 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:03.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:03.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:03.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:03.661 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:03.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:03.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:03.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:03.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:03.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:03.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:03.926 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 44115 44099 125.269 102.688 0.0110934 0.00797735 2024-04-25T05:47:03.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:04.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:04.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:04.160 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:04.174 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:47:04.252 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:04.252 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:04.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:04.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:04.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:04.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:04 smithi060 ceph-mon[30898]: osdmap e400: 8 total, 8 up, 7 in 2024-04-25T05:47:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3399721165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:04.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:04.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:04.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:04.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[27010]: osdmap e400: 8 total, 8 up, 7 in 2024-04-25T05:47:04.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3399721165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:04.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:04.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:04.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[37492]: osdmap e400: 8 total, 8 up, 7 in 2024-04-25T05:47:04.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3399721165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:04.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:04.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:04.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:04.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:04.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 45932 45916 124.759 113.562 0.00560575 0.00801054 2024-04-25T05:47:04.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:04.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:05.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:05.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:47:05.118 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:05.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:05.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:05.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:05.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[37492]: pgmap v701: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-04-25T05:47:05.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:05.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:05.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[37492]: osdmap e401: 8 total, 8 up, 7 in 2024-04-25T05:47:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[27010]: pgmap v701: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-04-25T05:47:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:05.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:05 smithi043 ceph-mon[27010]: osdmap e401: 8 total, 8 up, 7 in 2024-04-25T05:47:05.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.393 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:05.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:05.494 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:05 smithi060 ceph-mon[30898]: pgmap v701: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.61k op/s 2024-04-25T05:47:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:05 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2383831177' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:05 smithi060 ceph-mon[30898]: osdmap e401: 8 total, 8 up, 7 in 2024-04-25T05:47:05.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:05.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:05.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:05.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:05.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:05.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 47690 47674 124.139 109.875 0.00646679 0.00804984 2024-04-25T05:47:05.937 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:05.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:06.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:06.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:06.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:06.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:06.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.341 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:06.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:06.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:06.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:06.705 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:06.739 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:06.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:06.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:06.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:06.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 16 49401 49385 123.451 106.938 0.00336629 0.00809373 2024-04-25T05:47:06.977 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:07.011 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:07.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:07.145 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:07.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:07.280 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:07.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:07.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:07.471 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:47:07.471 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:47:07.471 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-25T05:47:07.472 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T05:47:07.472 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T05:47:07.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-25T05:47:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:07 smithi060 ceph-mon[30898]: pgmap v703: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:47:07.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:07.551 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:07 smithi043 ceph-mon[27010]: pgmap v703: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:47:07.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:07 smithi043 ceph-mon[37492]: pgmap v703: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-04-25T05:47:07.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:07.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.704 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:07.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:07.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:07.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:07.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:07.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 50878 50862 122.253 92.3125 0.0123703 0.00817445 2024-04-25T05:47:07.925 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:47:07.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:47:07.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.008 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:08.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:08.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:08.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:08.245 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.281 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:08.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:08.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:08.509 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:08.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:08.553 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:08.577 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:08.612 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:47:08.612 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "" 2024-04-25T05:47:08.612 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:47:08.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:08.689 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:08.791 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:08.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 16 52284 52268 120.979 87.875 0.00254648 0.00825817 2024-04-25T05:47:08.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:08.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:08.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:09.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:09.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:09.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:09.269 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:09 smithi043 ceph-mon[27010]: pgmap v704: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:47:09.269 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:09 smithi043 ceph-mon[37492]: pgmap v704: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:47:09.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:09.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:09.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:09.479 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:09 smithi060 ceph-mon[30898]: pgmap v704: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:47:09.495 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:09.496 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:09.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:09.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:09.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:09.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:09.784 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:09.858 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:09.858 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:09.858 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:09.865 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:09.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:09.908 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:09.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:09.924 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 53778 53762 119.993 93.375 0.0118101 0.00832815 2024-04-25T05:47:09.949 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:09.990 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:10.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:10.028 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:10.031 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:10.057 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:10.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:10.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:10.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:10.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1264700782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:10.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1264700782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:10.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.479 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:10.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:10.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:47:10.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1264700782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:10.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:47:10.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:10.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:10.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:10.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:10.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:10.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:10.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:10.925 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 55291 55275 119.113 94.5625 0.00888029 0.00839002 2024-04-25T05:47:11.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:11.047 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:11.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:11.184 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:11.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:11.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:11 smithi043 ceph-mon[27010]: pgmap v705: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-04-25T05:47:11.290 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:11 smithi043 ceph-mon[37492]: pgmap v705: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-04-25T05:47:11.324 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:11.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:11.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:11 smithi060 ceph-mon[30898]: pgmap v705: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-04-25T05:47:11.539 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:11.540 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:11.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:11.609 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:11.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:11.745 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:11.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:11.889 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:11.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:11.925 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 56684 56668 118.044 87.0625 0.00442498 0.00846184 2024-04-25T05:47:11.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:12.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:12.160 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:12.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:12.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:12.406 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:12.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:12 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:47:12.533 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:12 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:47:12.534 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:12 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:47:12.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:12.588 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:12.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:12.742 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:12.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:12.877 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:12.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.912 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146292 7.4 7:27c079d3:::benchmark_data_smithi043_74356_object29423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.82933894200005, 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.830877023, 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146292 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078844+0000", 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078845+0000", 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.22e-06 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078850+0000", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8199999999999996e-06 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078865+0000", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5109000000000001e-05 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.078872+0000", 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0040000000000002e-06 2024-04-25T05:47:12.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079233+0000", 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036102900000000002 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079358+0000", 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012519399999999999 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:47:12.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.079469+0000", 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110436 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087939+0000", 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0084700910000000008 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.087953+0000", 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4079e-05 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734514+0000", 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.646560994 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734529+0000", 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4897e-05 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869085+0000", 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1345559490000001 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869249+0000", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016453999999999999 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909625+0000", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.040376196000000003 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909721+0000", 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5464999999999995e-05 2024-04-25T05:47:12.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146325 7.4 7:287eff09:::benchmark_data_smithi043_74356_object29456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.78579680799999, 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.78676667, 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.923 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146325 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122386+0000", 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122388+0000", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8509999999999999e-06 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122394+0000", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4629999999999998e-06 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122407+0000", 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2965e-05 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122416+0000", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1439999999999992e-06 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122487+0000", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0815000000000006e-05 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.122500+0000", 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2641e-05 2024-04-25T05:47:12.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734558+0000", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.61205790800000004 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734567+0000", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869583+0000", 2024-04-25T05:47:12.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.135015624 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869717+0000", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013412499999999999 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.869897+0000", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018037399999999999 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876526+0000", 2024-04-25T05:47:12.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066293460000000004 2024-04-25T05:47:12.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882816+0000", 2024-04-25T05:47:12.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062898030000000001 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889558+0000", 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0067413789999999996 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.908921+0000", 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.01936336 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909004+0000", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3045000000000002e-05 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909153+0000", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014846799999999999 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146329 7.4 7:2d613d15:::benchmark_data_smithi043_74356_object29460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:47:12.931 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.78322290599999, 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.784987538, 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146329 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124960+0000", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124962+0000", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.835e-06 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124968+0000", 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5849999999999996e-06 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124986+0000", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.823e-05 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.124996+0000", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3719999999999994e-06 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125091+0000", 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5061000000000003e-05 2024-04-25T05:47:12.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.125102+0000", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.094e-05 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734591+0000", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.60948939099999999 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734600+0000", 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4509999999999992e-06 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872375+0000", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1377753349999999 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872499+0000", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124499 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.872641+0000", 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141438 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876579+0000", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039384210000000001 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882926+0000", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0063468170000000003 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889565+0000", 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066388469999999998 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909760+0000", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020195310000000001 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909819+0000", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8798000000000002e-05 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909947+0000", 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012820799999999999 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.938 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146354 7.4 7:2311a945:::benchmark_data_smithi043_74356_object29485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.74300157300002, 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.745096427, 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146354 2024-04-25T05:47:12.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165181+0000", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165182+0000", 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.32e-07 2024-04-25T05:47:12.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165184+0000", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5189999999999999e-06 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165189+0000", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3580000000000002e-06 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165195+0000", 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.356e-06 2024-04-25T05:47:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165284+0000", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8756000000000002e-05 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.165295+0000", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1338000000000001e-05 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734624+0000", 2024-04-25T05:47:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.56932913799999996 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734632+0000", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4710000000000007e-06 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876617+0000", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.141984085 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876755+0000", 2024-04-25T05:47:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138923 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.876917+0000", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016150299999999999 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882874+0000", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059565750000000004 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883014+0000", 2024-04-25T05:47:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013997799999999999 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889572+0000", 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065581820000000001 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910091+0000", 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.02051905 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910155+0000", 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4060000000000007e-05 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910278+0000", 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012270299999999999 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146371 7.4 7:20dc29ab:::benchmark_data_smithi043_74356_object29502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.72494950600003, 2024-04-25T05:47:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7272659080000001, 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146371 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183233+0000", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183234+0000", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.46e-07 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183236+0000", 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4040000000000002e-06 2024-04-25T05:47:12.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183240+0000", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4000000000000002e-06 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183245+0000", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6750000000000001e-06 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183308+0000", 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2946000000000003e-05 2024-04-25T05:47:12.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.183320+0000", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1895e-05 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734658+0000", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.55133849499999998 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734668+0000", 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0145e-05 2024-04-25T05:47:12.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879759+0000", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.145090291 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.879914+0000", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154835 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.880070+0000", 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015658800000000001 2024-04-25T05:47:12.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.882969+0000", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028987420000000002 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889540+0000", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065711279999999999 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889579+0000", 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8476999999999998e-05 2024-04-25T05:47:12.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910326+0000", 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020747422000000001 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910386+0000", 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0270999999999999e-05 2024-04-25T05:47:12.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910499+0000", 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112848 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146375 7.4 7:20e96b3d:::benchmark_data_smithi043_74356_object29506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:47:12.954 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.72016457200004, 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7227024800000001, 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146375 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:47:12.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.956 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 57858 57842 116.603 73.375 0.0103968 0.00857107 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188018+0000", 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188019+0000", 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.333e-06 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188025+0000", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7180000000000001e-06 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188038+0000", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3264999999999999e-05 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188047+0000", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0450000000000006e-06 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188122+0000", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4140000000000005e-05 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.188132+0000", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0737000000000001e-05 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734692+0000", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.54655990700000001 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734701+0000", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2749999999999995e-06 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883051+0000", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1483502480000001 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883195+0000", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014417400000000001 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.883347+0000", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015209500000000001 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889584+0000", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062372699999999996 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889631+0000", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7012999999999999e-05 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889684+0000", 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3078999999999998e-05 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910550+0000", 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020865748999999999 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910610+0000", 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9883999999999998e-05 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910721+0000", 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110548 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146378 7.4 7:2089bf57:::benchmark_data_smithi043_74356_object29509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.71148889200003, 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.714255568, 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.963 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146378 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196694+0000", 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196695+0000", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.491e-06 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196703+0000", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4789999999999999e-06 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196715+0000", 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1938e-05 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196724+0000", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8410000000000006e-06 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196789+0000", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5946999999999997e-05 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.196801+0000", 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1279e-05 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734725+0000", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53792390300000004 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734733+0000", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3599999999999996e-06 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886492+0000", 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1517590280000001 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886663+0000", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000171094 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.886838+0000", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017525099999999999 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889701+0000", 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028622180000000001 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889754+0000", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3440000000000003e-05 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899237+0000", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0094831459999999996 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910763+0000", 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011526290999999999 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910824+0000", 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0829999999999999e-05 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910949+0000", 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012503199999999999 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.970 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146380 7.4 7:2f11d6b2:::benchmark_data_smithi043_74356_object29511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.71076412100001, 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7323310589999998, 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146380 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197419+0000", 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6000000000000003e-07 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197424+0000", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.369e-06 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197441+0000", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7419999999999999e-05 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197449+0000", 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4910000000000005e-06 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197538+0000", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8030999999999998e-05 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197548+0000", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0334e-05 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734761+0000", 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721291999999998 2024-04-25T05:47:12.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734769+0000", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4290000000000001e-06 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.889796+0000", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1550264050000001 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890050+0000", 2024-04-25T05:47:12.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025425699999999997 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.890229+0000", 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017900300000000001 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899288+0000", 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090589869999999992 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899330+0000", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2453999999999997e-05 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911000+0000", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011669538 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929593+0000", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.018593014000000001 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929639+0000", 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6319999999999997e-05 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.929750+0000", 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011042700000000001 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146381 7.4 7:2ef47468:::benchmark_data_smithi043_74356_object29512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:47:12.978 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.71071271100004, 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.732786876, 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146381 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197470+0000", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6800000000000001e-07 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197474+0000", 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3900000000000002e-06 2024-04-25T05:47:12.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197481+0000", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7730000000000002e-06 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197487+0000", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.421e-06 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197574+0000", 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7058000000000006e-05 2024-04-25T05:47:12.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.197582+0000", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3489999999999992e-06 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734793+0000", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.53721096300000004 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734802+0000", 2024-04-25T05:47:12.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5159999999999994e-06 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895295+0000", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1604930549999999 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895455+0000", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015972800000000001 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.895617+0000", 2024-04-25T05:47:12.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016215500000000001 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899375+0000", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037581239999999998 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909210+0000", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0098352049999999996 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911053+0000", 2024-04-25T05:47:12.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0018428959999999999 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930101+0000", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.019048197999999999 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930150+0000", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8424000000000003e-05 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930257+0000", 2024-04-25T05:47:12.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107153 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146407 7.4 7:23e67bf6:::benchmark_data_smithi043_74356_object29538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.67110511999999, 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.693338502, 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.986 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146407 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237078+0000", 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237079+0000", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.516e-06 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237089+0000", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6940000000000007e-06 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237100+0000", 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1114000000000001e-05 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237108+0000", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1040000000000007e-06 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237170+0000", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1851000000000001e-05 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.237180+0000", 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.063e-05 2024-04-25T05:47:12.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734826+0000", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.497645477 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.734834+0000", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1210000000000007e-06 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899407+0000", 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.164572851 2024-04-25T05:47:12.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899548+0000", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000141142 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.899702+0000", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015383899999999999 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909262+0000", 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0095605629999999994 2024-04-25T05:47:12.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.909994+0000", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00073198100000000004 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926123+0000", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.016128397999999999 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930265+0000", 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041419300000000003 2024-04-25T05:47:12.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930315+0000", 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0584000000000002e-05 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930416+0000", 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100707 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146411 7.4 7:2d6ecc61:::benchmark_data_smithi043_74356_object29542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.66885457399997, 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6912330820000001, 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146411 2024-04-25T05:47:12.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239328+0000", 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:12.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239329+0000", 2024-04-25T05:47:12.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2399999999999997e-07 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239333+0000", 2024-04-25T05:47:12.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5249999999999999e-06 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239343+0000", 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7049999999999994e-06 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239351+0000", 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1499999999999999e-06 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239474+0000", 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012281099999999999 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.239484+0000", 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9709999999999999e-06 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735030+0000", 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.49554585800000001 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:12.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.735043+0000", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3399e-05 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905732+0000", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.170688524 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.905903+0000", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017154299999999999 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:13.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.906084+0000", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000180502 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.910045+0000", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0039608480000000003 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916884+0000", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0068390710000000004 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.926215+0000", 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.009331331 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930423+0000", 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042081159999999996 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930465+0000", 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1239000000000003e-05 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930561+0000", 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6664999999999997e-05 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.24734.0:146416 7.4 7:20f93e30:::benchmark_data_smithi043_74356_object29547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 588.66108664299998, 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6920498500000001, 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.24734", 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 146416 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247096+0000", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247097+0000", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.268e-06 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247103+0000", 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6799999999999998e-06 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247113+0000", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5869999999999999e-06 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247120+0000", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8630000000000008e-06 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247187+0000", 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6701999999999999e-05 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.247197+0000", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0013000000000001e-05 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736134+0000", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.48893706999999997 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:24.736148+0000", 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3709999999999999e-05 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911091+0000", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1749430030000001 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911228+0000", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013674 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3,4", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.911385+0000", 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015695600000000001 2024-04-25T05:47:13.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.916945+0000", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0055600700000000003 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.930568+0000", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013623273999999999 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.931318+0000", 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075026300000000004 2024-04-25T05:47:13.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.935451+0000", 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041326820000000004 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939020+0000", 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035686450000000001 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:37:25.939146+0000", 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012632400000000001 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184730 8.c 8:3895024e:::benchmark_data_smithi043_104001_object60606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 307.00577292200001, 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0106298600000001, 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184730 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902411+0000", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.483e-06 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902418+0000", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3269999999999997e-06 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902437+0000", 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9673999999999998e-05 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902446+0000", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902550+0000", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104485 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902561+0000", 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0351e-05 2024-04-25T05:47:13.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906055+0000", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.003494664 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906151+0000", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5391999999999997e-05 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906246+0000", 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5552999999999998e-05 2024-04-25T05:47:13.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912838+0000", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006591675 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912925+0000", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7159999999999999e-05 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912941+0000", 2024-04-25T05:47:13.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5968999999999999e-05 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912992+0000", 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0841000000000003e-05 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913040+0000", 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7842e-05 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.017 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184731 8.c 8:3f348624:::benchmark_data_smithi043_104001_object60607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 307.00571883700002, 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.010906069, 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184731 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.018 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3099999999999998e-07 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902468+0000", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.337e-06 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902473+0000", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5310000000000001e-06 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902479+0000", 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4700000000000001e-06 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902585+0000", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106054 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902591+0000", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3790000000000003e-06 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908540+0000", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.00594837 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908663+0000", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123437 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908776+0000", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112724 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913122+0000", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043464560000000003 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913170+0000", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7431e-05 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913290+0000", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120698 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913330+0000", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0070000000000001e-05 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913370+0000", 2024-04-25T05:47:13.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9580999999999999e-05 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184780 8.c 8:3fdf1b59:::benchmark_data_smithi043_104001_object60656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 306.95084806900002, 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95616667600000005, 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.024 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184780 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957336+0000", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5799999999999999e-06 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957344+0000", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.807e-06 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957356+0000", 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2404e-05 2024-04-25T05:47:13.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957365+0000", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5040000000000002e-06 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957437+0000", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350999999999999e-05 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957448+0000", 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0669999999999999e-05 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910672+0000", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95322447200000004 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910797+0000", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124442 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910926+0000", 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128849 2024-04-25T05:47:13.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913223+0000", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022977219999999999 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913420+0000", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019708099999999999 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913431+0000", 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0312000000000001e-05 2024-04-25T05:47:13.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913459+0000", 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8577000000000001e-05 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913501+0000", 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1904999999999999e-05 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184787 8.c 8:3320ee50:::benchmark_data_smithi043_104001_object60663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 306.94351684200001, 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95582188800000001, 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184787 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.031 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964667+0000", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964673+0000", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964690+0000", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7629999999999999e-05 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964699+0000", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1390000000000001e-06 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964760+0000", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0217999999999998e-05 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964769+0000", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7860000000000008e-06 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913564+0000", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94879425100000003 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913643+0000", 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9556999999999997e-05 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913717+0000", 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4209999999999996e-05 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920254+0000", 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065365060000000001 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920304+0000", 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9512000000000003e-05 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920385+0000", 2024-04-25T05:47:13.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1979999999999998e-05 2024-04-25T05:47:13.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.037 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920439+0000", 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3928000000000003e-05 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920488+0000", 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8301999999999999e-05 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226338 8.6 8:6585b86d:::benchmark_data_smithi043_104001_object102214:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:47:13.038 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 273.01134633200002, 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.103761714, 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226338 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896840+0000", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2289999999999999e-06 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896861+0000", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1603999999999998e-05 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896888+0000", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7047999999999999e-05 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896903+0000", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4888999999999999e-05 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019552+0000", 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12264891999999999 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019561+0000", 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0690000000000008e-06 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000512+0000", 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98095072000000005 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000598+0000", 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6235000000000004e-05 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226349 8.6 8:6373a4df:::benchmark_data_smithi043_104001_object102225:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 273.01125358399997, 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.139697025, 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.043 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226349 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896930+0000", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4199999999999996e-07 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896933+0000", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.551e-06 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896939+0000", 2024-04-25T05:47:13.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1380000000000004e-06 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896945+0000", 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1959999999999999e-06 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019583+0000", 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122638941 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019591+0000", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.002927+0000", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98333629700000003 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003056+0000", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129037 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:47:13.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003403+0000", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000346733 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.005083+0000", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016799779999999999 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.008504+0000", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034209710000000001 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036533+0000", 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028029505999999999 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036586+0000", 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2278000000000003e-05 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036626+0000", 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0438000000000001e-05 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228017 8.16 8:6e8160c2:::benchmark_data_smithi043_104001_object103893:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 269.93028224199998, 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.134225407, 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228017 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.050 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977902+0000", 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5659999999999999e-06 2024-04-25T05:47:13.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977908+0000", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.592e-06 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977923+0000", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5132e-05 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977934+0000", 2024-04-25T05:47:13.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0869999999999999e-05 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978035+0000", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101284 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978042+0000", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5229999999999998e-06 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982071+0000", 2024-04-25T05:47:13.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004028472 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982081+0000", 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0589e-05 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104437+0000", 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.122355209 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104572+0000", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013536399999999999 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112038+0000", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074657969999999997 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112126+0000", 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8009000000000004e-05 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228023 8.16 8:6ad7e352:::benchmark_data_smithi043_104001_object103899:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 269.93022654200001, 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1340453349999999, 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:47:13.056 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228023 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977957+0000", 2024-04-25T05:47:13.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9500000000000003e-07 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977960+0000", 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3230000000000002e-06 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977967+0000", 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:13.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977974+0000", 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9659999999999999e-06 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978175+0000", 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020133299999999999 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting_for_map not empty", 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978182+0000", 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1530000000000002e-06 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982727+0000", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045456259999999997 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982737+0000", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3440000000000007e-06 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104931+0000", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.12219398 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:13.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105055+0000", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012434499999999999 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5,6,7", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105352+0000", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297082 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.108987+0000", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00363503 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:47:13.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111435+0000", 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024477959999999999 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111738+0000", 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030323699999999998 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111789+0000", 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0565999999999997e-05 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111869+0000", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0865999999999995e-05 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111908+0000", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8052000000000001e-05 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112001+0000", 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3929000000000001e-05 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:13.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:13.064 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:47:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.064 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:13.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:13.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:13.190 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:47:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.204 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:13.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:13 smithi060 ceph-mon[30898]: pgmap v706: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 583 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:47:13.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:13.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:13.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:47:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:13.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:13.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:13.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:13.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:13.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:13 smithi043 ceph-mon[27010]: pgmap v706: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 583 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:47:13.724 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:13.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:13 smithi043 ceph-mon[37492]: pgmap v706: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 583 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-25T05:47:13.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:13.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:13.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:13.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:13.905 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:13.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:13.925 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 59238 59222 115.655 86.25 0.013695 0.00864074 2024-04-25T05:47:14.006 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:14.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:14.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:14.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:14.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.340 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:14.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:14.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:14.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:14.465 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: osdmap e402: 8 total, 8 up, 7 in 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: osdmap e402: 8 total, 8 up, 7 in 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:47:14.466 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:14.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3532445279' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: osdmap e402: 8 total, 8 up, 7 in 2024-04-25T05:47:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:47:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:47:14.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:14.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:14.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:14.661 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:47:14.662 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:47:14.662 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-25T05:47:14.662 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-25T05:47:14.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:47:14.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.770 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:14.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:14.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:14.929 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 60548 60532 114.62 81.875 0.0129965 0.00871929 2024-04-25T05:47:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:14.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:14.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:15.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:15.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:15.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:15.209 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:15.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:15.343 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:15.377 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[27010]: pgmap v708: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[27010]: osdmap e403: 8 total, 8 up, 7 in 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[37492]: pgmap v708: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:47:15.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:15 smithi043 ceph-mon[37492]: osdmap e403: 8 total, 8 up, 7 in 2024-04-25T05:47:15.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:47:15.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:15 smithi060 ceph-mon[30898]: pgmap v708: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:15 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:47:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:15 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:47:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:15 smithi060 ceph-mon[30898]: osdmap e403: 8 total, 8 up, 7 in 2024-04-25T05:47:15.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:15.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:15.644 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:15.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:15.781 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:15.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:15.906 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:15.916 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:15.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:15.921 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:15.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 61907 61891 113.743 84.9375 0.00534556 0.00878643 2024-04-25T05:47:15.951 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:16.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:16.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:16.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:16.286 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:16.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:16.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:16 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:47:16.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:16 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:47:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:16 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:47:16.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:16 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:47:16.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:16.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:16.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:47:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:16 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:47:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:16 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:47:16.551 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:16.590 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:16.692 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:16.722 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:16.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:16.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:16.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:16.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 63090 63074 112.605 73.9375 0.00954495 0.00887554 2024-04-25T05:47:16.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:17.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:17.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:17.137 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:17.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:17 smithi060 ceph-mon[30898]: pgmap v710: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:47:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:17 smithi060 ceph-mon[30898]: 4.1d scrub starts 2024-04-25T05:47:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:17 smithi060 ceph-mon[30898]: 4.1d scrub ok 2024-04-25T05:47:17.273 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:17.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:17.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[37492]: pgmap v710: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:47:17.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[37492]: 4.1d scrub starts 2024-04-25T05:47:17.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[37492]: 4.1d scrub ok 2024-04-25T05:47:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[27010]: pgmap v710: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-04-25T05:47:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[27010]: 4.1d scrub starts 2024-04-25T05:47:17.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:17 smithi043 ceph-mon[27010]: 4.1d scrub ok 2024-04-25T05:47:17.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:17.523 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:17.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:17.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:17.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:17.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:17.805 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:17.805 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:17.813 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:47:17.818 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:47:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:17.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:17.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:17.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:17.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:17.843 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:17.864 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:17.897 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:17.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 64290 64274 111.561 75 0.00882982 0.00895829 2024-04-25T05:47:17.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:17.942 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:17.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:17.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:17.985 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:17.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:17.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:18.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:18.112 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:18.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:18.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:18.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:18.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:47:18.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:47:18.371 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4068400266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:18.372 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:18.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:47:18.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:47:18.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4068400266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:18.372 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:18.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:18.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:18 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:47:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:18 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:47:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4068400266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:18.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:18.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:18.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:18.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:18.781 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:18.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:18.914 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:18.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:18.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 65662 65646 110.863 85.75 0.0227279 0.00901476 2024-04-25T05:47:19.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:19.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:19.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:47:19.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:19.326 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:19.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:19.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: from='client.25880 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: pgmap v711: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 4.f scrub starts 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 4.f scrub ok 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 5.14 scrub starts 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 5.14 scrub ok 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:47:19.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:19 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: from='client.25880 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: pgmap v711: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 4.f scrub starts 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 4.f scrub ok 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 5.14 scrub starts 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 5.14 scrub ok 2024-04-25T05:47:19.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: from='client.25880 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: pgmap v711: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 968 op/s 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 4.f scrub starts 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 4.f scrub ok 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 5.14 scrub starts 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 5.14 scrub ok 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:47:19.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:47:19.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:19.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:19.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.719 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:19.779 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2024-04-25T05:47:15.939920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3807320603,"num_objects":58316,"num_object_clones":0,"num_object_copies":174948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58316,"num_whiteouts":0,"num_read":2187,"num_read_kb":1907,"num_write":59000,"num_write_kb":3717549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":1196524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60566,"ondisk_log_size":60566,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15342912,"kb_used_data":11097216,"kb_used_omap":10,"kb_used_meta":4245621,"kb_avail":640873152,"statfs":{"total":671965249536,"available":656254107648,"internally_reserved":0,"allocated":11363549184,"data_stored":11359409502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4347516011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,6,6,8,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1005649920,"num_objects":15345,"num_object_clones":0,"num_object_copies":46035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15345,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15345,"num_write_kb":982080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007231"},"pg_stats":[{"pgid":"10.2","version":"402'3688","reported_seq":3728,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.876995+0000","last_change":"2024-04-25T05:46:37.979403+0000","last_active":"2024-04-25T05:47:13.876995+0000","last_peered":"2024-04-25T05:47:13.876995+0000","last_clean":"2024-04-25T05:47:13.876995+0000","last_became_active":"2024-04-25T05:46:37.979097+0000","last_became_peered":"2024-04-25T05:46:37.979097+0000","last_unstale":"2024-04-25T05:47:13.876995+0000","last_undegraded":"2024-04-25T05:47:13.876995+0000","last_fullsized":"2024-04-25T05:47:13.876995+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.041548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241696768,"num_objects":3688,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":236032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":437,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197469+0000","last_change":"2024-04-25T05:46:09.856702+0000","last_active":"2024-04-25T05:47:13.197469+0000","last_peered":"2024-04-25T05:47:13.197469+0000","last_clean":"2024-04-25T05:47:13.197469+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:47:13.197469+0000","last_undegraded":"2024-04-25T05:47:13.197469+0000","last_fullsized":"2024-04-25T05:47:13.197469+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:46:09.856439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.856439+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:35.321244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":727,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196351+0000","last_change":"2024-04-25T05:46:03.753399+0000","last_active":"2024-04-25T05:47:13.196351+0000","last_peered":"2024-04-25T05:47:13.196351+0000","last_clean":"2024-04-25T05:47:13.196351+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:47:13.196351+0000","last_undegraded":"2024-04-25T05:47:13.196351+0000","last_fullsized":"2024-04-25T05:47:13.196351+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:03.753274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:03.753274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:20.423586+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":82,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180173+0000","last_change":"2024-04-25T05:46:35.013019+0000","last_active":"2024-04-25T05:47:04.180173+0000","last_peered":"2024-04-25T05:47:04.180173+0000","last_clean":"2024-04-25T05:47:04.180173+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:47:04.180173+0000","last_undegraded":"2024-04-25T05:47:04.180173+0000","last_fullsized":"2024-04-25T05:47:04.180173+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:45:36.284279+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.284279+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:08.946560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":398,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187121+0000","last_change":"2024-04-25T05:46:19.993548+0000","last_active":"2024-04-25T05:47:13.187121+0000","last_peered":"2024-04-25T05:47:13.187121+0000","last_clean":"2024-04-25T05:47:13.187121+0000","last_became_active":"2024-04-25T05:46:19.990605+0000","last_became_peered":"2024-04-25T05:46:19.990605+0000","last_unstale":"2024-04-25T05:47:13.187121+0000","last_undegraded":"2024-04-25T05:47:13.187121+0000","last_fullsized":"2024-04-25T05:47:13.187121+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.275700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.275700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1041,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171170+0000","last_change":"2024-04-25T05:46:23.149014+0000","last_active":"2024-04-25T05:47:04.171170+0000","last_peered":"2024-04-25T05:47:04.171170+0000","last_clean":"2024-04-25T05:47:04.171170+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:47:04.171170+0000","last_undegraded":"2024-04-25T05:47:04.171170+0000","last_fullsized":"2024-04-25T05:47:04.171170+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"401'3617","reported_seq":3653,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.092978+0000","last_change":"2024-04-25T05:46:37.983434+0000","last_active":"2024-04-25T05:47:13.092978+0000","last_peered":"2024-04-25T05:47:13.092978+0000","last_clean":"2024-04-25T05:47:13.092978+0000","last_became_active":"2024-04-25T05:46:37.983078+0000","last_became_peered":"2024-04-25T05:46:37.983078+0000","last_unstale":"2024-04-25T05:47:13.092978+0000","last_undegraded":"2024-04-25T05:47:13.092978+0000","last_fullsized":"2024-04-25T05:47:13.092978+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.882443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237043712,"num_objects":3617,"num_object_clones":0,"num_object_copies":10851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":231488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":108,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180244+0000","last_change":"2024-04-25T05:46:48.976027+0000","last_active":"2024-04-25T05:47:04.180244+0000","last_peered":"2024-04-25T05:47:04.180244+0000","last_clean":"2024-04-25T05:47:04.180244+0000","last_became_active":"2024-04-25T05:46:36.984563+0000","last_became_peered":"2024-04-25T05:46:36.984563+0000","last_unstale":"2024-04-25T05:47:04.180244+0000","last_undegraded":"2024-04-25T05:47:04.180244+0000","last_fullsized":"2024-04-25T05:47:04.180244+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:46:48.975826+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.975826+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:11.159170+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":439,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.174652+0000","last_change":"2024-04-25T05:46:42.728489+0000","last_active":"2024-04-25T05:47:04.174652+0000","last_peered":"2024-04-25T05:47:04.174652+0000","last_clean":"2024-04-25T05:47:04.174652+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:47:04.174652+0000","last_undegraded":"2024-04-25T05:47:04.174652+0000","last_fullsized":"2024-04-25T05:47:04.174652+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:42.728367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:46:42.728367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:04.407596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":733,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188350+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:47:13.188350+0000","last_peered":"2024-04-25T05:47:13.188350+0000","last_clean":"2024-04-25T05:47:13.188350+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:47:13.188350+0000","last_undegraded":"2024-04-25T05:47:13.188350+0000","last_fullsized":"2024-04-25T05:47:13.188350+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":58,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179902+0000","last_change":"2024-04-25T05:46:20.012603+0000","last_active":"2024-04-25T05:47:04.179902+0000","last_peered":"2024-04-25T05:47:04.179902+0000","last_clean":"2024-04-25T05:47:04.179902+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:47:04.179902+0000","last_undegraded":"2024-04-25T05:47:04.179902+0000","last_fullsized":"2024-04-25T05:47:04.179902+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:09.175587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:09.175587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926117+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":630,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179851+0000","last_change":"2024-04-25T05:46:37.981262+0000","last_active":"2024-04-25T05:47:04.179851+0000","last_peered":"2024-04-25T05:47:04.179851+0000","last_clean":"2024-04-25T05:47:04.179851+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:47:04.179851+0000","last_undegraded":"2024-04-25T05:47:04.179851+0000","last_fullsized":"2024-04-25T05:47:04.179851+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:43:19.441092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:43:19.441092+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"401'3671","reported_seq":3708,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.962158+0000","last_change":"2024-04-25T05:46:37.984708+0000","last_active":"2024-04-25T05:47:12.962158+0000","last_peered":"2024-04-25T05:47:12.962158+0000","last_clean":"2024-04-25T05:47:12.962158+0000","last_became_active":"2024-04-25T05:46:37.984393+0000","last_became_peered":"2024-04-25T05:46:37.984393+0000","last_unstale":"2024-04-25T05:47:12.962158+0000","last_undegraded":"2024-04-25T05:47:12.962158+0000","last_fullsized":"2024-04-25T05:47:12.962158+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3671,"log_dups_size":0,"ondisk_log_size":3671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:41.038864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240582656,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3671,"num_write_kb":234944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":744,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172358+0000","last_change":"2024-04-25T05:45:26.763116+0000","last_active":"2024-04-25T05:47:04.172358+0000","last_peered":"2024-04-25T05:47:04.172358+0000","last_clean":"2024-04-25T05:47:04.172358+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:47:04.172358+0000","last_undegraded":"2024-04-25T05:47:04.172358+0000","last_fullsized":"2024-04-25T05:47:04.172358+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:45:26.762873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:26.762873+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:04.647680+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":744,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.168257+0000","last_change":"2024-04-25T05:46:19.963247+0000","last_active":"2024-04-25T05:47:04.168257+0000","last_peered":"2024-04-25T05:47:04.168257+0000","last_clean":"2024-04-25T05:47:04.168257+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:47:04.168257+0000","last_undegraded":"2024-04-25T05:47:04.168257+0000","last_fullsized":"2024-04-25T05:47:04.168257+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.613376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.613376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:44.222624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180491+0000","last_change":"2024-04-25T05:46:34.982029+0000","last_active":"2024-04-25T05:47:04.180491+0000","last_peered":"2024-04-25T05:47:04.180491+0000","last_clean":"2024-04-25T05:47:04.180491+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:47:04.180491+0000","last_undegraded":"2024-04-25T05:47:04.180491+0000","last_fullsized":"2024-04-25T05:47:04.180491+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:18.666459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:46:18.666459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.700514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":721,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197164+0000","last_change":"2024-04-25T05:46:51.568352+0000","last_active":"2024-04-25T05:47:13.197164+0000","last_peered":"2024-04-25T05:47:13.197164+0000","last_clean":"2024-04-25T05:47:13.197164+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:47:13.197164+0000","last_undegraded":"2024-04-25T05:47:13.197164+0000","last_fullsized":"2024-04-25T05:47:13.197164+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:51.568255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:51.568255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.506539+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"401'3702","reported_seq":3740,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.925012+0000","last_change":"2024-04-25T05:46:37.983353+0000","last_active":"2024-04-25T05:47:12.925012+0000","last_peered":"2024-04-25T05:47:12.925012+0000","last_clean":"2024-04-25T05:47:12.925012+0000","last_became_active":"2024-04-25T05:46:37.983037+0000","last_became_peered":"2024-04-25T05:46:37.983037+0000","last_unstale":"2024-04-25T05:47:12.925012+0000","last_undegraded":"2024-04-25T05:47:12.925012+0000","last_fullsized":"2024-04-25T05:47:12.925012+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:51.005000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242614272,"num_objects":3702,"num_object_clones":0,"num_object_copies":11106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":236928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":755,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187598+0000","last_change":"2024-04-25T05:42:50.870199+0000","last_active":"2024-04-25T05:47:13.187598+0000","last_peered":"2024-04-25T05:47:13.187598+0000","last_clean":"2024-04-25T05:47:13.187598+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:47:13.187598+0000","last_undegraded":"2024-04-25T05:47:13.187598+0000","last_fullsized":"2024-04-25T05:47:13.187598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'22","last_scrub_stamp":"2024-04-25T05:42:50.869985+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:42:50.869985+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":710,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195676+0000","last_change":"2024-04-25T05:44:13.053742+0000","last_active":"2024-04-25T05:47:13.195676+0000","last_peered":"2024-04-25T05:47:13.195676+0000","last_clean":"2024-04-25T05:47:13.195676+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:47:13.195676+0000","last_undegraded":"2024-04-25T05:47:13.195676+0000","last_fullsized":"2024-04-25T05:47:13.195676+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.052894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.052894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474821+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'15","reported_seq":53,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180267+0000","last_change":"2024-04-25T05:46:35.014060+0000","last_active":"2024-04-25T05:47:04.180267+0000","last_peered":"2024-04-25T05:47:04.180267+0000","last_clean":"2024-04-25T05:47:04.180267+0000","last_became_active":"2024-04-25T05:46:34.989992+0000","last_became_peered":"2024-04-25T05:46:34.989992+0000","last_unstale":"2024-04-25T05:47:04.180267+0000","last_undegraded":"2024-04-25T05:47:04.180267+0000","last_fullsized":"2024-04-25T05:47:04.180267+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:09.887854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179733+0000","last_change":"2024-04-25T05:46:34.990045+0000","last_active":"2024-04-25T05:47:04.179733+0000","last_peered":"2024-04-25T05:47:04.179733+0000","last_clean":"2024-04-25T05:47:04.179733+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:47:04.179733+0000","last_undegraded":"2024-04-25T05:47:04.179733+0000","last_fullsized":"2024-04-25T05:47:04.179733+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":595,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.181404+0000","last_change":"2024-04-25T05:46:37.982298+0000","last_active":"2024-04-25T05:47:04.181404+0000","last_peered":"2024-04-25T05:47:04.181404+0000","last_clean":"2024-04-25T05:47:04.181404+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:47:04.181404+0000","last_undegraded":"2024-04-25T05:47:04.181404+0000","last_fullsized":"2024-04-25T05:47:04.181404+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:44:04.667533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.667533+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"401'3597","reported_seq":3635,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.944647+0000","last_change":"2024-04-25T05:46:37.983920+0000","last_active":"2024-04-25T05:47:12.944647+0000","last_peered":"2024-04-25T05:47:12.944647+0000","last_clean":"2024-04-25T05:47:12.944647+0000","last_became_active":"2024-04-25T05:46:37.983600+0000","last_became_peered":"2024-04-25T05:46:37.983600+0000","last_unstale":"2024-04-25T05:47:12.944647+0000","last_undegraded":"2024-04-25T05:47:12.944647+0000","last_fullsized":"2024-04-25T05:47:12.944647+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3597,"log_dups_size":0,"ondisk_log_size":3597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:37.012992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235732992,"num_objects":3597,"num_object_clones":0,"num_object_copies":10791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":230208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":716,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.168139+0000","last_change":"2024-04-25T05:45:29.415240+0000","last_active":"2024-04-25T05:47:04.168139+0000","last_peered":"2024-04-25T05:47:04.168139+0000","last_clean":"2024-04-25T05:47:04.168139+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:47:04.168139+0000","last_undegraded":"2024-04-25T05:47:04.168139+0000","last_fullsized":"2024-04-25T05:47:04.168139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.415117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.415117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:07.585214+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":864,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195758+0000","last_change":"2024-04-25T05:44:28.232756+0000","last_active":"2024-04-25T05:47:13.195758+0000","last_peered":"2024-04-25T05:47:13.195758+0000","last_clean":"2024-04-25T05:47:13.195758+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:47:13.195758+0000","last_undegraded":"2024-04-25T05:47:13.195758+0000","last_fullsized":"2024-04-25T05:47:13.195758+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:44:20.972395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:20.972395+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474946+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":742,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.201451+0000","last_change":"2024-04-25T05:46:34.972300+0000","last_active":"2024-04-25T05:47:13.201451+0000","last_peered":"2024-04-25T05:47:13.201451+0000","last_clean":"2024-04-25T05:47:13.201451+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:47:13.201451+0000","last_undegraded":"2024-04-25T05:47:13.201451+0000","last_fullsized":"2024-04-25T05:47:13.201451+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:21.023940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:21.023940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":392,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.198047+0000","last_change":"2024-04-25T05:46:19.998231+0000","last_active":"2024-04-25T05:47:13.198047+0000","last_peered":"2024-04-25T05:47:13.198047+0000","last_clean":"2024-04-25T05:47:13.198047+0000","last_became_active":"2024-04-25T05:46:19.997163+0000","last_became_peered":"2024-04-25T05:46:19.997163+0000","last_unstale":"2024-04-25T05:47:13.198047+0000","last_undegraded":"2024-04-25T05:47:13.198047+0000","last_fullsized":"2024-04-25T05:47:13.198047+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.919114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.919114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474749+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":945,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.167647+0000","last_change":"2024-04-25T05:46:23.130821+0000","last_active":"2024-04-25T05:47:04.167647+0000","last_peered":"2024-04-25T05:47:04.167647+0000","last_clean":"2024-04-25T05:47:04.167647+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:47:04.167647+0000","last_undegraded":"2024-04-25T05:47:04.167647+0000","last_fullsized":"2024-04-25T05:47:04.167647+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:36.587388+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.587388+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:07.585275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"401'3570","reported_seq":3608,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.952196+0000","last_change":"2024-04-25T05:46:37.985638+0000","last_active":"2024-04-25T05:47:12.952196+0000","last_peered":"2024-04-25T05:47:12.952196+0000","last_clean":"2024-04-25T05:47:12.952196+0000","last_became_active":"2024-04-25T05:46:37.982701+0000","last_became_peered":"2024-04-25T05:46:37.982701+0000","last_unstale":"2024-04-25T05:47:12.952196+0000","last_undegraded":"2024-04-25T05:47:12.952196+0000","last_fullsized":"2024-04-25T05:47:12.952196+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:41.010962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233963520,"num_objects":3570,"num_object_clones":0,"num_object_copies":10710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":228480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":879,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195761+0000","last_change":"2024-04-25T05:45:36.207163+0000","last_active":"2024-04-25T05:47:13.195761+0000","last_peered":"2024-04-25T05:47:13.195761+0000","last_clean":"2024-04-25T05:47:13.195761+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:47:13.195761+0000","last_undegraded":"2024-04-25T05:47:13.195761+0000","last_fullsized":"2024-04-25T05:47:13.195761+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:45:34.969209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:34.969209+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":60,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179374+0000","last_change":"2024-04-25T05:46:35.010369+0000","last_active":"2024-04-25T05:47:04.179374+0000","last_peered":"2024-04-25T05:47:04.179374+0000","last_clean":"2024-04-25T05:47:04.179374+0000","last_became_active":"2024-04-25T05:46:34.980031+0000","last_became_peered":"2024-04-25T05:46:34.980031+0000","last_unstale":"2024-04-25T05:47:04.179374+0000","last_undegraded":"2024-04-25T05:47:04.179374+0000","last_fullsized":"2024-04-25T05:47:04.179374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:45:27.764695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.764695+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":414,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187477+0000","last_change":"2024-04-25T05:44:28.221246+0000","last_active":"2024-04-25T05:47:13.187477+0000","last_peered":"2024-04-25T05:47:13.187477+0000","last_clean":"2024-04-25T05:47:13.187477+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:47:13.187477+0000","last_undegraded":"2024-04-25T05:47:13.187477+0000","last_fullsized":"2024-04-25T05:47:13.187477+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:43:23.347909+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:35:36.294192+0000","last_clean_scrub_stamp":"2024-04-25T05:43:23.347909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":92,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187941+0000","last_change":"2024-04-25T05:46:19.992323+0000","last_active":"2024-04-25T05:47:13.187941+0000","last_peered":"2024-04-25T05:47:13.187941+0000","last_clean":"2024-04-25T05:47:13.187941+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:47:13.187941+0000","last_undegraded":"2024-04-25T05:47:13.187941+0000","last_fullsized":"2024-04-25T05:47:13.187941+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:00.031327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:00.031327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":911,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196790+0000","last_change":"2024-04-25T05:46:37.983775+0000","last_active":"2024-04-25T05:47:13.196790+0000","last_peered":"2024-04-25T05:47:13.196790+0000","last_clean":"2024-04-25T05:47:13.196790+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:47:13.196790+0000","last_undegraded":"2024-04-25T05:47:13.196790+0000","last_fullsized":"2024-04-25T05:47:13.196790+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.882669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.882669+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"401'3660","reported_seq":3698,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.005840+0000","last_change":"2024-04-25T05:46:37.986037+0000","last_active":"2024-04-25T05:47:13.005840+0000","last_peered":"2024-04-25T05:47:13.005840+0000","last_clean":"2024-04-25T05:47:13.005840+0000","last_became_active":"2024-04-25T05:46:37.985670+0000","last_became_peered":"2024-04-25T05:46:37.985670+0000","last_unstale":"2024-04-25T05:47:13.005840+0000","last_undegraded":"2024-04-25T05:47:13.005840+0000","last_fullsized":"2024-04-25T05:47:13.005840+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:52.279419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239861760,"num_objects":3660,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":234240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":468,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172524+0000","last_change":"2024-04-25T05:46:46.694407+0000","last_active":"2024-04-25T05:47:04.172524+0000","last_peered":"2024-04-25T05:47:04.172524+0000","last_clean":"2024-04-25T05:47:04.172524+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:47:04.172524+0000","last_undegraded":"2024-04-25T05:47:04.172524+0000","last_fullsized":"2024-04-25T05:47:04.172524+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:46:46.694273+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.694273+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:04.268414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":460,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188015+0000","last_change":"2024-04-25T05:46:45.354890+0000","last_active":"2024-04-25T05:47:13.188015+0000","last_peered":"2024-04-25T05:47:13.188015+0000","last_clean":"2024-04-25T05:47:13.188015+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:47:13.188015+0000","last_undegraded":"2024-04-25T05:47:13.188015+0000","last_fullsized":"2024-04-25T05:47:13.188015+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:46:45.354712+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.354712+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:50.813205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179772+0000","last_change":"2024-04-25T05:46:34.988645+0000","last_active":"2024-04-25T05:47:04.179772+0000","last_peered":"2024-04-25T05:47:04.179772+0000","last_clean":"2024-04-25T05:47:04.179772+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:47:04.179772+0000","last_undegraded":"2024-04-25T05:47:04.179772+0000","last_fullsized":"2024-04-25T05:47:04.179772+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:20.410408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:20.410408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":356,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195557+0000","last_change":"2024-04-25T05:46:44.669771+0000","last_active":"2024-04-25T05:47:13.195557+0000","last_peered":"2024-04-25T05:47:13.195557+0000","last_clean":"2024-04-25T05:47:13.195557+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:47:13.195557+0000","last_undegraded":"2024-04-25T05:47:13.195557+0000","last_fullsized":"2024-04-25T05:47:13.195557+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.669615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.669615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.090431+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":894,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172879+0000","last_change":"2024-04-25T05:45:40.844494+0000","last_active":"2024-04-25T05:47:04.172879+0000","last_peered":"2024-04-25T05:47:04.172879+0000","last_clean":"2024-04-25T05:47:04.172879+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:47:04.172879+0000","last_undegraded":"2024-04-25T05:47:04.172879+0000","last_fullsized":"2024-04-25T05:47:04.172879+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:40.844354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:40.844354+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:04.647453+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"401'3548","reported_seq":3586,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.929427+0000","last_change":"2024-04-25T05:46:37.986967+0000","last_active":"2024-04-25T05:47:12.929427+0000","last_peered":"2024-04-25T05:47:12.929427+0000","last_clean":"2024-04-25T05:47:12.929427+0000","last_became_active":"2024-04-25T05:46:37.986694+0000","last_became_peered":"2024-04-25T05:46:37.986694+0000","last_unstale":"2024-04-25T05:47:12.929427+0000","last_undegraded":"2024-04-25T05:47:12.929427+0000","last_fullsized":"2024-04-25T05:47:12.929427+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.428344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232521728,"num_objects":3548,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":227072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":469,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.174805+0000","last_change":"2024-04-25T05:46:34.966272+0000","last_active":"2024-04-25T05:47:04.174805+0000","last_peered":"2024-04-25T05:47:04.174805+0000","last_clean":"2024-04-25T05:47:04.174805+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:47:04.174805+0000","last_undegraded":"2024-04-25T05:47:04.174805+0000","last_fullsized":"2024-04-25T05:47:04.174805+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:12.712241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:46:12.712241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.000979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":88,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179913+0000","last_change":"2024-04-25T05:46:36.996561+0000","last_active":"2024-04-25T05:47:04.179913+0000","last_peered":"2024-04-25T05:47:04.179913+0000","last_clean":"2024-04-25T05:47:04.179913+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:47:04.179913+0000","last_undegraded":"2024-04-25T05:47:04.179913+0000","last_fullsized":"2024-04-25T05:47:04.179913+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:46:07.457049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.457049+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:13.362844+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":748,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188157+0000","last_change":"2024-04-25T05:46:34.974950+0000","last_active":"2024-04-25T05:47:13.188157+0000","last_peered":"2024-04-25T05:47:13.188157+0000","last_clean":"2024-04-25T05:47:13.188157+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:47:13.188157+0000","last_undegraded":"2024-04-25T05:47:13.188157+0000","last_fullsized":"2024-04-25T05:47:13.188157+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.307605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.307605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":730,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188957+0000","last_change":"2024-04-25T05:46:47.388550+0000","last_active":"2024-04-25T05:47:13.188957+0000","last_peered":"2024-04-25T05:47:13.188957+0000","last_clean":"2024-04-25T05:47:13.188957+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:47:13.188957+0000","last_undegraded":"2024-04-25T05:47:13.188957+0000","last_fullsized":"2024-04-25T05:47:13.188957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.388440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.388440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:09.383423+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":920,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171634+0000","last_change":"2024-04-25T05:46:52.649991+0000","last_active":"2024-04-25T05:47:04.171634+0000","last_peered":"2024-04-25T05:47:04.171634+0000","last_clean":"2024-04-25T05:47:04.171634+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:47:04.171634+0000","last_undegraded":"2024-04-25T05:47:04.171634+0000","last_fullsized":"2024-04-25T05:47:04.171634+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:52.649850+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.649850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.101172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"401'3627","reported_seq":3663,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.138921+0000","last_change":"2024-04-25T05:46:37.986465+0000","last_active":"2024-04-25T05:47:13.138921+0000","last_peered":"2024-04-25T05:47:13.138921+0000","last_clean":"2024-04-25T05:47:13.138921+0000","last_became_active":"2024-04-25T05:46:37.986157+0000","last_became_peered":"2024-04-25T05:46:37.986157+0000","last_unstale":"2024-04-25T05:47:13.138921+0000","last_undegraded":"2024-04-25T05:47:13.138921+0000","last_fullsized":"2024-04-25T05:47:13.138921+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:03.585583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237699072,"num_objects":3627,"num_object_clones":0,"num_object_copies":10881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3627,"num_write_kb":232128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180494+0000","last_change":"2024-04-25T05:46:34.986045+0000","last_active":"2024-04-25T05:47:04.180494+0000","last_peered":"2024-04-25T05:47:04.180494+0000","last_clean":"2024-04-25T05:47:04.180494+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:47:04.180494+0000","last_undegraded":"2024-04-25T05:47:04.180494+0000","last_fullsized":"2024-04-25T05:47:04.180494+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:08.268145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:08.268145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":56,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180392+0000","last_change":"2024-04-25T05:46:35.019569+0000","last_active":"2024-04-25T05:47:04.180392+0000","last_peered":"2024-04-25T05:47:04.180392+0000","last_clean":"2024-04-25T05:47:04.180392+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:47:04.180392+0000","last_undegraded":"2024-04-25T05:47:04.180392+0000","last_fullsized":"2024-04-25T05:47:04.180392+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:44:18.440137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.440137+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1543,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:11.519577+0000","last_change":"2024-04-25T05:44:28.208035+0000","last_active":"2024-04-25T05:47:11.519577+0000","last_peered":"2024-04-25T05:47:11.519577+0000","last_clean":"2024-04-25T05:47:11.519577+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:47:11.519577+0000","last_undegraded":"2024-04-25T05:47:11.519577+0000","last_fullsized":"2024-04-25T05:47:11.519577+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:44:12.006440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.006440+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:12.839687+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":776,"num_read_kb":771,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":341,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.168293+0000","last_change":"2024-04-25T05:46:34.967478+0000","last_active":"2024-04-25T05:47:04.168293+0000","last_peered":"2024-04-25T05:47:04.168293+0000","last_clean":"2024-04-25T05:47:04.168293+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:47:04.168293+0000","last_undegraded":"2024-04-25T05:47:04.168293+0000","last_fullsized":"2024-04-25T05:47:04.168293+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:30.434101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.434101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:07.585242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":718,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188180+0000","last_change":"2024-04-25T05:46:34.975022+0000","last_active":"2024-04-25T05:47:13.188180+0000","last_peered":"2024-04-25T05:47:13.188180+0000","last_clean":"2024-04-25T05:47:13.188180+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:47:13.188180+0000","last_undegraded":"2024-04-25T05:47:13.188180+0000","last_fullsized":"2024-04-25T05:47:13.188180+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:36.702085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.702085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":900,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187826+0000","last_change":"2024-04-25T05:46:43.418452+0000","last_active":"2024-04-25T05:47:13.187826+0000","last_peered":"2024-04-25T05:47:13.187826+0000","last_clean":"2024-04-25T05:47:13.187826+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:47:13.187826+0000","last_undegraded":"2024-04-25T05:47:13.187826+0000","last_fullsized":"2024-04-25T05:47:13.187826+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:43.418024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.418024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.516553+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"401'3656","reported_seq":3694,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.941536+0000","last_change":"2024-04-25T05:46:37.985455+0000","last_active":"2024-04-25T05:47:12.941536+0000","last_peered":"2024-04-25T05:47:12.941536+0000","last_clean":"2024-04-25T05:47:12.941536+0000","last_became_active":"2024-04-25T05:46:37.985141+0000","last_became_peered":"2024-04-25T05:46:37.985141+0000","last_unstale":"2024-04-25T05:47:12.941536+0000","last_undegraded":"2024-04-25T05:47:12.941536+0000","last_fullsized":"2024-04-25T05:47:12.941536+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.374506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239599616,"num_objects":3656,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3656,"num_write_kb":233984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":738,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172880+0000","last_change":"2024-04-25T05:46:34.982419+0000","last_active":"2024-04-25T05:47:04.172880+0000","last_peered":"2024-04-25T05:47:04.172880+0000","last_clean":"2024-04-25T05:47:04.172880+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:47:04.172880+0000","last_undegraded":"2024-04-25T05:47:04.172880+0000","last_fullsized":"2024-04-25T05:47:04.172880+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.090299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.090299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870936+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":559,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172175+0000","last_change":"2024-04-25T05:45:38.816178+0000","last_active":"2024-04-25T05:47:04.172175+0000","last_peered":"2024-04-25T05:47:04.172175+0000","last_clean":"2024-04-25T05:47:04.172175+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:47:04.172175+0000","last_undegraded":"2024-04-25T05:47:04.172175+0000","last_fullsized":"2024-04-25T05:47:04.172175+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:45:38.815913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.815913+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:09.815258+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":734,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197489+0000","last_change":"2024-04-25T05:46:34.970673+0000","last_active":"2024-04-25T05:47:13.197489+0000","last_peered":"2024-04-25T05:47:13.197489+0000","last_clean":"2024-04-25T05:47:13.197489+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:47:13.197489+0000","last_undegraded":"2024-04-25T05:47:13.197489+0000","last_fullsized":"2024-04-25T05:47:13.197489+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.811914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.811914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1033,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172714+0000","last_change":"2024-04-25T05:46:06.483237+0000","last_active":"2024-04-25T05:47:04.172714+0000","last_peered":"2024-04-25T05:47:04.172714+0000","last_clean":"2024-04-25T05:47:04.172714+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:47:04.172714+0000","last_undegraded":"2024-04-25T05:47:04.172714+0000","last_fullsized":"2024-04-25T05:47:04.172714+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:06.483108+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.483108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:33.443264+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187823+0000","last_change":"2024-04-25T05:47:02.157593+0000","last_active":"2024-04-25T05:47:13.187823+0000","last_peered":"2024-04-25T05:47:13.187823+0000","last_clean":"2024-04-25T05:47:13.187823+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:47:13.187823+0000","last_undegraded":"2024-04-25T05:47:13.187823+0000","last_fullsized":"2024-04-25T05:47:13.187823+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:37.037435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.037435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"401'3569","reported_seq":3605,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.888336+0000","last_change":"2024-04-25T05:46:37.984020+0000","last_active":"2024-04-25T05:47:12.888336+0000","last_peered":"2024-04-25T05:47:12.888336+0000","last_clean":"2024-04-25T05:47:12.888336+0000","last_became_active":"2024-04-25T05:46:37.983253+0000","last_became_peered":"2024-04-25T05:46:37.983253+0000","last_unstale":"2024-04-25T05:47:12.888336+0000","last_undegraded":"2024-04-25T05:47:12.888336+0000","last_fullsized":"2024-04-25T05:47:12.888336+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3569,"log_dups_size":0,"ondisk_log_size":3569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:53.170282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233897984,"num_objects":3569,"num_object_clones":0,"num_object_copies":10707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3569,"num_write_kb":228416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":64,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179492+0000","last_change":"2024-04-25T05:46:34.971368+0000","last_active":"2024-04-25T05:47:04.179492+0000","last_peered":"2024-04-25T05:47:04.179492+0000","last_clean":"2024-04-25T05:47:04.179492+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:47:04.179492+0000","last_undegraded":"2024-04-25T05:47:04.179492+0000","last_fullsized":"2024-04-25T05:47:04.179492+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.149849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.149849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":857,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187035+0000","last_change":"2024-04-25T05:46:35.010157+0000","last_active":"2024-04-25T05:47:13.187035+0000","last_peered":"2024-04-25T05:47:13.187035+0000","last_clean":"2024-04-25T05:47:13.187035+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:47:13.187035+0000","last_undegraded":"2024-04-25T05:47:13.187035+0000","last_fullsized":"2024-04-25T05:47:13.187035+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'25","last_scrub_stamp":"2024-04-25T05:43:22.350489+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:43:22.350489+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.178538+0000","last_change":"2024-04-25T05:46:50.947463+0000","last_active":"2024-04-25T05:47:04.178538+0000","last_peered":"2024-04-25T05:47:04.178538+0000","last_clean":"2024-04-25T05:47:04.178538+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:47:04.178538+0000","last_undegraded":"2024-04-25T05:47:04.178538+0000","last_fullsized":"2024-04-25T05:47:04.178538+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.947345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.947345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:59.498121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":754,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187110+0000","last_change":"2024-04-25T05:46:19.992163+0000","last_active":"2024-04-25T05:47:13.187110+0000","last_peered":"2024-04-25T05:47:13.187110+0000","last_clean":"2024-04-25T05:47:13.187110+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:47:13.187110+0000","last_undegraded":"2024-04-25T05:47:13.187110+0000","last_fullsized":"2024-04-25T05:47:13.187110+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":716,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171571+0000","last_change":"2024-04-25T05:43:30.152257+0000","last_active":"2024-04-25T05:47:04.171571+0000","last_peered":"2024-04-25T05:47:04.171571+0000","last_clean":"2024-04-25T05:47:04.171571+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:47:04.171571+0000","last_undegraded":"2024-04-25T05:47:04.171571+0000","last_fullsized":"2024-04-25T05:47:04.171571+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:43:30.152081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870830+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1041,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170507+0000","last_change":"2024-04-25T05:46:23.137608+0000","last_active":"2024-04-25T05:47:04.170507+0000","last_peered":"2024-04-25T05:47:04.170507+0000","last_clean":"2024-04-25T05:47:04.170507+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:47:04.170507+0000","last_undegraded":"2024-04-25T05:47:04.170507+0000","last_fullsized":"2024-04-25T05:47:04.170507+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"401'3484","reported_seq":3522,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.940318+0000","last_change":"2024-04-25T05:46:37.985580+0000","last_active":"2024-04-25T05:47:12.940318+0000","last_peered":"2024-04-25T05:47:12.940318+0000","last_clean":"2024-04-25T05:47:12.940318+0000","last_became_active":"2024-04-25T05:46:37.985031+0000","last_became_peered":"2024-04-25T05:46:37.985031+0000","last_unstale":"2024-04-25T05:47:12.940318+0000","last_undegraded":"2024-04-25T05:47:12.940318+0000","last_fullsized":"2024-04-25T05:47:12.940318+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3484,"log_dups_size":0,"ondisk_log_size":3484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:45.128788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228327424,"num_objects":3484,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":222976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":343,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187605+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:47:13.187605+0000","last_peered":"2024-04-25T05:47:13.187605+0000","last_clean":"2024-04-25T05:47:13.187605+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:47:13.187605+0000","last_undegraded":"2024-04-25T05:47:13.187605+0000","last_fullsized":"2024-04-25T05:47:13.187605+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1023,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188308+0000","last_change":"2024-04-25T05:46:52.415241+0000","last_active":"2024-04-25T05:47:13.188308+0000","last_peered":"2024-04-25T05:47:13.188308+0000","last_clean":"2024-04-25T05:47:13.188308+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:47:13.188308+0000","last_undegraded":"2024-04-25T05:47:13.188308+0000","last_fullsized":"2024-04-25T05:47:13.188308+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:46:52.415149+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.415149+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:12.643185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":511,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172309+0000","last_change":"2024-04-25T05:46:50.698635+0000","last_active":"2024-04-25T05:47:04.172309+0000","last_peered":"2024-04-25T05:47:04.172309+0000","last_clean":"2024-04-25T05:47:04.172309+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:47:04.172309+0000","last_undegraded":"2024-04-25T05:47:04.172309+0000","last_fullsized":"2024-04-25T05:47:04.172309+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.698507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.698507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:59.959496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":54,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.181108+0000","last_change":"2024-04-25T05:46:49.868732+0000","last_active":"2024-04-25T05:47:04.181108+0000","last_peered":"2024-04-25T05:47:04.181108+0000","last_clean":"2024-04-25T05:47:04.181108+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:47:04.181108+0000","last_undegraded":"2024-04-25T05:47:04.181108+0000","last_fullsized":"2024-04-25T05:47:04.181108+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:46:49.868648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.868648+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.090373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":698,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.200545+0000","last_change":"2024-04-25T05:46:16.748067+0000","last_active":"2024-04-25T05:47:13.200545+0000","last_peered":"2024-04-25T05:47:13.200545+0000","last_clean":"2024-04-25T05:47:13.200545+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:47:13.200545+0000","last_undegraded":"2024-04-25T05:47:13.200545+0000","last_fullsized":"2024-04-25T05:47:13.200545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.747963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.747963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.649984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":742,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179479+0000","last_change":"2024-04-25T05:46:37.982952+0000","last_active":"2024-04-25T05:47:04.179479+0000","last_peered":"2024-04-25T05:47:04.179479+0000","last_clean":"2024-04-25T05:47:04.179479+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:47:04.179479+0000","last_undegraded":"2024-04-25T05:47:04.179479+0000","last_fullsized":"2024-04-25T05:47:04.179479+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:11.595125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:11.595125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:14.254842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"401'3672","reported_seq":3708,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.105059+0000","last_change":"2024-04-25T05:46:37.986363+0000","last_active":"2024-04-25T05:47:13.105059+0000","last_peered":"2024-04-25T05:47:13.105059+0000","last_clean":"2024-04-25T05:47:13.105059+0000","last_became_active":"2024-04-25T05:46:37.986037+0000","last_became_peered":"2024-04-25T05:46:37.986037+0000","last_unstale":"2024-04-25T05:47:13.105059+0000","last_undegraded":"2024-04-25T05:47:13.105059+0000","last_fullsized":"2024-04-25T05:47:13.105059+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:06.423561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240648192,"num_objects":3672,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":235008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":428,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187081+0000","last_change":"2024-04-25T05:44:28.239758+0000","last_active":"2024-04-25T05:47:13.187081+0000","last_peered":"2024-04-25T05:47:13.187081+0000","last_clean":"2024-04-25T05:47:13.187081+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:47:13.187081+0000","last_undegraded":"2024-04-25T05:47:13.187081+0000","last_fullsized":"2024-04-25T05:47:13.187081+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'20","last_scrub_stamp":"2024-04-25T05:43:16.552113+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:43:16.552113+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179419+0000","last_change":"2024-04-25T05:46:34.977874+0000","last_active":"2024-04-25T05:47:04.179419+0000","last_peered":"2024-04-25T05:47:04.179419+0000","last_clean":"2024-04-25T05:47:04.179419+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:47:04.179419+0000","last_undegraded":"2024-04-25T05:47:04.179419+0000","last_fullsized":"2024-04-25T05:47:04.179419+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.178993+0000","last_change":"2024-04-25T05:46:19.996324+0000","last_active":"2024-04-25T05:47:04.178993+0000","last_peered":"2024-04-25T05:47:04.178993+0000","last_clean":"2024-04-25T05:47:04.178993+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:47:04.178993+0000","last_undegraded":"2024-04-25T05:47:04.178993+0000","last_fullsized":"2024-04-25T05:47:04.178993+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":333,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171016+0000","last_change":"2024-04-25T05:46:19.989240+0000","last_active":"2024-04-25T05:47:04.171016+0000","last_peered":"2024-04-25T05:47:04.171016+0000","last_clean":"2024-04-25T05:47:04.171016+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:47:04.171016+0000","last_undegraded":"2024-04-25T05:47:04.171016+0000","last_fullsized":"2024-04-25T05:47:04.171016+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:42:54.545798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.545798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"402'3665","reported_seq":3705,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.797576+0000","last_change":"2024-04-25T05:46:37.979066+0000","last_active":"2024-04-25T05:47:13.797576+0000","last_peered":"2024-04-25T05:47:13.797576+0000","last_clean":"2024-04-25T05:47:13.797576+0000","last_became_active":"2024-04-25T05:46:37.978316+0000","last_became_peered":"2024-04-25T05:46:37.978316+0000","last_unstale":"2024-04-25T05:47:13.797576+0000","last_undegraded":"2024-04-25T05:47:13.797576+0000","last_fullsized":"2024-04-25T05:47:13.797576+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:49.031349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":485,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.174558+0000","last_change":"2024-04-25T05:46:06.689035+0000","last_active":"2024-04-25T05:47:04.174558+0000","last_peered":"2024-04-25T05:47:04.174558+0000","last_clean":"2024-04-25T05:47:04.174558+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:47:04.174558+0000","last_undegraded":"2024-04-25T05:47:04.174558+0000","last_fullsized":"2024-04-25T05:47:04.174558+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'41","last_scrub_stamp":"2024-04-25T05:46:06.688883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.688883+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:19.872066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":769,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171413+0000","last_change":"2024-04-25T05:46:45.682570+0000","last_active":"2024-04-25T05:47:04.171413+0000","last_peered":"2024-04-25T05:47:04.171413+0000","last_clean":"2024-04-25T05:47:04.171413+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:47:04.171413+0000","last_undegraded":"2024-04-25T05:47:04.171413+0000","last_fullsized":"2024-04-25T05:47:04.171413+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:46:45.682441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.682441+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:12.440746+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":64,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.178641+0000","last_change":"2024-04-25T05:46:34.972681+0000","last_active":"2024-04-25T05:47:04.178641+0000","last_peered":"2024-04-25T05:47:04.178641+0000","last_clean":"2024-04-25T05:47:04.178641+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:47:04.178641+0000","last_undegraded":"2024-04-25T05:47:04.178641+0000","last_fullsized":"2024-04-25T05:47:04.178641+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:03.424113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:03.424113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":716,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172506+0000","last_change":"2024-04-25T05:46:19.987220+0000","last_active":"2024-04-25T05:47:04.172506+0000","last_peered":"2024-04-25T05:47:04.172506+0000","last_clean":"2024-04-25T05:47:04.172506+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:47:04.172506+0000","last_undegraded":"2024-04-25T05:47:04.172506+0000","last_fullsized":"2024-04-25T05:47:04.172506+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:00.144906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:00.144906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:12.839716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"401'3657","reported_seq":3693,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.950880+0000","last_change":"2024-04-25T05:46:37.986817+0000","last_active":"2024-04-25T05:47:12.950880+0000","last_peered":"2024-04-25T05:47:12.950880+0000","last_clean":"2024-04-25T05:47:12.950880+0000","last_became_active":"2024-04-25T05:46:37.983489+0000","last_became_peered":"2024-04-25T05:46:37.983489+0000","last_unstale":"2024-04-25T05:47:12.950880+0000","last_undegraded":"2024-04-25T05:47:12.950880+0000","last_fullsized":"2024-04-25T05:47:12.950880+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3657,"log_dups_size":0,"ondisk_log_size":3657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.028685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239665152,"num_objects":3657,"num_object_clones":0,"num_object_copies":10971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3657,"num_write_kb":234048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":64,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179131+0000","last_change":"2024-04-25T05:46:34.971470+0000","last_active":"2024-04-25T05:47:04.179131+0000","last_peered":"2024-04-25T05:47:04.179131+0000","last_clean":"2024-04-25T05:47:04.179131+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:47:04.179131+0000","last_undegraded":"2024-04-25T05:47:04.179131+0000","last_fullsized":"2024-04-25T05:47:04.179131+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:29.464845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:29.464845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":367,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188489+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:47:13.188489+0000","last_peered":"2024-04-25T05:47:13.188489+0000","last_clean":"2024-04-25T05:47:13.188489+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:47:13.188489+0000","last_undegraded":"2024-04-25T05:47:13.188489+0000","last_fullsized":"2024-04-25T05:47:13.188489+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":723,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171572+0000","last_change":"2024-04-25T05:46:46.685480+0000","last_active":"2024-04-25T05:47:04.171572+0000","last_peered":"2024-04-25T05:47:04.171572+0000","last_clean":"2024-04-25T05:47:04.171572+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:47:04.171572+0000","last_undegraded":"2024-04-25T05:47:04.171572+0000","last_fullsized":"2024-04-25T05:47:04.171572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.685365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.685365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:57.792441+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":709,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.198177+0000","last_change":"2024-04-25T05:46:52.779088+0000","last_active":"2024-04-25T05:47:13.198177+0000","last_peered":"2024-04-25T05:47:13.198177+0000","last_clean":"2024-04-25T05:47:13.198177+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:47:13.198177+0000","last_undegraded":"2024-04-25T05:47:13.198177+0000","last_fullsized":"2024-04-25T05:47:13.198177+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:52.779013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.779013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.137766+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"401'3703","reported_seq":3739,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:12.893569+0000","last_change":"2024-04-25T05:46:37.981240+0000","last_active":"2024-04-25T05:47:12.893569+0000","last_peered":"2024-04-25T05:47:12.893569+0000","last_clean":"2024-04-25T05:47:12.893569+0000","last_became_active":"2024-04-25T05:46:37.980913+0000","last_became_peered":"2024-04-25T05:46:37.980913+0000","last_unstale":"2024-04-25T05:47:12.893569+0000","last_undegraded":"2024-04-25T05:47:12.893569+0000","last_fullsized":"2024-04-25T05:47:12.893569+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":3703,"log_dups_size":0,"ondisk_log_size":3703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:43.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242679808,"num_objects":3703,"num_object_clones":0,"num_object_copies":11109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":236992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":907,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170593+0000","last_change":"2024-04-25T05:46:47.674917+0000","last_active":"2024-04-25T05:47:04.170593+0000","last_peered":"2024-04-25T05:47:04.170593+0000","last_clean":"2024-04-25T05:47:04.170593+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:47:04.170593+0000","last_undegraded":"2024-04-25T05:47:04.170593+0000","last_fullsized":"2024-04-25T05:47:04.170593+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:46:47.674789+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.674789+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.201644+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":62,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179918+0000","last_change":"2024-04-25T05:46:35.009698+0000","last_active":"2024-04-25T05:47:04.179918+0000","last_peered":"2024-04-25T05:47:04.179918+0000","last_clean":"2024-04-25T05:47:04.179918+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:47:04.179918+0000","last_undegraded":"2024-04-25T05:47:04.179918+0000","last_fullsized":"2024-04-25T05:47:04.179918+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'27","last_scrub_stamp":"2024-04-25T05:42:54.961020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:42:54.961020+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":763,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172045+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:47:04.172045+0000","last_peered":"2024-04-25T05:47:04.172045+0000","last_clean":"2024-04-25T05:47:04.172045+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:47:04.172045+0000","last_undegraded":"2024-04-25T05:47:04.172045+0000","last_fullsized":"2024-04-25T05:47:04.172045+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.181306+0000","last_change":"2024-04-25T05:46:34.973794+0000","last_active":"2024-04-25T05:47:04.181306+0000","last_peered":"2024-04-25T05:47:04.181306+0000","last_clean":"2024-04-25T05:47:04.181306+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:47:04.181306+0000","last_undegraded":"2024-04-25T05:47:04.181306+0000","last_fullsized":"2024-04-25T05:47:04.181306+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:10.692888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.692888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:24.236919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":735,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.202171+0000","last_change":"2024-04-25T05:44:12.039068+0000","last_active":"2024-04-25T05:47:13.202171+0000","last_peered":"2024-04-25T05:47:13.202171+0000","last_clean":"2024-04-25T05:47:13.202171+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:47:13.202171+0000","last_undegraded":"2024-04-25T05:47:13.202171+0000","last_fullsized":"2024-04-25T05:47:13.202171+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:44:12.038920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.038920+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474985+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":469,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170523+0000","last_change":"2024-04-25T05:46:47.655135+0000","last_active":"2024-04-25T05:47:04.170523+0000","last_peered":"2024-04-25T05:47:04.170523+0000","last_clean":"2024-04-25T05:47:04.170523+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:47:04.170523+0000","last_undegraded":"2024-04-25T05:47:04.170523+0000","last_fullsized":"2024-04-25T05:47:04.170523+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.655013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.655013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:48.222095+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180297+0000","last_change":"2024-04-25T05:46:34.988515+0000","last_active":"2024-04-25T05:47:04.180297+0000","last_peered":"2024-04-25T05:47:04.180297+0000","last_clean":"2024-04-25T05:47:04.180297+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:47:04.180297+0000","last_undegraded":"2024-04-25T05:47:04.180297+0000","last_fullsized":"2024-04-25T05:47:04.180297+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:19.931173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.931173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":715,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171092+0000","last_change":"2024-04-25T05:44:01.145723+0000","last_active":"2024-04-25T05:47:04.171092+0000","last_peered":"2024-04-25T05:47:04.171092+0000","last_clean":"2024-04-25T05:47:04.171092+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:47:04.171092+0000","last_undegraded":"2024-04-25T05:47:04.171092+0000","last_fullsized":"2024-04-25T05:47:04.171092+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:01.145562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:01.145562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870909+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":424,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.200416+0000","last_change":"2024-04-25T05:46:20.806968+0000","last_active":"2024-04-25T05:47:13.200416+0000","last_peered":"2024-04-25T05:47:13.200416+0000","last_clean":"2024-04-25T05:47:13.200416+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:47:13.200416+0000","last_undegraded":"2024-04-25T05:47:13.200416+0000","last_fullsized":"2024-04-25T05:47:13.200416+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:46:20.806869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:20.806869+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.735644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":733,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170757+0000","last_change":"2024-04-25T05:43:34.099370+0000","last_active":"2024-04-25T05:47:04.170757+0000","last_peered":"2024-04-25T05:47:04.170757+0000","last_clean":"2024-04-25T05:47:04.170757+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:47:04.170757+0000","last_undegraded":"2024-04-25T05:47:04.170757+0000","last_fullsized":"2024-04-25T05:47:04.170757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:34.099150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:34.099150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":693,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.201997+0000","last_change":"2024-04-25T05:44:18.947820+0000","last_active":"2024-04-25T05:47:13.201997+0000","last_peered":"2024-04-25T05:47:13.201997+0000","last_clean":"2024-04-25T05:47:13.201997+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:47:13.201997+0000","last_undegraded":"2024-04-25T05:47:13.201997+0000","last_fullsized":"2024-04-25T05:47:13.201997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:18.947682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:18.947682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475018+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.178695+0000","last_change":"2024-04-25T05:46:19.984536+0000","last_active":"2024-04-25T05:47:04.178695+0000","last_peered":"2024-04-25T05:47:04.178695+0000","last_clean":"2024-04-25T05:47:04.178695+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:47:04.178695+0000","last_undegraded":"2024-04-25T05:47:04.178695+0000","last_fullsized":"2024-04-25T05:47:04.178695+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.695090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.695090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:28.011559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":54,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179436+0000","last_change":"2024-04-25T05:46:35.006236+0000","last_active":"2024-04-25T05:47:04.179436+0000","last_peered":"2024-04-25T05:47:04.179436+0000","last_clean":"2024-04-25T05:47:04.179436+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:47:04.179436+0000","last_undegraded":"2024-04-25T05:47:04.179436+0000","last_fullsized":"2024-04-25T05:47:04.179436+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:37.317498+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.317498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":327,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170254+0000","last_change":"2024-04-25T05:44:28.172179+0000","last_active":"2024-04-25T05:47:04.170254+0000","last_peered":"2024-04-25T05:47:04.170254+0000","last_clean":"2024-04-25T05:47:04.170254+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:47:04.170254+0000","last_undegraded":"2024-04-25T05:47:04.170254+0000","last_fullsized":"2024-04-25T05:47:04.170254+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.458095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.458095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":880,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197662+0000","last_change":"2024-04-25T05:46:20.022186+0000","last_active":"2024-04-25T05:47:13.197662+0000","last_peered":"2024-04-25T05:47:13.197662+0000","last_clean":"2024-04-25T05:47:13.197662+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:47:13.197662+0000","last_undegraded":"2024-04-25T05:47:13.197662+0000","last_fullsized":"2024-04-25T05:47:13.197662+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'9","last_scrub_stamp":"2024-04-25T05:44:11.019737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:11.019737+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474788+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":64,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179802+0000","last_change":"2024-04-25T05:46:34.972368+0000","last_active":"2024-04-25T05:47:04.179802+0000","last_peered":"2024-04-25T05:47:04.179802+0000","last_clean":"2024-04-25T05:47:04.179802+0000","last_became_active":"2024-04-25T05:46:34.971901+0000","last_became_peered":"2024-04-25T05:46:34.971901+0000","last_unstale":"2024-04-25T05:47:04.179802+0000","last_undegraded":"2024-04-25T05:47:04.179802+0000","last_fullsized":"2024-04-25T05:47:04.179802+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:04.540333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:04.540333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":759,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197245+0000","last_change":"2024-04-25T05:45:32.015466+0000","last_active":"2024-04-25T05:47:13.197245+0000","last_peered":"2024-04-25T05:47:13.197245+0000","last_clean":"2024-04-25T05:47:13.197245+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:47:13.197245+0000","last_undegraded":"2024-04-25T05:47:13.197245+0000","last_fullsized":"2024-04-25T05:47:13.197245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:45:32.015242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.015242+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475132+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":727,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170851+0000","last_change":"2024-04-25T05:46:19.985933+0000","last_active":"2024-04-25T05:47:04.170851+0000","last_peered":"2024-04-25T05:47:04.170851+0000","last_clean":"2024-04-25T05:47:04.170851+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:47:04.170851+0000","last_undegraded":"2024-04-25T05:47:04.170851+0000","last_fullsized":"2024-04-25T05:47:04.170851+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.015315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.015315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":53,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179679+0000","last_change":"2024-04-25T05:46:34.997195+0000","last_active":"2024-04-25T05:47:04.179679+0000","last_peered":"2024-04-25T05:47:04.179679+0000","last_clean":"2024-04-25T05:47:04.179679+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:47:04.179679+0000","last_undegraded":"2024-04-25T05:47:04.179679+0000","last_fullsized":"2024-04-25T05:47:04.179679+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:46:19.516621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:19.516621+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:31.276451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":327,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171358+0000","last_change":"2024-04-25T05:44:28.179559+0000","last_active":"2024-04-25T05:47:04.171358+0000","last_peered":"2024-04-25T05:47:04.171358+0000","last_clean":"2024-04-25T05:47:04.171358+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:47:04.171358+0000","last_undegraded":"2024-04-25T05:47:04.171358+0000","last_fullsized":"2024-04-25T05:47:04.171358+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:24.340557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:24.340557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":853,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.168137+0000","last_change":"2024-04-25T05:46:36.988922+0000","last_active":"2024-04-25T05:47:04.168137+0000","last_peered":"2024-04-25T05:47:04.168137+0000","last_clean":"2024-04-25T05:47:04.168137+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:47:04.168137+0000","last_undegraded":"2024-04-25T05:47:04.168137+0000","last_fullsized":"2024-04-25T05:47:04.168137+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:46:13.591210+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:46:13.591210+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:38.574846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":395,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172666+0000","last_change":"2024-04-25T05:45:33.294954+0000","last_active":"2024-04-25T05:47:04.172666+0000","last_peered":"2024-04-25T05:47:04.172666+0000","last_clean":"2024-04-25T05:47:04.172666+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:47:04.172666+0000","last_undegraded":"2024-04-25T05:47:04.172666+0000","last_fullsized":"2024-04-25T05:47:04.172666+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.857697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":716,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.186878+0000","last_change":"2024-04-25T05:46:34.974010+0000","last_active":"2024-04-25T05:47:13.186878+0000","last_peered":"2024-04-25T05:47:13.186878+0000","last_clean":"2024-04-25T05:47:13.186878+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:47:13.186878+0000","last_undegraded":"2024-04-25T05:47:13.186878+0000","last_fullsized":"2024-04-25T05:47:13.186878+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:18.283932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.283932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":741,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195917+0000","last_change":"2024-04-25T05:46:53.519364+0000","last_active":"2024-04-25T05:47:13.195917+0000","last_peered":"2024-04-25T05:47:13.195917+0000","last_clean":"2024-04-25T05:47:13.195917+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:47:13.195917+0000","last_undegraded":"2024-04-25T05:47:13.195917+0000","last_fullsized":"2024-04-25T05:47:13.195917+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.519223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.519223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:57.533661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":271,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188664+0000","last_change":"2024-04-25T05:46:20.039638+0000","last_active":"2024-04-25T05:47:13.188664+0000","last_peered":"2024-04-25T05:47:13.188664+0000","last_clean":"2024-04-25T05:47:13.188664+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:47:13.188664+0000","last_undegraded":"2024-04-25T05:47:13.188664+0000","last_fullsized":"2024-04-25T05:47:13.188664+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:46:07.665439+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.665439+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:29.578540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":723,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171198+0000","last_change":"2024-04-25T05:46:44.682768+0000","last_active":"2024-04-25T05:47:04.171198+0000","last_peered":"2024-04-25T05:47:04.171198+0000","last_clean":"2024-04-25T05:47:04.171198+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:47:04.171198+0000","last_undegraded":"2024-04-25T05:47:04.171198+0000","last_fullsized":"2024-04-25T05:47:04.171198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.682645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.682645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:10.110610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":717,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196377+0000","last_change":"2024-04-25T05:46:34.970569+0000","last_active":"2024-04-25T05:47:13.196377+0000","last_peered":"2024-04-25T05:47:13.196377+0000","last_clean":"2024-04-25T05:47:13.196377+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:47:13.196377+0000","last_undegraded":"2024-04-25T05:47:13.196377+0000","last_fullsized":"2024-04-25T05:47:13.196377+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:07.993472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.993472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474866+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179732+0000","last_change":"2024-04-25T05:46:53.839614+0000","last_active":"2024-04-25T05:47:04.179732+0000","last_peered":"2024-04-25T05:47:04.179732+0000","last_clean":"2024-04-25T05:47:04.179732+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:47:04.179732+0000","last_undegraded":"2024-04-25T05:47:04.179732+0000","last_fullsized":"2024-04-25T05:47:04.179732+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.839498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.839498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:11.237992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"320'25","reported_seq":788,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196091+0000","last_change":"2024-04-25T05:44:14.079319+0000","last_active":"2024-04-25T05:47:13.196091+0000","last_peered":"2024-04-25T05:47:13.196091+0000","last_clean":"2024-04-25T05:47:13.196091+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:47:13.196091+0000","last_undegraded":"2024-04-25T05:47:13.196091+0000","last_fullsized":"2024-04-25T05:47:13.196091+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:44:14.079158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.079158+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180128+0000","last_change":"2024-04-25T05:46:34.970402+0000","last_active":"2024-04-25T05:47:04.180128+0000","last_peered":"2024-04-25T05:47:04.180128+0000","last_clean":"2024-04-25T05:47:04.180128+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:47:04.180128+0000","last_undegraded":"2024-04-25T05:47:04.180128+0000","last_fullsized":"2024-04-25T05:47:04.180128+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:17.056470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:17.056470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":725,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.173003+0000","last_change":"2024-04-25T05:43:31.095252+0000","last_active":"2024-04-25T05:47:04.173003+0000","last_peered":"2024-04-25T05:47:04.173003+0000","last_clean":"2024-04-25T05:47:04.173003+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:47:04.173003+0000","last_undegraded":"2024-04-25T05:47:04.173003+0000","last_fullsized":"2024-04-25T05:47:04.173003+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:31.095134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:43:31.095134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":733,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197029+0000","last_change":"2024-04-25T05:46:19.986880+0000","last_active":"2024-04-25T05:47:13.197029+0000","last_peered":"2024-04-25T05:47:13.197029+0000","last_clean":"2024-04-25T05:47:13.197029+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:47:13.197029+0000","last_undegraded":"2024-04-25T05:47:13.197029+0000","last_fullsized":"2024-04-25T05:47:13.197029+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:17.841621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:46:17.841621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:39.909197+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":741,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195659+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:47:13.195659+0000","last_peered":"2024-04-25T05:47:13.195659+0000","last_clean":"2024-04-25T05:47:13.195659+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:47:13.195659+0000","last_undegraded":"2024-04-25T05:47:13.195659+0000","last_fullsized":"2024-04-25T05:47:13.195659+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475164+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180383+0000","last_change":"2024-04-25T05:46:34.978621+0000","last_active":"2024-04-25T05:47:04.180383+0000","last_peered":"2024-04-25T05:47:04.180383+0000","last_clean":"2024-04-25T05:47:04.180383+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:47:04.180383+0000","last_undegraded":"2024-04-25T05:47:04.180383+0000","last_fullsized":"2024-04-25T05:47:04.180383+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:26.399461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:43:26.399461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":431,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170442+0000","last_change":"2024-04-25T05:45:29.823062+0000","last_active":"2024-04-25T05:47:04.170442+0000","last_peered":"2024-04-25T05:47:04.170442+0000","last_clean":"2024-04-25T05:47:04.170442+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:47:04.170442+0000","last_undegraded":"2024-04-25T05:47:04.170442+0000","last_fullsized":"2024-04-25T05:47:04.170442+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:45:29.822937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:04.647711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":724,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170633+0000","last_change":"2024-04-25T05:46:20.027611+0000","last_active":"2024-04-25T05:47:04.170633+0000","last_peered":"2024-04-25T05:47:04.170633+0000","last_clean":"2024-04-25T05:47:04.170633+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:47:04.170633+0000","last_undegraded":"2024-04-25T05:47:04.170633+0000","last_fullsized":"2024-04-25T05:47:04.170633+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871231+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":551,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.173045+0000","last_change":"2024-04-25T05:46:44.699964+0000","last_active":"2024-04-25T05:47:04.173045+0000","last_peered":"2024-04-25T05:47:04.173045+0000","last_clean":"2024-04-25T05:47:04.173045+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:47:04.173045+0000","last_undegraded":"2024-04-25T05:47:04.173045+0000","last_fullsized":"2024-04-25T05:47:04.173045+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:46:44.699808+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.699808+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:10.283576+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":733,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.195589+0000","last_change":"2024-04-25T05:46:19.997617+0000","last_active":"2024-04-25T05:47:13.195589+0000","last_peered":"2024-04-25T05:47:13.195589+0000","last_clean":"2024-04-25T05:47:13.195589+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:47:13.195589+0000","last_undegraded":"2024-04-25T05:47:13.195589+0000","last_fullsized":"2024-04-25T05:47:13.195589+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:22.966386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:44:22.966386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475078+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":333,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171613+0000","last_change":"2024-04-25T05:46:19.988527+0000","last_active":"2024-04-25T05:47:04.171613+0000","last_peered":"2024-04-25T05:47:04.171613+0000","last_clean":"2024-04-25T05:47:04.171613+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:47:04.171613+0000","last_undegraded":"2024-04-25T05:47:04.171613+0000","last_fullsized":"2024-04-25T05:47:04.171613+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":715,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.186915+0000","last_change":"2024-04-25T05:46:34.974816+0000","last_active":"2024-04-25T05:47:13.186915+0000","last_peered":"2024-04-25T05:47:13.186915+0000","last_clean":"2024-04-25T05:47:13.186915+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:47:13.186915+0000","last_undegraded":"2024-04-25T05:47:13.186915+0000","last_fullsized":"2024-04-25T05:47:13.186915+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:43:21.410794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:43:21.410794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":400,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171698+0000","last_change":"2024-04-25T05:44:28.227641+0000","last_active":"2024-04-25T05:47:04.171698+0000","last_peered":"2024-04-25T05:47:04.171698+0000","last_clean":"2024-04-25T05:47:04.171698+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:47:04.171698+0000","last_undegraded":"2024-04-25T05:47:04.171698+0000","last_fullsized":"2024-04-25T05:47:04.171698+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'22","last_scrub_stamp":"2024-04-25T05:43:18.467028+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:43:18.467028+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.870754+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":718,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188598+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:47:13.188598+0000","last_peered":"2024-04-25T05:47:13.188598+0000","last_clean":"2024-04-25T05:47:13.188598+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:47:13.188598+0000","last_undegraded":"2024-04-25T05:47:13.188598+0000","last_fullsized":"2024-04-25T05:47:13.188598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":535,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.175019+0000","last_change":"2024-04-25T05:46:15.701376+0000","last_active":"2024-04-25T05:47:04.175019+0000","last_peered":"2024-04-25T05:47:04.175019+0000","last_clean":"2024-04-25T05:47:04.175019+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:47:04.175019+0000","last_undegraded":"2024-04-25T05:47:04.175019+0000","last_fullsized":"2024-04-25T05:47:04.175019+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:46:15.701250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:15.701250+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:30.567412+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":752,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.172773+0000","last_change":"2024-04-25T05:46:34.991626+0000","last_active":"2024-04-25T05:47:04.172773+0000","last_peered":"2024-04-25T05:47:04.172773+0000","last_clean":"2024-04-25T05:47:04.172773+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:47:04.172773+0000","last_undegraded":"2024-04-25T05:47:04.172773+0000","last_fullsized":"2024-04-25T05:47:04.172773+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:46:09.493093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.493093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:27.723926+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":741,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197018+0000","last_change":"2024-04-25T05:43:32.430131+0000","last_active":"2024-04-25T05:47:13.197018+0000","last_peered":"2024-04-25T05:47:13.197018+0000","last_clean":"2024-04-25T05:47:13.197018+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:47:13.197018+0000","last_undegraded":"2024-04-25T05:47:13.197018+0000","last_fullsized":"2024-04-25T05:47:13.197018+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-04-25T05:43:32.429992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:43:32.429992+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.474712+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":749,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.169340+0000","last_change":"2024-04-25T05:46:44.693255+0000","last_active":"2024-04-25T05:47:04.169340+0000","last_peered":"2024-04-25T05:47:04.169340+0000","last_clean":"2024-04-25T05:47:04.169340+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:47:04.169340+0000","last_undegraded":"2024-04-25T05:47:04.169340+0000","last_fullsized":"2024-04-25T05:47:04.169340+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.693144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.693144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:00.099880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":400,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197388+0000","last_change":"2024-04-25T05:46:47.610533+0000","last_active":"2024-04-25T05:47:13.197388+0000","last_peered":"2024-04-25T05:47:13.197388+0000","last_clean":"2024-04-25T05:47:13.197388+0000","last_became_active":"2024-04-25T05:46:20.009022+0000","last_became_peered":"2024-04-25T05:46:20.009022+0000","last_unstale":"2024-04-25T05:47:13.197388+0000","last_undegraded":"2024-04-25T05:47:13.197388+0000","last_fullsized":"2024-04-25T05:47:13.197388+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.610387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.610387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.887722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":722,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.202081+0000","last_change":"2024-04-25T05:46:49.931810+0000","last_active":"2024-04-25T05:47:13.202081+0000","last_peered":"2024-04-25T05:47:13.202081+0000","last_clean":"2024-04-25T05:47:13.202081+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:47:13.202081+0000","last_undegraded":"2024-04-25T05:47:13.202081+0000","last_fullsized":"2024-04-25T05:47:13.202081+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:49.930039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.930039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:55.913953+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":74,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.178528+0000","last_change":"2024-04-25T05:46:20.037905+0000","last_active":"2024-04-25T05:47:04.178528+0000","last_peered":"2024-04-25T05:47:04.178528+0000","last_clean":"2024-04-25T05:47:04.178528+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:47:04.178528+0000","last_undegraded":"2024-04-25T05:47:04.178528+0000","last_fullsized":"2024-04-25T05:47:04.178528+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'22","last_scrub_stamp":"2024-04-25T05:42:52.924624+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:42:52.924624+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.926026+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":203,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171037+0000","last_change":"2024-04-25T05:44:28.167551+0000","last_active":"2024-04-25T05:47:04.171037+0000","last_peered":"2024-04-25T05:47:04.171037+0000","last_clean":"2024-04-25T05:47:04.171037+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:47:04.171037+0000","last_undegraded":"2024-04-25T05:47:04.171037+0000","last_fullsized":"2024-04-25T05:47:04.171037+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:21.391598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:21.391598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":733,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.187485+0000","last_change":"2024-04-25T05:46:34.974900+0000","last_active":"2024-04-25T05:47:13.187485+0000","last_peered":"2024-04-25T05:47:13.187485+0000","last_clean":"2024-04-25T05:47:13.187485+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:47:13.187485+0000","last_undegraded":"2024-04-25T05:47:13.187485+0000","last_fullsized":"2024-04-25T05:47:13.187485+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":744,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.171826+0000","last_change":"2024-04-25T05:46:43.757487+0000","last_active":"2024-04-25T05:47:04.171826+0000","last_peered":"2024-04-25T05:47:04.171826+0000","last_clean":"2024-04-25T05:47:04.171826+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:47:04.171826+0000","last_undegraded":"2024-04-25T05:47:04.171826+0000","last_fullsized":"2024-04-25T05:47:04.171826+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:43.757378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.757378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.378972+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":459,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196269+0000","last_change":"2024-04-25T05:46:05.752045+0000","last_active":"2024-04-25T05:47:13.196269+0000","last_peered":"2024-04-25T05:47:13.196269+0000","last_clean":"2024-04-25T05:47:13.196269+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:47:13.196269+0000","last_undegraded":"2024-04-25T05:47:13.196269+0000","last_fullsized":"2024-04-25T05:47:13.196269+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:46:05.751904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:05.751904+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179528+0000","last_change":"2024-04-25T05:46:34.989054+0000","last_active":"2024-04-25T05:47:04.179528+0000","last_peered":"2024-04-25T05:47:04.179528+0000","last_clean":"2024-04-25T05:47:04.179528+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:47:04.179528+0000","last_undegraded":"2024-04-25T05:47:04.179528+0000","last_fullsized":"2024-04-25T05:47:04.179528+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:00.435460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:45:00.435460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":713,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170419+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:47:04.170419+0000","last_peered":"2024-04-25T05:47:04.170419+0000","last_clean":"2024-04-25T05:47:04.170419+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:47:04.170419+0000","last_undegraded":"2024-04-25T05:47:04.170419+0000","last_fullsized":"2024-04-25T05:47:04.170419+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:06.871341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179579+0000","last_change":"2024-04-25T05:46:34.985777+0000","last_active":"2024-04-25T05:47:04.179579+0000","last_peered":"2024-04-25T05:47:04.179579+0000","last_clean":"2024-04-25T05:47:04.179579+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:47:04.179579+0000","last_undegraded":"2024-04-25T05:47:04.179579+0000","last_fullsized":"2024-04-25T05:47:04.179579+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:38.767074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.767074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":516,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.197649+0000","last_change":"2024-04-25T05:46:20.046230+0000","last_active":"2024-04-25T05:47:13.197649+0000","last_peered":"2024-04-25T05:47:13.197649+0000","last_clean":"2024-04-25T05:47:13.197649+0000","last_became_active":"2024-04-25T05:46:20.009817+0000","last_became_peered":"2024-04-25T05:46:20.009817+0000","last_unstale":"2024-04-25T05:47:13.197649+0000","last_undegraded":"2024-04-25T05:47:13.197649+0000","last_fullsized":"2024-04-25T05:47:13.197649+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'28","last_scrub_stamp":"2024-04-25T05:46:10.858811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.858811+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:26.610333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.179542+0000","last_change":"2024-04-25T05:46:34.979633+0000","last_active":"2024-04-25T05:47:04.179542+0000","last_peered":"2024-04-25T05:47:04.179542+0000","last_clean":"2024-04-25T05:47:04.179542+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:47:04.179542+0000","last_undegraded":"2024-04-25T05:47:04.179542+0000","last_fullsized":"2024-04-25T05:47:04.179542+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:06.726223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:44:06.726223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180048+0000","last_change":"2024-04-25T05:46:34.979661+0000","last_active":"2024-04-25T05:47:04.180048+0000","last_peered":"2024-04-25T05:47:04.180048+0000","last_clean":"2024-04-25T05:47:04.180048+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:47:04.180048+0000","last_undegraded":"2024-04-25T05:47:04.180048+0000","last_fullsized":"2024-04-25T05:47:04.180048+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":343,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.375293+0000","last_change":"2024-04-25T05:47:13.375334+0000","last_active":"2024-04-25T05:47:13.375293+0000","last_peered":"2024-04-25T05:47:13.375293+0000","last_clean":"2024-04-25T05:47:13.375293+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:47:13.375293+0000","last_undegraded":"2024-04-25T05:47:13.375293+0000","last_fullsized":"2024-04-25T05:47:13.375293+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:13.375091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:13.375091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:19.270626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1006,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196131+0000","last_change":"2024-04-25T05:46:48.610335+0000","last_active":"2024-04-25T05:47:13.196131+0000","last_peered":"2024-04-25T05:47:13.196131+0000","last_clean":"2024-04-25T05:47:13.196131+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:47:13.196131+0000","last_undegraded":"2024-04-25T05:47:13.196131+0000","last_fullsized":"2024-04-25T05:47:13.196131+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:46:48.610157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.610157+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:07.926651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":734,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.200470+0000","last_change":"2024-04-25T05:46:19.997604+0000","last_active":"2024-04-25T05:47:13.200470+0000","last_peered":"2024-04-25T05:47:13.200470+0000","last_clean":"2024-04-25T05:47:13.200470+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:47:13.200470+0000","last_undegraded":"2024-04-25T05:47:13.200470+0000","last_fullsized":"2024-04-25T05:47:13.200470+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:16.995813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:44:16.995813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:15.475051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":682,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.170802+0000","last_change":"2024-04-25T05:45:32.772720+0000","last_active":"2024-04-25T05:47:04.170802+0000","last_peered":"2024-04-25T05:47:04.170802+0000","last_clean":"2024-04-25T05:47:04.170802+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:47:04.170802+0000","last_undegraded":"2024-04-25T05:47:04.170802+0000","last_fullsized":"2024-04-25T05:47:04.170802+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.772481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.772481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:04.647208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":709,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.196482+0000","last_change":"2024-04-25T05:46:46.830806+0000","last_active":"2024-04-25T05:47:13.196482+0000","last_peered":"2024-04-25T05:47:13.196482+0000","last_clean":"2024-04-25T05:47:13.196482+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:47:13.196482+0000","last_undegraded":"2024-04-25T05:47:13.196482+0000","last_fullsized":"2024-04-25T05:47:13.196482+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.830676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.830676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:01.793071+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":805,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.167333+0000","last_change":"2024-04-25T05:46:52.599923+0000","last_active":"2024-04-25T05:47:04.167333+0000","last_peered":"2024-04-25T05:47:04.167333+0000","last_clean":"2024-04-25T05:47:04.167333+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:47:04.167333+0000","last_undegraded":"2024-04-25T05:47:04.167333+0000","last_fullsized":"2024-04-25T05:47:04.167333+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:46:52.599775+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.599775+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:52.650570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":50,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.180265+0000","last_change":"2024-04-25T05:46:44.971898+0000","last_active":"2024-04-25T05:47:04.180265+0000","last_peered":"2024-04-25T05:47:04.180265+0000","last_clean":"2024-04-25T05:47:04.180265+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:47:04.180265+0000","last_undegraded":"2024-04-25T05:47:04.180265+0000","last_fullsized":"2024-04-25T05:47:04.180265+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.971755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.971755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:02.005304+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":727,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-04-25T05:47:13.188860+0000","last_change":"2024-04-25T05:46:34.974443+0000","last_active":"2024-04-25T05:47:13.188860+0000","last_peered":"2024-04-25T05:47:13.188860+0000","last_clean":"2024-04-25T05:47:13.188860+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:47:13.188860+0000","last_undegraded":"2024-04-25T05:47:13.188860+0000","last_fullsized":"2024-04-25T05:47:13.188860+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-04-25T05:47:04.181500+0000","last_change":"2024-04-25T05:46:34.988586+0000","last_active":"2024-04-25T05:47:04.181500+0000","last_peered":"2024-04-25T05:47:04.181500+0000","last_clean":"2024-04-25T05:47:04.181500+0000","last_became_active":"2024-04-25T05:46:34.986413+0000","last_became_peered":"2024-04-25T05:46:34.986413+0000","last_unstale":"2024-04-25T05:47:04.181500+0000","last_undegraded":"2024-04-25T05:47:04.181500+0000","last_fullsized":"2024-04-25T05:47:04.181500+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:39.734373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.734373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:05.578868+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3806724096,"num_objects":58086,"num_object_clones":0,"num_object_copies":174258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58086,"num_write_kb":3717504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11459231744,"data_stored":11459231744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58086,"ondisk_log_size":58086,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":776,"num_read_kb":771,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1394,"num_read_kb":1119,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":10814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":958,"ondisk_log_size":958,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070628,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1844536,"kb_used_data":1163632,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91900616,"statfs":{"total":95995035648,"available":94106230784,"internally_reserved":0,"allocated":1191559168,"data_stored":1190955247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697235914},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364968,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142028,"kb_used_data":1601348,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91603124,"statfs":{"total":95995035648,"available":93801598976,"internally_reserved":0,"allocated":1639780352,"data_stored":1639219903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553646534},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193364,"kb_used_data":1418056,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":91551788,"statfs":{"total":95995035648,"available":93749030912,"internally_reserved":0,"allocated":1452089344,"data_stored":1451466620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738567,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411516,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031076,"kb_used_data":1607004,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91714076,"statfs":{"total":95995035648,"available":93915213824,"internally_reserved":0,"allocated":1645572096,"data_stored":1644987117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2793092,"kb_used_data":1873400,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":90952060,"statfs":{"total":95995035648,"available":93134909440,"internally_reserved":0,"allocated":1918361600,"data_stored":1917755537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":941750727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986353,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2581800,"kb_used_data":2077216,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91163352,"statfs":{"total":95995035648,"available":93351272448,"internally_reserved":0,"allocated":2127069184,"data_stored":2126480245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692013,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1757016,"kb_used_data":1356560,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91988136,"statfs":{"total":95995035648,"available":94195851264,"internally_reserved":0,"allocated":1389117440,"data_stored":1388544833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1468399616,"data_stored":1468399616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1651572736,"data_stored":1651572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1933574144,"data_stored":1933574144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2144206848,"data_stored":2144206848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1416953856,"data_stored":1416953856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1650524160,"data_stored":1650524160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1194000384,"data_stored":1194000384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:47:19.787 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-04-25T05:47:19.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:19.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:19.887 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:19.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 66831 66815 109.868 73.0625 0.0179352 0.00909729 2024-04-25T05:47:19.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:19.997 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:19.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:20.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.130 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:20.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:20.231 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:20.254 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:20.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:20.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:20.397 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:20 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:47:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:20 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:47:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:20 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:47:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:20 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:47:20.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:20.532 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:20.570 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:47:20.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:20 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:47:20.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:20.673 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:20.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:20.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:20.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:20.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 15 68067 68052 109.033 77.3125 0.0207308 0.00916616 2024-04-25T05:47:20.945 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:20.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:20.946 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:21.047 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:21.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:21.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:21.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:21.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:21.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:47:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:47:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[27010]: pgmap v712: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:47:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:47:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:47:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:21 smithi043 ceph-mon[37492]: pgmap v712: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:47:21.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:21.481 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:21.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:21 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:47:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:21 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:47:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:21 smithi060 ceph-mon[30898]: pgmap v712: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:47:21.583 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:21.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:21.621 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:21.621 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:21.718 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:21.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:21.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:21.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:21.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:21.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:47:21.931816+0000 min lat: 0.0012506 max lat: 0.243151 avg lat: 0.00924628 2024-04-25T05:47:21.930 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:47:21.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 69213 69197 108.097 71.5625 0.00678081 0.00924628 2024-04-25T05:47:22.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:22.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:22.146 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:22.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:22.255 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:47:22.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:22.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:22.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:47:22.412 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:22.447 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:22 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:47:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:22 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:47:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:22.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:22.583 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:47:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:22.685 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:22.729 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:22.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:22.885 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:22.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:22.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 70446 70430 107.34 77.0625 0.00945082 0.00931108 2024-04-25T05:47:22.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:23.021 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:23.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:23.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:23.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:23.390 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:23.427 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:23.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:23.566 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: 4.1e scrub starts 2024-04-25T05:47:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: 4.1e scrub ok 2024-04-25T05:47:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: pgmap v713: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 584 B/s rd, 52 MiB/s wr, 835 op/s 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: osdmap e404: 8 total, 8 up, 7 in 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:47:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: 4.1e scrub starts 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: 4.1e scrub ok 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: pgmap v713: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 584 B/s rd, 52 MiB/s wr, 835 op/s 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: osdmap e404: 8 total, 8 up, 7 in 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:23.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:47:23.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:23 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:47:23.668 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:23.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:23.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: 4.1e scrub starts 2024-04-25T05:47:23.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: 4.1e scrub ok 2024-04-25T05:47:23.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: pgmap v713: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 584 B/s rd, 52 MiB/s wr, 835 op/s 2024-04-25T05:47:23.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:23.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1199032729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: osdmap e404: 8 total, 8 up, 7 in 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:47:23.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:23 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:47:23.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:23.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 15 71639 71624 106.56 74.625 0.0119628 0.00937903 2024-04-25T05:47:23.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:23.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:47:23.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:23.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:24.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:24.109 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:24.210 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:24.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:24.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:24.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:24.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:24.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[27010]: osdmap e405: 8 total, 8 up, 7 in 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[37492]: osdmap e405: 8 total, 8 up, 7 in 2024-04-25T05:47:24.572 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:47:24.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:47:24.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:47:24.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:24 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:47:24.618 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:24.655 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:24 smithi060 ceph-mon[30898]: osdmap e405: 8 total, 8 up, 7 in 2024-04-25T05:47:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:24 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:47:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:24 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:47:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:24 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:47:24.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:24 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:47:24.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:24.788 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:47:24.789 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:47:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:24.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:24.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:24.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:24.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:24.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 72961 72945 106.002 82.5625 0.0106645 0.00942873 2024-04-25T05:47:25.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:25.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:25.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:25.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:25.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:25.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:25.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:25.471 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:25.484 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:25.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: pgmap v716: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: pgmap v716: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:25.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:47:25.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:47:25.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:47:25.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:47:25.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:47:25.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:25 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:47:25.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:25.608 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:25.649 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:25.649 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: pgmap v716: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:47:25.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:25 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:47:25.710 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:25.748 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:25.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:25.881 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:25.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:25.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 74202 74186 105.356 77.5625 0.010247 0.00948629 2024-04-25T05:47:25.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:26.015 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:26.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:26.149 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:26.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:26.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:26.446 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: 5.13 scrub starts 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: 5.13 scrub ok 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: 3.1 deep-scrub starts 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: 3.1 deep-scrub ok 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.534 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: 5.13 scrub starts 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: 5.13 scrub ok 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: 3.1 deep-scrub starts 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: 3.1 deep-scrub ok 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.535 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:26 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:26.575 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:26.653 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:26.653 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:26.677 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:26.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: 5.13 scrub starts 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: 5.13 scrub ok 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: 3.1 deep-scrub starts 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: 3.1 deep-scrub ok 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.734 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:26 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:26.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:26.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:26.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 75311 75295 104.555 69.3125 0.00763092 0.00955942 2024-04-25T05:47:26.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:26.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:26.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:27.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:47:27.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:27.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:27.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:27.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:27.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:27.422 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:27.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:27.558 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:27.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:47:27.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: pgmap v717: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: 1.8 deep-scrub starts 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: 1.8 deep-scrub ok 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: pgmap v717: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:47:27.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: 1.8 deep-scrub starts 2024-04-25T05:47:27.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: 1.8 deep-scrub ok 2024-04-25T05:47:27.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:47:27.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:47:27.622 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:27 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:27.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:27.696 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: pgmap v717: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: 1.8 deep-scrub starts 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: 1.8 deep-scrub ok 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:47:27.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:27 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:27.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:27.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 15 76512 76497 103.915 75.125 0.00810079 0.00961809 2024-04-25T05:47:27.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:27.969 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:27.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:28.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:28.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:28.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:47:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:28.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:28.242 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:28.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:28.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:28.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:28.519 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:28.529 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:28.570 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:28 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:47:28.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:28 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:47:28.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:28 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:47:28.571 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:28 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:47:28.571 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:28.609 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:28.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:28.639 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:28.653 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:28.675 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:28.715 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:28 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:47:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:28 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:47:28.753 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:28.754 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:28.787 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:28.797 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:28.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:28.921 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:28.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:28.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 77778 77762 103.386 79.0625 0.0086004 0.00966741 2024-04-25T05:47:29.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:29.056 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:29.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:29.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:29.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:29.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:29.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:29.445 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:29.467 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: pgmap v718: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3767209421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: pgmap v718: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:47:29.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:47:29.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3767209421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:29.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:47:29.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:47:29.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:47:29.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:47:29.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:29.598 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:29.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:29.733 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: pgmap v718: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 765 op/s 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3767209421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:47:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:29 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:47:29.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:29.870 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:29.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:29.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:29.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 79005 78989 102.83 76.6875 0.0160313 0.00972009 2024-04-25T05:47:29.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:30.012 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:30.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:30.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:30.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:30.284 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:30.331 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:47:30.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:30.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:30.541 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:47:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:47:30.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:47:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:47:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:47:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:30 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:47:30.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:30.677 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:47:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:30 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:47:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:30 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:47:30.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.815 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:30.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:30.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:30.931 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 80126 80110 102.161 70.0625 0.00420737 0.00978395 2024-04-25T05:47:30.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:30.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:31.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:31.086 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:31.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:31.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:31.359 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:31.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: pgmap v719: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: osdmap e406: 8 total, 8 up, 7 in 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: 4.11 deep-scrub starts 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: 4.11 deep-scrub ok 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:47:31.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: pgmap v719: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: osdmap e406: 8 total, 8 up, 7 in 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: 4.11 deep-scrub starts 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: 4.11 deep-scrub ok 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:47:31.577 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:31 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:47:31.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.629 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:31.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:31.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: pgmap v719: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3225597370' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: osdmap e406: 8 total, 8 up, 7 in 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: 4.11 deep-scrub starts 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: 4.11 deep-scrub ok 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:47:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:31 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:47:31.758 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:31.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:31.896 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:31.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:31.932 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 16 81423 81407 101.739 81.0625 0.00878897 0.00982409 2024-04-25T05:47:31.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:32.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:32.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:32.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:32.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:32.305 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:32.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:32.444 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:32.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:32.580 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:47:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:47:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 3.11 deep-scrub starts 2024-04-25T05:47:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 3.11 deep-scrub ok 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 3.11 deep-scrub starts 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 3.11 deep-scrub ok 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:47:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:32 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:47:32.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:47:32.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:32.715 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 3.11 deep-scrub starts 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 3.11 deep-scrub ok 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:47:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:32 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:47:32.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:32.850 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:32.933 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 15 82579 82564 101.16 72.3125 0.041288 0.00987904 2024-04-25T05:47:32.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:32.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:33.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:33.123 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:33.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:33.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:33.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:33.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:33.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:47:33.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: pgmap v721: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 589 B/s rd, 49 MiB/s wr, 785 op/s 2024-04-25T05:47:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 4.17 scrub starts 2024-04-25T05:47:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 4.17 scrub ok 2024-04-25T05:47:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:47:33.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: pgmap v721: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 589 B/s rd, 49 MiB/s wr, 785 op/s 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 4.17 scrub starts 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 4.17 scrub ok 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:47:33.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:33.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:33.664 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:33.730 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: pgmap v721: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 589 B/s rd, 49 MiB/s wr, 785 op/s 2024-04-25T05:47:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 4.17 scrub starts 2024-04-25T05:47:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 4.17 scrub ok 2024-04-25T05:47:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:47:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:47:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:47:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:47:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:33.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:33.765 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:33.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:33.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 83822 83806 100.705 77.625 0.00922547 0.00992508 2024-04-25T05:47:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:33.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:33.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:34.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:34.097 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:34.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:34.240 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:34.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:34.353 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:47:34.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:34.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:34.512 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:34.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:47:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:47:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:47:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:47:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:47:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:47:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:34 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:34.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:34.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:47:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:34 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:47:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:34 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:47:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:34 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:34 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:34.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:34.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:34.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:34.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 16 85025 85009 100.224 75.1875 0.0098814 0.00997268 2024-04-25T05:47:35.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.057 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:35.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:35.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:35.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:35.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.332 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:35.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:35.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:35.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:35.585 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:47:35.585 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:47:35.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:85835 10.a e407/387 10:51c27476:::benchmark_data_smithi043_152558_object85834:head v 407'5284, mlcod=407'5282)", 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:47:35.575205+0000", 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00699885, 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070545290000000004, 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575205+0000", 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:35.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575205+0000", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575205+0000", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6300000000000001e-07 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575236+0000", 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1062999999999997e-05 2024-04-25T05:47:35.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575243+0000", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7970000000000003e-06 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575248+0000", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1209999999999998e-06 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575901+0000", 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00065259200000000002 2024-04-25T05:47:35.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.575920+0000", 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9108000000000001e-05 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.582018+0000", 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060976379999999998 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.589 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:85839 10.a e407/387 10:519f3f16:::benchmark_data_smithi043_152558_object85838:head v 407'5285, mlcod=407'5282)", 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:47:35.578722+0000", 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0034815829999999999, 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036324899999999999, 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578722+0000", 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.590 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578722+0000", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578723+0000", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2440000000000001e-06 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578789+0000", 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5975000000000004e-05 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578804+0000", 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4918e-05 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.578813+0000", 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3089999999999996e-06 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.579034+0000", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000221177 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.579056+0000", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2469000000000001e-05 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:35.582351+0000", 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0032942370000000002 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:35.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:35.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:35.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:35.594 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:47:35.594 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-04-25T05:47:35.594 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:47:35.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:35.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:47:35.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: pgmap v722: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[27010]: osdmap e407: 8 total, 8 up, 7 in 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: pgmap v722: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:47:35.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:35 smithi043 ceph-mon[37492]: osdmap e407: 8 total, 8 up, 7 in 2024-04-25T05:47:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:35.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:35.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: pgmap v722: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3218153089' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:47:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:35 smithi060 ceph-mon[30898]: osdmap e407: 8 total, 8 up, 7 in 2024-04-25T05:47:35.755 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:35.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:35.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:35.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 86280 86264 99.8201 78.4375 0.0107383 0.0100127 2024-04-25T05:47:35.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:36.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:36.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:36.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:36.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:36.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:36.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:36.436 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:36.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:36.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:36.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:36.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:36.845 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:36.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 87573 87557 99.4744 80.8125 0.0114064 0.0100477 2024-04-25T05:47:36.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:36.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:36.996 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:47:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:37.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:37.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:37.276 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:37.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:37.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:37.491 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:37 smithi043 ceph-mon[27010]: pgmap v724: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:37.492 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:37 smithi043 ceph-mon[37492]: pgmap v724: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:37.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:37.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:37.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.697 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:37.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:37 smithi060 ceph-mon[30898]: pgmap v724: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-25T05:47:37.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:37.833 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:37.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 88818 88802 99.0875 77.8125 0.0162487 0.0100872 2024-04-25T05:47:37.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:37.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:37.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:38.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:38.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:38.167 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:47:38.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.242 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:38.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:38.372 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:38.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:38.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:38.509 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:38.558 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:38.602 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:38.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:38.635 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:38.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:38.683 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:38.718 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:38.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:38.763 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.775 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:38.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:38.806 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:38.811 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:38.811 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:38.811 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:38.811 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:38.812 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:38.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:38.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:38.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:38.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 89972 89956 98.6144 72.125 0.0217127 0.0101356 2024-04-25T05:47:39.018 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:39.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:39.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:39.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:39.206 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:39.206 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:39.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:39.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:47:39.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:39.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:39.602 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:39.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[27010]: pgmap v725: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-04-25T05:47:39.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3039092421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1864211436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[37492]: pgmap v725: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3039092421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1864211436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.704 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:39.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:39 smithi060 ceph-mon[30898]: pgmap v725: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 764 op/s 2024-04-25T05:47:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3039092421' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1864211436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:39 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:47:39.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:39.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:39.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 15 91261 91246 98.3041 80.625 0.00839024 0.0101677 2024-04-25T05:47:39.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:40.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:40.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:40.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:40.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:40.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:40.412 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:40.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:40.547 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:40.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:47:40.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[37492]: osdmap e408: 8 total, 8 up, 7 in 2024-04-25T05:47:40.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[37492]: osdmap e409: 8 total, 8 up, 7 in 2024-04-25T05:47:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:47:40.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[27010]: osdmap e408: 8 total, 8 up, 7 in 2024-04-25T05:47:40.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:40 smithi043 ceph-mon[27010]: osdmap e409: 8 total, 8 up, 7 in 2024-04-25T05:47:40.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:40.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:47:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:40 smithi060 ceph-mon[30898]: osdmap e408: 8 total, 8 up, 7 in 2024-04-25T05:47:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:40 smithi060 ceph-mon[30898]: osdmap e409: 8 total, 8 up, 7 in 2024-04-25T05:47:40.780 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:40.814 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:40.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:40.934 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 92495 92479 97.944 77.0625 0.0145605 0.0102049 2024-04-25T05:47:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:40.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:41.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:41.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:41.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:41.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:41.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:41.418 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:47:41.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:41.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:41.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:41.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:41 smithi043 ceph-mon[27010]: pgmap v727: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:41.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:41 smithi043 ceph-mon[37492]: pgmap v727: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:41.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:41.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:41 smithi060 ceph-mon[30898]: pgmap v727: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:41.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:41.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:41.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:47:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:41.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:41.935 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:47:41.935768+0000 min lat: 0.0012506 max lat: 0.243151 avg lat: 0.0102347 2024-04-25T05:47:41.935 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:47:41.935 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 93789 93773 97.6595 80.875 0.0169556 0.0102347 2024-04-25T05:47:41.947 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:42.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:42.022 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:42.041 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:42.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:42.177 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:42.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:42.317 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:42.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:42.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:42.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:42.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:42.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:42.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:42.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:42.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:47:42.878 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:42.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:42.936 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 95001 94985 97.2988 75.75 0.0124693 0.0102722 2024-04-25T05:47:42.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:43.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:43.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:43.243 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:43.267 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:43.268 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:43.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:43.338 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:43.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:43.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:43.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:43.631 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:43 smithi043 ceph-mon[27010]: pgmap v729: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 51 MiB/s wr, 817 op/s 2024-04-25T05:47:43.631 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:43 smithi043 ceph-mon[37492]: pgmap v729: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 51 MiB/s wr, 817 op/s 2024-04-25T05:47:43.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:43 smithi060 ceph-mon[30898]: pgmap v729: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 673 B/s rd, 51 MiB/s wr, 817 op/s 2024-04-25T05:47:43.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:43.753 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:43.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:43.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:43.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:43.936 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 96102 96086 96.8393 68.8125 0.0115869 0.0103218 2024-04-25T05:47:43.995 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.029 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:44.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:44.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:47:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:44.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:44.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:44.418 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:47:44.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:44.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:44.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:44.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:44.731 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2108308042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2108308042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.732 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:44 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:44 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:44 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:44 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2108308042' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:44 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:44.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:44.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:44.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:44.937 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 97301 97285 96.4901 74.9375 0.00815323 0.0103591 2024-04-25T05:47:44.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:44.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:45.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.099 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:45.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:45.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:45.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:45.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:45.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:45.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:45.507 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:45.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:45.649 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: pgmap v730: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: osdmap e410: 8 total, 8 up, 7 in 2024-04-25T05:47:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:45 smithi060 ceph-mon[30898]: osdmap e411: 8 total, 8 up, 7 in 2024-04-25T05:47:45.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:45.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:45.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: pgmap v730: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:45.774 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: osdmap e410: 8 total, 8 up, 7 in 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[27010]: osdmap e411: 8 total, 8 up, 7 in 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: pgmap v730: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: osdmap e410: 8 total, 8 up, 7 in 2024-04-25T05:47:45.775 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:45 smithi043 ceph-mon[37492]: osdmap e411: 8 total, 8 up, 7 in 2024-04-25T05:47:45.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:45.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:45.927 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:45.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:45.936 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 98536 98520 96.1884 77.1875 0.00547366 0.0103909 2024-04-25T05:47:46.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:46.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:46.141 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:46.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:46.175 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:46.219 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:46.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:46.255 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:46.290 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:46.322 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:46.328 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:46.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:46.371 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:46.413 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:46.427 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:47:46.460 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:46.461 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:46.467 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:46.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:46.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:47:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2862901116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:46.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:47:46.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:46.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:46.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2862901116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:46.642 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:46.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:47:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:46 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:46 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2862901116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:46.782 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:46.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:46.921 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:46.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:46.937 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 99767 99751 95.8922 76.9375 0.0167475 0.0104238 2024-04-25T05:47:47.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:47.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:47.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:47.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:47.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:47.327 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:47.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:47.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:47.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[37492]: pgmap v733: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[37492]: osdmap e412: 8 total, 8 up, 7 in 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[27010]: pgmap v733: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:47:47.537 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:47 smithi043 ceph-mon[27010]: osdmap e412: 8 total, 8 up, 7 in 2024-04-25T05:47:47.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:47.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:47.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:47 smithi060 ceph-mon[30898]: pgmap v733: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2134539614' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:47:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:47 smithi060 ceph-mon[30898]: osdmap e412: 8 total, 8 up, 7 in 2024-04-25T05:47:47.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:47.864 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:47.937 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 100934 100918 95.5443 72.9375 0.00910161 0.0104621 2024-04-25T05:47:47.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:47.997 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:47.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:48.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:48.130 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:48.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:47:48.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:48.309 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:99865 10.4 10:25c17dc0:::benchmark_data_smithi043_152558_object99864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:47:47.015159+0000", 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0034004980000000001, 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003440224, 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:47:48.365 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 99865 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015159+0000", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015159+0000", 2024-04-25T05:47:48.366 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015161+0000", 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.765e-06 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015184+0000", 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3329e-05 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015198+0000", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.398e-05 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015207+0000", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3359999999999995e-06 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015296+0000", 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8874000000000003e-05 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015448+0000", 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015226199999999999 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.016205+0000", 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00075720099999999999 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:99861 10.3 e412/387 10:c327acab:::benchmark_data_smithi043_152558_object99860:head v 412'6187, mlcod=412'6185)", 2024-04-25T05:47:48.369 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:47:47.015234+0000", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0033262040000000001, 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034672539999999999, 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015234+0000", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015234+0000", 2024-04-25T05:47:48.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015234+0000", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7599999999999997e-07 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015253+0000", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9279999999999998e-05 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015265+0000", 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1919000000000001e-05 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015271+0000", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9379999999999997e-06 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015468+0000", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019636799999999999 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.015486+0000", 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8586000000000001e-05 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:99864 10.7 e412/387 10:e239eaad:::benchmark_data_smithi043_152558_object99863:head v 412'6240, mlcod=412'6237)", 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:47:47.017974+0000", 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.00058604100000000004, 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080972300000000004, 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.017974+0000", 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.017974+0000", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.017975+0000", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7000000000000003e-07 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.018003+0000", 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8617999999999999e-05 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.018013+0000", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5640000000000002e-06 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.018020+0000", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.729e-06 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.018123+0000", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010288799999999999 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:47:47.018144+0000", 2024-04-25T05:47:48.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0993e-05 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 3 2024-04-25T05:47:48.376 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:47:48.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:48.453 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:48.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:48.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:48.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:48.721 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:48.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:48.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:48.937 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 102217 102201 95.3151 80.1875 0.0168772 0.0104871 2024-04-25T05:47:48.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:48.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:48.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:49.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:47:49.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:49.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:49.190 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:49.219 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:49.261 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:49.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:49.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[27010]: pgmap v735: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:49.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:49.339 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:49.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[37492]: pgmap v735: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:49.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:49.340 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:49.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:49.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:49 smithi060 ceph-mon[30898]: pgmap v735: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:47:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:47:49.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.590 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:49.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:49.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:49.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:47:49.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:49.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:49.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:49.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:49.938 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 103443 103427 95.0381 76.625 0.00716749 0.0105175 2024-04-25T05:47:49.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:50.000 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:50.019 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:50.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:50.157 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:50.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:50.295 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:50.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:50.431 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:50.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:50.582 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:50.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:50.631 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:50.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:50.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:50.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:50.878 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:50.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:50.939 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 104607 104591 94.7137 72.75 0.0160738 0.0105528 2024-04-25T05:47:50.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:51.015 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:51.097 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:47:51.102 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:47:51.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:51.150 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:50 smithi043 ceph-mon[27010]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[27010]: osdmap e413: 8 total, 8 up, 7 in 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3269782695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:50 smithi043 ceph-mon[37492]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[37492]: osdmap e413: 8 total, 8 up, 7 in 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3269782695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:50 smithi060 ceph-mon[30898]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:50 smithi060 ceph-mon[30898]: osdmap e413: 8 total, 8 up, 7 in 2024-04-25T05:47:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:50 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3269782695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:50 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:47:51.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:51.283 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:51.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:51.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:51.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:51.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:51.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:51.703 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:47:51.772 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:47:51.798 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:47:51.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:51.823 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:47:51.825 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:47:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:51.930 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:51.940 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 15 105759 105744 94.3901 72.0625 0.0222564 0.0105892 2024-04-25T05:47:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:51.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:51.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:52.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:52.106 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/835248281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:52.118 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: osdmap e414: 8 total, 8 up, 7 in 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2776632381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:51 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/835248281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.119 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: osdmap e414: 8 total, 8 up, 7 in 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2776632381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:47:52.120 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:52 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:47:52.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/835248281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: osdmap e414: 8 total, 8 up, 7 in 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:47:52.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2776632381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:47:52.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:47:52.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:47:52.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:47:52.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:51 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:47:52.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:52.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:52.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:52.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:52.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:52.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:52.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:52.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:52.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:52.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:52.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:52.940 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 15 107046 107031 94.1935 80.4375 0.0197961 0.0106119 2024-04-25T05:47:53.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:53.062 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:53.107 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:53.107 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:53.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:53.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:53.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:53.327 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:53.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:53 smithi043 ceph-mon[27010]: pgmap v739: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:53.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:53 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:47:53.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:53 smithi043 ceph-mon[37492]: pgmap v739: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:53.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:53 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:47:53.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:53.457 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:53.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:53 smithi060 ceph-mon[30898]: pgmap v739: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-25T05:47:53.489 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:53 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:47:53.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:53.593 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:53.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:53.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:53.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:47:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:47:53.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:53.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:53.940 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 108367 108351 94.0311 82.5 0.0110857 0.0106305 2024-04-25T05:47:53.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:53.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:53.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:54.078 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:47:54.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:54.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:54.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:54.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:54 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:47:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:54 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:47:54.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:54.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:54 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:47:54.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:54.575 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:54.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:47:54.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:47:54.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.716 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:54.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:54.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.848 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:54.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:54.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 109601 109585 93.7983 77.125 0.0164913 0.0106561 2024-04-25T05:47:54.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:54.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:54.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:55.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:55.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:55.264 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:55.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[27010]: pgmap v740: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[27010]: osdmap e415: 8 total, 8 up, 7 in 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[37492]: pgmap v740: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-25T05:47:55.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:55 smithi043 ceph-mon[37492]: osdmap e415: 8 total, 8 up, 7 in 2024-04-25T05:47:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:47:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:55 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:47:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:55 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:47:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:55 smithi060 ceph-mon[30898]: pgmap v740: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-25T05:47:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:55 smithi060 ceph-mon[30898]: osdmap e415: 8 total, 8 up, 7 in 2024-04-25T05:47:55.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:55.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:55.623 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:55.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:55.667 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:55.683 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:47:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:55.797 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:55.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:55.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:55.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 110790 110774 93.5349 74.3125 0.0089504 0.0106862 2024-04-25T05:47:55.968 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:55.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:56.070 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:56.106 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:56.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:56.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:56.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 5.17 scrub starts 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 5.17 scrub ok 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:47:56.365 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:47:56.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:47:56.366 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:47:56.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:47:56.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 5.17 scrub starts 2024-04-25T05:47:56.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 5.17 scrub ok 2024-04-25T05:47:56.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:47:56.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:56 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:47:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:56.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:56.476 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:56.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:47:56.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:47:56.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:47:56.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:47:56.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 5.17 scrub starts 2024-04-25T05:47:56.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 5.17 scrub ok 2024-04-25T05:47:56.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:47:56.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:56 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:47:56.511 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:56.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:56.652 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:56.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:56.787 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:56.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:56.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:56.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:56.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 112048 112032 93.336 78.625 0.014653 0.0107094 2024-04-25T05:47:57.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.061 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:57.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:57.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:57.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[27010]: pgmap v742: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:47:57.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:47:57.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[27010]: 5.16 scrub starts 2024-04-25T05:47:57.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[27010]: 5.16 scrub ok 2024-04-25T05:47:57.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[37492]: pgmap v742: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:57.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:47:57.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:47:57.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[37492]: 5.16 scrub starts 2024-04-25T05:47:57.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:57 smithi043 ceph-mon[37492]: 5.16 scrub ok 2024-04-25T05:47:57.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:57.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:57.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:47:57.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:57 smithi060 ceph-mon[30898]: pgmap v742: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-04-25T05:47:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:57 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:47:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:57 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:47:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:57 smithi060 ceph-mon[30898]: 5.16 scrub starts 2024-04-25T05:47:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:57 smithi060 ceph-mon[30898]: 5.16 scrub ok 2024-04-25T05:47:57.582 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:57.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:57.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:57.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:57.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:57.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:57.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:57.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 112954 112938 92.853 56.625 0.00557412 0.0107651 2024-04-25T05:47:57.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:47:58.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:47:58.109 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:58.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:58.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:47:58.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:47:58.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: 10.e scrub starts 2024-04-25T05:47:58.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: osdmap e416: 8 total, 8 up, 7 in 2024-04-25T05:47:58.144 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: 3.1e scrub starts 2024-04-25T05:47:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: 3.1e scrub ok 2024-04-25T05:47:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/106450165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2500978927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.145 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.153 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:47:58.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:58.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:58.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:58.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: 10.e scrub starts 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: osdmap e416: 8 total, 8 up, 7 in 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: 3.1e scrub starts 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: 3.1e scrub ok 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/106450165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2500978927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:58 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:58.549 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: 10.e scrub starts 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: osdmap e416: 8 total, 8 up, 7 in 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: 3.1e scrub starts 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: 3.1e scrub ok 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/106450165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2500978927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:58 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:47:58.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:47:58.691 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:47:58.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:47:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:58.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:47:58.928 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:47:58.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 114098 114082 92.5757 71.5 0.0106959 0.0107968 2024-04-25T05:47:58.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:58.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:47:59.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:59.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:59.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:47:59.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:47:59.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:47:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:47:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 745 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-25T05:47:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[37492]: osdmap e417: 8 total, 8 up, 7 in 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 745 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-mon[27010]: osdmap e417: 8 total, 8 up, 7 in 2024-04-25T05:47:59.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:47:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: pgmap v744: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 745 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:47:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:47:59 smithi060 ceph-mon[30898]: osdmap e417: 8 total, 8 up, 7 in 2024-04-25T05:47:59.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:47:59.621 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:47:59.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:59.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:47:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:47:59.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:47:59.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:47:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:47:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:47:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:47:59.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:47:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:47:59.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:47:59.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 115213 115197 92.2822 69.6875 0.00795412 0.0108317 2024-04-25T05:48:00.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:00.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:00.174 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:00.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:00.313 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:00.334 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:48:00.415 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:00.549 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:00.587 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:00.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:00.727 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:00.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:00.941 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 116373 116357 92.0318 72.5 0.0108834 0.0108612 2024-04-25T05:48:00.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:00.997 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:00.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:01.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:01.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:01.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 10.b scrub starts 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 10.e scrub ok 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:48:01.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:00 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:48:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:01.369 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 10.b scrub starts 2024-04-25T05:48:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 10.e scrub ok 2024-04-25T05:48:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 10.b scrub starts 2024-04-25T05:48:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 10.e scrub ok 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: pgmap v746: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:48:01.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:00 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:48:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:01.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:01.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:01.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:01.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:01.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:01.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:01.795 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:01.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:01.929 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:01.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:01.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:48:01.942713+0000 min lat: 0.0012506 max lat: 0.243151 avg lat: 0.0108936 2024-04-25T05:48:01.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:48:01.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 117490 117474 91.754 69.8125 0.0194356 0.0108936 2024-04-25T05:48:02.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:02.066 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:02.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:02.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:01 smithi060 ceph-mon[30898]: 3.18 scrub starts 2024-04-25T05:48:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:01 smithi060 ceph-mon[30898]: 3.18 scrub ok 2024-04-25T05:48:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:01 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:48:02.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:01 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:48:02.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:02.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[27010]: 3.18 scrub starts 2024-04-25T05:48:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[27010]: 3.18 scrub ok 2024-04-25T05:48:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:48:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:48:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[37492]: 3.18 scrub starts 2024-04-25T05:48:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[37492]: 3.18 scrub ok 2024-04-25T05:48:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:48:02.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:01 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:48:02.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:02.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:02.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:02.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:02.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:02.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:02.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:02.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:02.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:02.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 118634 118618 91.5039 71.5 0.00744155 0.0109241 2024-04-25T05:48:02.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:03.005 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:03.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:03.142 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:03.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 10.b scrub ok 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 6.17 scrub starts 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 734 B/s rd, 53 MiB/s wr, 842 op/s 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 6.17 scrub ok 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:48:03.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:48:03.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 10.b scrub ok 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:48:03.248 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 6.17 scrub starts 2024-04-25T05:48:03.249 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 734 B/s rd, 53 MiB/s wr, 842 op/s 2024-04-25T05:48:03.249 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 6.17 scrub ok 2024-04-25T05:48:03.249 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:48:03.249 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:02 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 10.b scrub ok 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 6.17 scrub starts 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: pgmap v747: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 734 B/s rd, 53 MiB/s wr, 842 op/s 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 6.17 scrub ok 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:48:03.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:02 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:48:03.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:03.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.446 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:03.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:03.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:03.574 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:03.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:03.708 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:03.718 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:03.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:03.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:03.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:03.942 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 119700 119684 91.2005 66.625 0.0045024 0.0109597 2024-04-25T05:48:03.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:03.991 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:03.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:04.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.126 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:04.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:04.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: 10.f scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: 4.7 scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: 4.7 scrub ok 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: 10.f scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: 4.7 scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: 4.7 scrub ok 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:48:04.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:48:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:04.143 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: 10.f scrub starts 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: 4.7 scrub starts 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: 4.7 scrub ok 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:04.175 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:04.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:04.255 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:04.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:04.382 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:04.405 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:04.405 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:04.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:04.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:04.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:48:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:04.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:04.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:04.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:04.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:04.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:04.960 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 120746 120730 90.8686 65.375 0.00786506 0.0109993 2024-04-25T05:48:05.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:05.070 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:05.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:05.208 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:05.258 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:05.300 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:05.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:05.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: pgmap v748: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T05:48:05.335 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: pgmap v748: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:48:05.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:05 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:48:05.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:05.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:05.412 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:05.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:05.451 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:05.484 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: pgmap v748: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:48:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:05 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:48:05.500 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:05.538 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:05.541 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:05.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:05.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:05.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:48:05.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:05.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:05.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:05.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:48:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:05.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:05.961 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 121833 121817 90.5957 67.9375 0.0070702 0.0110327 2024-04-25T05:48:06.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:06.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:06.125 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:06.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:06.192 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:06.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:06.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 3.6 deep-scrub starts 2024-04-25T05:48:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 3.6 deep-scrub ok 2024-04-25T05:48:06.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 10.f scrub ok 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 10.f scrub starts 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4289252834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 3.6 deep-scrub starts 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 3.6 deep-scrub ok 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 10.f scrub ok 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 10.f scrub starts 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:48:06.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4289252834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 3.6 deep-scrub starts 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 3.6 deep-scrub ok 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 10.f scrub ok 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 10.f scrub starts 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:48:06.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4289252834' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:06.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:06.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:06.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:06.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:06.736 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:06.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:06.874 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:06.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:06.899 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:06.963 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 122763 122747 90.212 58.125 0.00934331 0.0110802 2024-04-25T05:48:06.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:06.978 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:48:06.978 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:48:06.978 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:48:06.978 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:48:06.978 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:48:07.008 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:07.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:07.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:07 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:48:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:07 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:48:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:07 smithi060 ceph-mon[30898]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T05:48:07.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:07 smithi060 ceph-mon[30898]: 10.0 scrub starts 2024-04-25T05:48:07.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:07.286 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:07.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[27010]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[27010]: 10.0 scrub starts 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[37492]: pgmap v749: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-25T05:48:07.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:07 smithi043 ceph-mon[37492]: 10.0 scrub starts 2024-04-25T05:48:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:07.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:07.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:07.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:07.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:07.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:07.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:07.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:48:07.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:07.962 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 123793 123777 89.9116 64.375 0.0159911 0.0111162 2024-04-25T05:48:07.991 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:07.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:08.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.134 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:08.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:08.139 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:48:08.139 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:48:08.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:08.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:08.332 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:08.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:08.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4065900346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1093698510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4065900346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1093698510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4065900346' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1093698510' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.473 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:08.507 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:08.518 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:48:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:08.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:08.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:08.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:08.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:08.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:08.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:08.962 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 15 124863 124848 89.6475 66.9375 0.00887392 0.0111498 2024-04-25T05:48:09.017 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196432 7.4 7:201f4639:::benchmark_data_smithi043_74356_object79563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 577.11120565900001, 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.18528801, 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:48:09.018 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196432 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901542+0000", 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901544+0000", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3030000000000001e-06 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901549+0000", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901564+0000", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5019e-05 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901572+0000", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2009999999999999e-06 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901623+0000", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1400000000000003e-05 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901753+0000", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012954700000000001 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.901869+0000", 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011555199999999999 2024-04-25T05:48:09.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904486+0000", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002616967 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904523+0000", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7156000000000001e-05 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086719+0000", 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.182196424 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086782+0000", 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2833e-05 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.086830+0000", 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8378000000000003e-05 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24734.0:196436 7.1 7:86601c62:::benchmark_data_smithi043_74356_object79567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-04-25T05:48:09.023 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 577.11070737099999, 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.177590574, 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24734", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/956430992", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 196436 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902041+0000", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8000000000000005e-07 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902048+0000", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9419999999999998e-06 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902058+0000", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.008e-05 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902066+0000", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2790000000000001e-06 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902249+0000", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018350800000000001 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902354+0000", 2024-04-25T05:48:09.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105227 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.902460+0000", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000105164 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.904687+0000", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022273990000000001 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:31.907211+0000", 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002523941 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079433+0000", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.17222240799999999 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079565+0000", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013155999999999999 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:38:32.079631+0000", 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6285999999999995e-05 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:124112 8.2 8:4cdac3fe:::benchmark_data_smithi043_104001_object124111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e242)", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 431.90084872800003, 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.16378072499999999, 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 124112 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111899+0000", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111905+0000", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1059999999999996e-06 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111939+0000", 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3980999999999999e-05 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111951+0000", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1635e-05 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.111959+0000", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1689999999999992e-06 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112075+0000", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011596500000000001 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112203+0000", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012803900000000001 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.112363+0000", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000160039 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116878+0000", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045148760000000001 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.116924+0000", 2024-04-25T05:48:09.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5593999999999997e-05 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.119845+0000", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002921589 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275525+0000", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.15567968900000001 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275567+0000", 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1882999999999999e-05 2024-04-25T05:48:09.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:40:57.275680+0000", 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011315999999999999 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:125664 8.2 8:4acd28bf:::benchmark_data_smithi043_104001_object1540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 428.72649181200001, 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11907625199999999, 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.034 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125664 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286256+0000", 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286258+0000", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.415e-06 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286264+0000", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8159999999999996e-06 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286279+0000", 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5085e-05 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.286287+0000", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9650000000000008e-06 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287683+0000", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001395269 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287830+0000", 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000147777 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.287984+0000", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015359799999999999 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291040+0000", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030556089999999999 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291225+0000", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018544300000000001 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.291399+0000", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017389099999999999 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405138+0000", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11373876199999999 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405204+0000", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6484999999999998e-05 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:00.405332+0000", 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012813700000000001 2024-04-25T05:48:09.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:126959 8.2 8:4e62c023:::benchmark_data_smithi043_104001_object2835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 427.16105227499997, 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.14632261699999999, 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 126959 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.040 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851696+0000", 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851697+0000", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.237e-06 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851705+0000", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8320000000000006e-06 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851714+0000", 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.064e-06 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851720+0000", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0859999999999998e-06 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.851962+0000", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024216799999999999 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852126+0000", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016425399999999999 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.852306+0000", 2024-04-25T05:48:09.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000180082 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.854938+0000", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026320139999999998 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855570+0000", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00063133700000000004 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.855614+0000", 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4480000000000001e-05 2024-04-25T05:48:09.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997822+0000", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.142207745 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.997883+0000", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1085000000000002e-05 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:01.998018+0000", 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013523300000000001 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.045 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129538 8.2 8:4c65616f:::benchmark_data_smithi043_104001_object5414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 424.00172454900002, 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12767905500000001, 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129538 2024-04-25T05:48:09.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011023+0000", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011025+0000", 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1650000000000001e-06 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011031+0000", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2639999999999997e-06 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011042+0000", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1385e-05 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.011051+0000", 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3149999999999992e-06 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.012950+0000", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018998400000000001 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013084+0000", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013384000000000001 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.013219+0000", 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013510700000000001 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018871+0000", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056513270000000003 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018951+0000", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0836999999999999e-05 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022366+0000", 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0034148400000000001 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023969+0000", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001602526 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.024068+0000", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9500999999999995e-05 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138702+0000", 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.114634108 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129549 8.2 8:40741811:::benchmark_data_smithi043_104001_object5425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.994315836, 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.13464008199999999, 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129549 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018432+0000", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018434+0000", 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.528e-06 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018439+0000", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3040000000000004e-06 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018450+0000", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0788000000000001e-05 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018458+0000", 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.405e-06 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.018994+0000", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053616199999999997 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019116+0000", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121228 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.019243+0000", 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012719599999999999 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.022328+0000", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030850640000000002 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138716+0000", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.11638823700000001 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138755+0000", 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8992999999999997e-05 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152783+0000", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014027773 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.152919+0000", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013651599999999999 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153072+0000", 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000152888 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129543 8.e e246/240 8:7455de0e:::benchmark_data_smithi043_104001_object5419:head v 246'7976, mlcod=245'7972)", 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.99270412099997, 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12636956799999999, 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020044+0000", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.82e-07 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020049+0000", 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2490000000000003e-06 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020060+0000", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1070000000000001e-05 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020065+0000", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2310000000000004e-06 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020859+0000", 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000793862 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.020882+0000", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3170999999999999e-05 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146330+0000", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12544777300000001 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146413+0000", 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3629999999999997e-05 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.061 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop_reply(client.25115.0:129549 8.2 e246/233 ondisk, result = 0)", 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.98961067800002, 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.1156257, 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:48:09.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023137+0000", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023138+0000", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8999999999999996e-07 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023142+0000", 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2039999999999999e-06 2024-04-25T05:48:09.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023149+0000", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5019999999999998e-06 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023155+0000", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0229999999999998e-06 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138742+0000", 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115587254 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138750+0000", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8089999999999993e-06 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138763+0000", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3318e-05 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.25115.0:129546 8.e e246/240 8:758a41b0:::benchmark_data_smithi043_104001_object5422:head v 246'7977, mlcod=245'7972)", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:48:09.065 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.98889518499999, 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12274840099999999, 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:09.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023853+0000", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9200000000000001e-07 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023859+0000", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023867+0000", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0909999999999993e-06 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.023874+0000", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8240000000000002e-06 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139217+0000", 2024-04-25T05:48:09.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.115343705 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139245+0000", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7317999999999998e-05 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146417+0000", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0071721129999999999 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.146601+0000", 2024-04-25T05:48:09.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018442799999999999 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:129556 8.2 8:484428a6:::benchmark_data_smithi043_104001_object5432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.98468659700001, 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.125238935, 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 129556 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028061+0000", 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028062+0000", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.091e-06 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.028067+0000", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8489999999999997e-06 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137388+0000", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.109320815 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.137405+0000", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6677999999999999e-05 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138822+0000", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001417614 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.138949+0000", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012701800000000001 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3,5", 2024-04-25T05:48:09.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.139092+0000", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014223099999999999 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.141864+0000", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027722189999999998 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143235+0000", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001370778 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.143353+0000", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118484 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153140+0000", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0097873479999999995 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153195+0000", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5000999999999997e-05 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:41:05.153300+0000", 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010480900000000001 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.2 8.3a1a8b12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 329.142109576, 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12946094299999999, 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.077 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870638+0000", 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870640+0000", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7379999999999999e-06 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870647+0000", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0330000000000003e-06 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870659+0000", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1748e-05 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.870667+0000", 2024-04-25T05:48:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3100000000000001e-06 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000085+0000", 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.12941729699999999 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000099+0000", 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4817000000000001e-05 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.080 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.2 8.516c972 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 329.13426194599998, 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122103355, 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:48:09.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878486+0000", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878488+0000", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0209999999999998e-06 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878500+0000", 2024-04-25T05:48:09.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1629e-05 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878519+0000", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9522999999999999e-05 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878530+0000", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0815000000000001e-05 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000576+0000", 2024-04-25T05:48:09.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.122046021 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000589+0000", 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3346000000000001e-05 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.12 8:48d1585c:::benchmark_data_smithi043_104001_object102215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 329.119230835, 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1294363890000001, 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893519+0000", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1519999999999999e-06 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893531+0000", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1849e-05 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893551+0000", 2024-04-25T05:48:09.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9638000000000001e-05 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893564+0000", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.323e-05 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000742+0000", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10717832200000001 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000752+0000", 2024-04-25T05:48:09.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9930000000000007e-06 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006085+0000", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0053329209999999 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006212+0000", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126873 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006455+0000", 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024327100000000001 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011490+0000", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050344350000000003 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.012956+0000", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001465872 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022845+0000", 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0098890200000000001 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022904+0000", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9239e-05 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022954+0000", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9573999999999997e-05 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.12 8:4e9368a0:::benchmark_data_smithi043_104001_object102219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 329.11915619400003, 2024-04-25T05:48:09.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1317489059999999, 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4000000000000002e-07 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893595+0000", 2024-04-25T05:48:09.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8600000000000001e-06 2024-04-25T05:48:09.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893603+0000", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9000000000000006e-06 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893609+0000", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2500000000000003e-06 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000774+0000", 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10716527300000001 2024-04-25T05:48:09.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000780+0000", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7119999999999997e-06 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009444+0000", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086639690000001 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009577+0000", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013284399999999999 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:48:09.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009825+0000", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024752200000000002 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.013047+0000", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003222042 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020080+0000", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070331810000000003 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025198+0000", 2024-04-25T05:48:09.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051180369999999998 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025295+0000", 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6797000000000002e-05 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025341+0000", 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6078999999999997e-05 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.097 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228033 8.14 8:2d5c3048:::benchmark_data_smithi043_104001_object103909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 327.02459441500002, 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0985097700000002, 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228033 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.098 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988156+0000", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9879999999999999e-06 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988162+0000", 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7870000000000004e-06 2024-04-25T05:48:09.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988180+0000", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7886000000000001e-05 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988190+0000", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9729999999999992e-06 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988272+0000", 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1977999999999993e-05 2024-04-25T05:48:09.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988285+0000", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3176000000000001e-05 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980591+0000", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99230595399999999 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980607+0000", 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5548000000000001e-05 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081723+0000", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1011164630000001 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081910+0000", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018638699999999999 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.082205+0000", 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00029534400000000001 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085239+0000", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030338330000000001 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085283+0000", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3695000000000001e-05 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085372+0000", 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.954e-05 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086212+0000", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084000600000000002 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086435+0000", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022244300000000001 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086501+0000", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6000000000000005e-05 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086663+0000", 2024-04-25T05:48:09.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162769 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228057 8.14 8:29afc790:::benchmark_data_smithi043_104001_object103933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 327.00278221100001, 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0771824429999999, 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:09.105 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228057 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009967+0000", 2024-04-25T05:48:09.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.418e-06 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009982+0000", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5211000000000001e-05 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009997+0000", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4542000000000001e-05 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010006+0000", 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3619999999999995e-06 2024-04-25T05:48:09.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010067+0000", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0965000000000003e-05 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010076+0000", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1249999999999999e-06 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980702+0000", 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97062569499999996 2024-04-25T05:48:09.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980712+0000", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0081999999999999e-05 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083607+0000", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1028946399999999 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083747+0000", 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013999600000000001 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.084002+0000", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025568299999999998 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085297+0000", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001294033 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085711+0000", 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041416199999999998 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086266+0000", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055577100000000004 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086728+0000", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000461335 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086920+0000", 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019192499999999999 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086999+0000", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9742999999999999e-05 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.087148+0000", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014875500000000001 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237380 9.16 9:6e797660:::benchmark_data_smithi043_129820_object116066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 112.844748753, 2024-04-25T05:48:09.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94758949999999997, 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237380 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.167999+0000", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168000+0000", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8400000000000002e-07 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168004+0000", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7050000000000001e-06 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168011+0000", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4739999999999999e-06 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168017+0000", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5160000000000002e-06 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168065+0000", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8109000000000003e-05 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:09.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.168072+0000", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2139999999999999e-06 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.111902+0000", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94382929999999998 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.112059+0000", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000157455 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:09.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.112290+0000", 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023091 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113696+0000", 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014061670000000001 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:09.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113711+0000", 2024-04-25T05:48:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.118 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:09.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:09.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5133000000000001e-05 2024-04-25T05:48:09.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115483+0000", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017713340000000001 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115543+0000", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0755999999999999e-05 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115589+0000", 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5442999999999998e-05 2024-04-25T05:48:09.119 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.15975.0:122447 10.0 e417/387 10:0848a772:::benchmark_data_smithi043_152558_object122446:head v 417'7716, mlcod=417'7712)", 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:48:06.595571+0000", 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 2.417176526, 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033748606, 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595571+0000", 2024-04-25T05:48:09.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595571+0000", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595573+0000", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.212e-06 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595599+0000", 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6560999999999999e-05 2024-04-25T05:48:09.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595616+0000", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6886000000000001e-05 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.595626+0000", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4690000000000003e-06 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.596771+0000", 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011456870000000001 2024-04-25T05:48:09.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.596790+0000", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8836000000000001e-05 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.629287+0000", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.032497223999999998 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:06.629320+0000", 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2731000000000002e-05 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.123 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.15975.0:123753 10.6 e417/387 10:6df55faf:::benchmark_data_smithi043_152558_object123752:head v 417'7795, mlcod=417'7791)", 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:48:07.934568+0000", 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 1.078180001, 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035227057999999999, 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934568+0000", 2024-04-25T05:48:09.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934568+0000", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934569+0000", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9700000000000005e-07 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934636+0000", 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7527000000000006e-05 2024-04-25T05:48:09.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934660+0000", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3614000000000001e-05 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934667+0000", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1620000000000002e-06 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934747+0000", 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9938999999999995e-05 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.934766+0000", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8405000000000001e-05 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.969691+0000", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.034925838000000001 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:48:07.969795+0000", 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103676 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.127 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.128 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:09.128 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:09.128 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:48:09.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:09.251 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:09.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:09.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[27010]: pgmap v750: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:48:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:48:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:09.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[27010]: osdmap e418: 8 total, 8 up, 7 in 2024-04-25T05:48:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[37492]: pgmap v750: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:48:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:48:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:09.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-mon[37492]: osdmap e418: 8 total, 8 up, 7 in 2024-04-25T05:48:09.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:09 smithi060 ceph-mon[30898]: pgmap v750: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-04-25T05:48:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:09 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:48:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:09 smithi060 ceph-mon[30898]: osdmap e418: 8 total, 8 up, 7 in 2024-04-25T05:48:09.520 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:09.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:09.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:09.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:09.794 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:09.874 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:09.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:09.909 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:09.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:48:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:09.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:09.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:09.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 16 126048 126032 89.4687 74 0.0115351 0.0111728 2024-04-25T05:48:10.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:10.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:10.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:10.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:10.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.378 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:10.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:10.479 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:10.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:10.525 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:48:10.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:10.640 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:10.646 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:10.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:10.687 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:10.726 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:10.759 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:10.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:10.794 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.810 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:10.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:10.833 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:10.875 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:10.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:10.919 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:10.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:10.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:10.963 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:10.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 127217 127201 89.2844 73.0625 0.0267111 0.0111953 2024-04-25T05:48:11.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:11.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:11.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.223 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:11.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:11.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:48:11.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:11.356 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:11 smithi043 ceph-mon[37492]: pgmap v752: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:11 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/899870435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:11 smithi043 ceph-mon[27010]: pgmap v752: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:11.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:11 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/899870435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:11.420 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:11 smithi060 ceph-mon[30898]: pgmap v752: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:11.420 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:11 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/899870435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:11.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:11.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:11.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:11.640 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:11.640 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:11.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:11.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:11.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:11.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:11.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0061 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 128347 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 89.1238 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 27.5206 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 195.125 2024-04-25T05:48:11.964 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 56.625 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1425 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 440.33 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3122 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 906 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0112164 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.00837924 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 0.243151 2024-04-25T05:48:11.965 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.0012506 2024-04-25T05:48:11.968 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-04-25T05:48:12.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:12.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:12.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:12.152 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:12.186 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:12.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:12.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:12.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:12.466 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:12.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:12.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:12.699 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:12.731 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:12.833 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:12.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:12.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:12.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:12.999 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:13.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:13.137 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:13.155 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:48:13.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:13.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:13.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:13.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:13 smithi060 ceph-mon[30898]: pgmap v753: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1864161950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[27010]: pgmap v753: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1864161950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[37492]: pgmap v753: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.6 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1864161950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.566 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:48:13.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:13.669 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:13.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:13.807 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:13.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:48:13.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:13.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:13.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:13.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:14.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:14.084 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:48:14.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:48:14.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:14.190 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:48:14.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:14.229 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:14.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:14.365 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:14.467 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:14.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:48:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:14 smithi060 ceph-mon[30898]: osdmap e419: 8 total, 8 up, 7 in 2024-04-25T05:48:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3396931471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:14.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:48:14.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[27010]: osdmap e419: 8 total, 8 up, 7 in 2024-04-25T05:48:14.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3396931471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.622 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:48:14.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[37492]: osdmap e419: 8 total, 8 up, 7 in 2024-04-25T05:48:14.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3396931471' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.623 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:14.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:14.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:14.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:14.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:14.900 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:14.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:15.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:15.034 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:15.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:15.084 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:15.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:15.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:15.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:15.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:15.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[27010]: pgmap v755: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.28k op/s 2024-04-25T05:48:15.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:15.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:15.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[27010]: osdmap e420: 8 total, 8 up, 7 in 2024-04-25T05:48:15.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:15.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[37492]: pgmap v755: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.28k op/s 2024-04-25T05:48:15.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:15.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:15.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:15 smithi043 ceph-mon[37492]: osdmap e420: 8 total, 8 up, 7 in 2024-04-25T05:48:15.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:15.477 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:15 smithi060 ceph-mon[30898]: pgmap v755: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.4 KiB/s rd, 70 MiB/s wr, 1.28k op/s 2024-04-25T05:48:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:15 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:15 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:15 smithi060 ceph-mon[30898]: osdmap e420: 8 total, 8 up, 7 in 2024-04-25T05:48:15.579 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:15.601 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:15.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:15.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:15.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:15.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:15.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:16.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:16.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:16.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:16.208 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:16.228 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:16.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:16.275 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.285 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:16.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:16.321 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:48:16.326 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:16.331 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:48:16.365 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:16.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:16.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:48:16.407 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:16.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:16.447 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:16.482 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:16.485 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:16.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:16.553 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:16.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:16.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:16.830 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:16.845 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:16.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:16.975 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:16.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:17.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:48:17.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:17.114 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:17.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:17.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:17.248 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:17 smithi060 ceph-mon[30898]: pgmap v757: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 783 B/s rd, 65 MiB/s wr, 1.25k op/s 2024-04-25T05:48:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1274458325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:17.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:17 smithi060 ceph-mon[30898]: from='client.25982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:17.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[27010]: pgmap v757: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 783 B/s rd, 65 MiB/s wr, 1.25k op/s 2024-04-25T05:48:17.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1274458325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:17.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[27010]: from='client.25982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:17.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:17.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[37492]: pgmap v757: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 783 B/s rd, 65 MiB/s wr, 1.25k op/s 2024-04-25T05:48:17.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1274458325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:17.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:17 smithi043 ceph-mon[37492]: from='client.25982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:17.384 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:17.395 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":757,"stamp":"2024-04-25T05:48:15.961299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8305646131,"num_objects":126956,"num_object_clones":0,"num_object_copies":380868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126956,"num_whiteouts":0,"num_read":2248,"num_read_kb":1968,"num_write":130884,"num_write_kb":8214254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":1196524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78857,"ondisk_log_size":78857,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":29088564,"kb_used_data":24426956,"kb_used_omap":10,"kb_used_meta":4660725,"kb_avail":627127500,"statfs":{"total":671965249536,"available":642178560000,"internally_reserved":0,"allocated":25013202944,"data_stored":25009153935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4772582507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,2,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":428146712,"num_objects":6534,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6534,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":9778,"num_write_kb":521921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.840626"},"pg_stats":[{"pgid":"10.2","version":"419'8112","reported_seq":8186,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.947042+0000","last_change":"2024-04-25T05:46:37.979403+0000","last_active":"2024-04-25T05:48:13.947042+0000","last_peered":"2024-04-25T05:48:13.947042+0000","last_clean":"2024-04-25T05:48:13.947042+0000","last_became_active":"2024-04-25T05:46:37.979097+0000","last_became_peered":"2024-04-25T05:46:37.979097+0000","last_unstale":"2024-04-25T05:48:13.947042+0000","last_undegraded":"2024-04-25T05:48:13.947042+0000","last_fullsized":"2024-04-25T05:48:13.947042+0000","mapping_epoch":387,"log_start":"403'3900","ondisk_log_start":"403'3900","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508428288,"num_objects":7758,"num_object_clones":0,"num_object_copies":23274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":507840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":471,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155980+0000","last_change":"2024-04-25T05:46:09.856702+0000","last_active":"2024-04-25T05:48:13.155980+0000","last_peered":"2024-04-25T05:48:13.155980+0000","last_clean":"2024-04-25T05:48:13.155980+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:48:13.155980+0000","last_undegraded":"2024-04-25T05:48:13.155980+0000","last_fullsized":"2024-04-25T05:48:13.155980+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:46:09.856439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.856439+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":762,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157430+0000","last_change":"2024-04-25T05:46:03.753399+0000","last_active":"2024-04-25T05:48:13.157430+0000","last_peered":"2024-04-25T05:48:13.157430+0000","last_clean":"2024-04-25T05:48:13.157430+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:48:13.157430+0000","last_undegraded":"2024-04-25T05:48:13.157430+0000","last_fullsized":"2024-04-25T05:48:13.157430+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:03.753274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:03.753274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":130,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.158400+0000","last_change":"2024-04-25T05:46:35.013019+0000","last_active":"2024-04-25T05:48:13.158400+0000","last_peered":"2024-04-25T05:48:13.158400+0000","last_clean":"2024-04-25T05:48:13.158400+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:48:13.158400+0000","last_undegraded":"2024-04-25T05:48:13.158400+0000","last_fullsized":"2024-04-25T05:48:13.158400+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:45:36.284279+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.284279+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":440,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150724+0000","last_change":"2024-04-25T05:47:28.424745+0000","last_active":"2024-04-25T05:48:13.150724+0000","last_peered":"2024-04-25T05:48:13.150724+0000","last_clean":"2024-04-25T05:48:13.150724+0000","last_became_active":"2024-04-25T05:46:19.990605+0000","last_became_peered":"2024-04-25T05:46:19.990605+0000","last_unstale":"2024-04-25T05:48:13.150724+0000","last_undegraded":"2024-04-25T05:48:13.150724+0000","last_fullsized":"2024-04-25T05:48:13.150724+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.424618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.424618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:55.358144+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1075,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142945+0000","last_change":"2024-04-25T05:46:23.149014+0000","last_active":"2024-04-25T05:48:08.142945+0000","last_peered":"2024-04-25T05:48:08.142945+0000","last_clean":"2024-04-25T05:48:08.142945+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:48:08.142945+0000","last_undegraded":"2024-04-25T05:48:08.142945+0000","last_fullsized":"2024-04-25T05:48:08.142945+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"419'7979","reported_seq":8051,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.180934+0000","last_change":"2024-04-25T05:46:37.983434+0000","last_active":"2024-04-25T05:48:13.180934+0000","last_peered":"2024-04-25T05:48:13.180934+0000","last_clean":"2024-04-25T05:48:13.180934+0000","last_became_active":"2024-04-25T05:46:37.983078+0000","last_became_peered":"2024-04-25T05:46:37.983078+0000","last_unstale":"2024-04-25T05:48:13.180934+0000","last_undegraded":"2024-04-25T05:48:13.180934+0000","last_fullsized":"2024-04-25T05:48:13.180934+0000","mapping_epoch":387,"log_start":"401'3100","ondisk_log_start":"401'3100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510722048,"num_objects":7793,"num_object_clones":0,"num_object_copies":23379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7979,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":144,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.158514+0000","last_change":"2024-04-25T05:46:48.976027+0000","last_active":"2024-04-25T05:48:13.158514+0000","last_peered":"2024-04-25T05:48:13.158514+0000","last_clean":"2024-04-25T05:48:13.158514+0000","last_became_active":"2024-04-25T05:46:36.984563+0000","last_became_peered":"2024-04-25T05:46:36.984563+0000","last_unstale":"2024-04-25T05:48:13.158514+0000","last_undegraded":"2024-04-25T05:48:13.158514+0000","last_fullsized":"2024-04-25T05:48:13.158514+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:46:48.975826+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.975826+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.768088+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":476,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149870+0000","last_change":"2024-04-25T05:46:42.728489+0000","last_active":"2024-04-25T05:48:13.149870+0000","last_peered":"2024-04-25T05:48:13.149870+0000","last_clean":"2024-04-25T05:48:13.149870+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:48:13.149870+0000","last_undegraded":"2024-04-25T05:48:13.149870+0000","last_fullsized":"2024-04-25T05:48:13.149870+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:42.728367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:46:42.728367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097616+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":768,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150462+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:48:13.150462+0000","last_peered":"2024-04-25T05:48:13.150462+0000","last_clean":"2024-04-25T05:48:13.150462+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:48:13.150462+0000","last_undegraded":"2024-04-25T05:48:13.150462+0000","last_fullsized":"2024-04-25T05:48:13.150462+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594504+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.160285+0000","last_change":"2024-04-25T05:48:03.971319+0000","last_active":"2024-04-25T05:48:13.160285+0000","last_peered":"2024-04-25T05:48:13.160285+0000","last_clean":"2024-04-25T05:48:13.160285+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:48:13.160285+0000","last_undegraded":"2024-04-25T05:48:13.160285+0000","last_fullsized":"2024-04-25T05:48:13.160285+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.971192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.971192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.239829+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":673,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144598+0000","last_change":"2024-04-25T05:47:25.988415+0000","last_active":"2024-04-25T05:48:08.144598+0000","last_peered":"2024-04-25T05:48:08.144598+0000","last_clean":"2024-04-25T05:48:08.144598+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:48:08.144598+0000","last_undegraded":"2024-04-25T05:48:08.144598+0000","last_fullsized":"2024-04-25T05:48:08.144598+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.988268+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:28.271995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"418'8158","reported_seq":8292,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.148274+0000","last_change":"2024-04-25T05:48:08.191377+0000","last_active":"2024-04-25T05:48:13.148274+0000","last_peered":"2024-04-25T05:48:13.148274+0000","last_clean":"2024-04-25T05:48:13.148274+0000","last_became_active":"2024-04-25T05:46:37.984393+0000","last_became_peered":"2024-04-25T05:46:37.984393+0000","last_unstale":"2024-04-25T05:48:13.148274+0000","last_undegraded":"2024-04-25T05:48:13.148274+0000","last_fullsized":"2024-04-25T05:48:13.148274+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":5549,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.185819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523632640,"num_objects":7990,"num_object_clones":0,"num_object_copies":23970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":788,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149316+0000","last_change":"2024-04-25T05:47:24.775449+0000","last_active":"2024-04-25T05:48:13.149316+0000","last_peered":"2024-04-25T05:48:13.149316+0000","last_clean":"2024-04-25T05:48:13.149316+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:48:13.149316+0000","last_undegraded":"2024-04-25T05:48:13.149316+0000","last_fullsized":"2024-04-25T05:48:13.149316+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:47:24.775084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.775084+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.201360+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":786,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.132530+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:48:08.132530+0000","last_peered":"2024-04-25T05:48:08.132530+0000","last_clean":"2024-04-25T05:48:08.132530+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:48:08.132530+0000","last_undegraded":"2024-04-25T05:48:08.132530+0000","last_fullsized":"2024-04-25T05:48:08.132530+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.578308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144216+0000","last_change":"2024-04-25T05:46:34.982029+0000","last_active":"2024-04-25T05:48:08.144216+0000","last_peered":"2024-04-25T05:48:08.144216+0000","last_clean":"2024-04-25T05:48:08.144216+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:48:08.144216+0000","last_undegraded":"2024-04-25T05:48:08.144216+0000","last_fullsized":"2024-04-25T05:48:08.144216+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:18.666459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:46:18.666459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":756,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155610+0000","last_change":"2024-04-25T05:46:51.568352+0000","last_active":"2024-04-25T05:48:13.155610+0000","last_peered":"2024-04-25T05:48:13.155610+0000","last_clean":"2024-04-25T05:48:13.155610+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:48:13.155610+0000","last_undegraded":"2024-04-25T05:48:13.155610+0000","last_fullsized":"2024-04-25T05:48:13.155610+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:51.568255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:51.568255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228961+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"418'8217","reported_seq":8289,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.034599+0000","last_change":"2024-04-25T05:46:37.983353+0000","last_active":"2024-04-25T05:48:13.034599+0000","last_peered":"2024-04-25T05:48:13.034599+0000","last_clean":"2024-04-25T05:48:13.034599+0000","last_became_active":"2024-04-25T05:46:37.983037+0000","last_became_peered":"2024-04-25T05:46:37.983037+0000","last_unstale":"2024-04-25T05:48:13.034599+0000","last_undegraded":"2024-04-25T05:48:13.034599+0000","last_fullsized":"2024-04-25T05:48:13.034599+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.725131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527499264,"num_objects":8049,"num_object_clones":0,"num_object_copies":24147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8217,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":798,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151665+0000","last_change":"2024-04-25T05:47:18.459728+0000","last_active":"2024-04-25T05:48:13.151665+0000","last_peered":"2024-04-25T05:48:13.151665+0000","last_clean":"2024-04-25T05:48:13.151665+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:48:13.151665+0000","last_undegraded":"2024-04-25T05:48:13.151665+0000","last_fullsized":"2024-04-25T05:48:13.151665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:47:18.459587+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.459587+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:21.437539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154518+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:48:13.154518+0000","last_peered":"2024-04-25T05:48:13.154518+0000","last_clean":"2024-04-25T05:48:13.154518+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:48:13.154518+0000","last_undegraded":"2024-04-25T05:48:13.154518+0000","last_fullsized":"2024-04-25T05:48:13.154518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.434575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'15","reported_seq":99,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145612+0000","last_change":"2024-04-25T05:46:35.014060+0000","last_active":"2024-04-25T05:48:08.145612+0000","last_peered":"2024-04-25T05:48:08.145612+0000","last_clean":"2024-04-25T05:48:08.145612+0000","last_became_active":"2024-04-25T05:46:34.989992+0000","last_became_peered":"2024-04-25T05:46:34.989992+0000","last_unstale":"2024-04-25T05:48:08.145612+0000","last_undegraded":"2024-04-25T05:48:08.145612+0000","last_fullsized":"2024-04-25T05:48:08.145612+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141704+0000","last_change":"2024-04-25T05:46:34.990045+0000","last_active":"2024-04-25T05:48:08.141704+0000","last_peered":"2024-04-25T05:48:08.141704+0000","last_clean":"2024-04-25T05:48:08.141704+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:48:08.141704+0000","last_undegraded":"2024-04-25T05:48:08.141704+0000","last_fullsized":"2024-04-25T05:48:08.141704+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":637,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141342+0000","last_change":"2024-04-25T05:47:51.226427+0000","last_active":"2024-04-25T05:48:08.141342+0000","last_peered":"2024-04-25T05:48:08.141342+0000","last_clean":"2024-04-25T05:48:08.141342+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:48:08.141342+0000","last_undegraded":"2024-04-25T05:48:08.141342+0000","last_fullsized":"2024-04-25T05:48:08.141342+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:51.226126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.226126+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:11.223819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"418'8172","reported_seq":8326,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149095+0000","last_change":"2024-04-25T05:47:59.467502+0000","last_active":"2024-04-25T05:48:13.149095+0000","last_peered":"2024-04-25T05:48:13.149095+0000","last_clean":"2024-04-25T05:48:13.149095+0000","last_became_active":"2024-04-25T05:46:37.983600+0000","last_became_peered":"2024-04-25T05:46:37.983600+0000","last_unstale":"2024-04-25T05:48:13.149095+0000","last_undegraded":"2024-04-25T05:48:13.149095+0000","last_fullsized":"2024-04-25T05:48:13.149095+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7199","last_scrub_stamp":"2024-04-25T05:47:59.467288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.467288+0000","objects_scrubbed":7112,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:03.703400+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523894784,"num_objects":7994,"num_object_clones":0,"num_object_copies":23982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":517312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":758,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.132369+0000","last_change":"2024-04-25T05:47:19.690057+0000","last_active":"2024-04-25T05:48:08.132369+0000","last_peered":"2024-04-25T05:48:08.132369+0000","last_clean":"2024-04-25T05:48:08.132369+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:48:08.132369+0000","last_undegraded":"2024-04-25T05:48:08.132369+0000","last_fullsized":"2024-04-25T05:48:08.132369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.689810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.689810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:37.489431+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":907,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154607+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:48:13.154607+0000","last_peered":"2024-04-25T05:48:13.154607+0000","last_clean":"2024-04-25T05:48:13.154607+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:48:13.154607+0000","last_undegraded":"2024-04-25T05:48:13.154607+0000","last_fullsized":"2024-04-25T05:48:13.154607+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:00.492808+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":784,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.166104+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:48:13.166104+0000","last_peered":"2024-04-25T05:48:13.166104+0000","last_clean":"2024-04-25T05:48:13.166104+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:48:13.166104+0000","last_undegraded":"2024-04-25T05:48:13.166104+0000","last_fullsized":"2024-04-25T05:48:13.166104+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.712277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":434,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155876+0000","last_change":"2024-04-25T05:47:17.513777+0000","last_active":"2024-04-25T05:48:13.155876+0000","last_peered":"2024-04-25T05:48:13.155876+0000","last_clean":"2024-04-25T05:48:13.155876+0000","last_became_active":"2024-04-25T05:46:19.997163+0000","last_became_peered":"2024-04-25T05:46:19.997163+0000","last_unstale":"2024-04-25T05:48:13.155876+0000","last_undegraded":"2024-04-25T05:48:13.155876+0000","last_fullsized":"2024-04-25T05:48:13.155876+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:17.513668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:17.513668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.470303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":987,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.133149+0000","last_change":"2024-04-25T05:47:28.786464+0000","last_active":"2024-04-25T05:48:08.133149+0000","last_peered":"2024-04-25T05:48:08.133149+0000","last_clean":"2024-04-25T05:48:08.133149+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:48:08.133149+0000","last_undegraded":"2024-04-25T05:48:08.133149+0000","last_fullsized":"2024-04-25T05:48:08.133149+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:28.786340+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.786340+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.786586+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"419'8097","reported_seq":8327,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156440+0000","last_change":"2024-04-25T05:48:08.175544+0000","last_active":"2024-04-25T05:48:13.156440+0000","last_peered":"2024-04-25T05:48:13.156440+0000","last_clean":"2024-04-25T05:48:13.156440+0000","last_became_active":"2024-04-25T05:46:37.982701+0000","last_became_peered":"2024-04-25T05:46:37.982701+0000","last_unstale":"2024-04-25T05:48:13.156440+0000","last_undegraded":"2024-04-25T05:48:13.156440+0000","last_fullsized":"2024-04-25T05:48:13.156440+0000","mapping_epoch":387,"log_start":"401'3100","ondisk_log_start":"401'3100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7494","last_scrub_stamp":"2024-04-25T05:48:05.024302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.024302+0000","objects_scrubbed":7249,"log_size":4997,"log_dups_size":0,"ondisk_log_size":4997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.164610+0000","scrub_duration":2970,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516620288,"num_objects":7883,"num_object_clones":0,"num_object_copies":23649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":927,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154745+0000","last_change":"2024-04-25T05:45:36.207163+0000","last_active":"2024-04-25T05:48:13.154745+0000","last_peered":"2024-04-25T05:48:13.154745+0000","last_clean":"2024-04-25T05:48:13.154745+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:48:13.154745+0000","last_undegraded":"2024-04-25T05:48:13.154745+0000","last_fullsized":"2024-04-25T05:48:13.154745+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:45:34.969209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:34.969209+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228518+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":102,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.146127+0000","last_change":"2024-04-25T05:48:05.149694+0000","last_active":"2024-04-25T05:48:08.146127+0000","last_peered":"2024-04-25T05:48:08.146127+0000","last_clean":"2024-04-25T05:48:08.146127+0000","last_became_active":"2024-04-25T05:46:34.980031+0000","last_became_peered":"2024-04-25T05:46:34.980031+0000","last_unstale":"2024-04-25T05:48:08.146127+0000","last_undegraded":"2024-04-25T05:48:08.146127+0000","last_fullsized":"2024-04-25T05:48:08.146127+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:48:05.148321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.148321+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:33.058689+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":456,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150914+0000","last_change":"2024-04-25T05:48:04.608424+0000","last_active":"2024-04-25T05:48:13.150914+0000","last_peered":"2024-04-25T05:48:13.150914+0000","last_clean":"2024-04-25T05:48:13.150914+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:48:13.150914+0000","last_undegraded":"2024-04-25T05:48:13.150914+0000","last_fullsized":"2024-04-25T05:48:13.150914+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.608284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:05.704760+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":134,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.164131+0000","last_change":"2024-04-25T05:47:14.571538+0000","last_active":"2024-04-25T05:48:13.164131+0000","last_peered":"2024-04-25T05:48:13.164131+0000","last_clean":"2024-04-25T05:48:13.164131+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:48:13.164131+0000","last_undegraded":"2024-04-25T05:48:13.164131+0000","last_fullsized":"2024-04-25T05:48:13.164131+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:14.571417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:14.571417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.666298+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":945,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155437+0000","last_change":"2024-04-25T05:46:37.983775+0000","last_active":"2024-04-25T05:48:13.155437+0000","last_peered":"2024-04-25T05:48:13.155437+0000","last_clean":"2024-04-25T05:48:13.155437+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:48:13.155437+0000","last_undegraded":"2024-04-25T05:48:13.155437+0000","last_fullsized":"2024-04-25T05:48:13.155437+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.882669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.882669+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"418'8186","reported_seq":8258,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.064704+0000","last_change":"2024-04-25T05:46:37.986037+0000","last_active":"2024-04-25T05:48:13.064704+0000","last_peered":"2024-04-25T05:48:13.064704+0000","last_clean":"2024-04-25T05:48:13.064704+0000","last_became_active":"2024-04-25T05:46:37.985670+0000","last_became_peered":"2024-04-25T05:46:37.985670+0000","last_unstale":"2024-04-25T05:48:13.064704+0000","last_undegraded":"2024-04-25T05:48:13.064704+0000","last_fullsized":"2024-04-25T05:48:13.064704+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.726596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524156928,"num_objects":7998,"num_object_clones":0,"num_object_copies":23994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8186,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":516,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150271+0000","last_change":"2024-04-25T05:46:46.694407+0000","last_active":"2024-04-25T05:48:13.150271+0000","last_peered":"2024-04-25T05:48:13.150271+0000","last_clean":"2024-04-25T05:48:13.150271+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:48:13.150271+0000","last_undegraded":"2024-04-25T05:48:13.150271+0000","last_fullsized":"2024-04-25T05:48:13.150271+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:46:46.694273+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.694273+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097566+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":494,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151779+0000","last_change":"2024-04-25T05:46:45.354890+0000","last_active":"2024-04-25T05:48:13.151779+0000","last_peered":"2024-04-25T05:48:13.151779+0000","last_clean":"2024-04-25T05:48:13.151779+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:48:13.151779+0000","last_undegraded":"2024-04-25T05:48:13.151779+0000","last_fullsized":"2024-04-25T05:48:13.151779+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:46:45.354712+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.354712+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594614+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141729+0000","last_change":"2024-04-25T05:47:15.018494+0000","last_active":"2024-04-25T05:48:08.141729+0000","last_peered":"2024-04-25T05:48:08.141729+0000","last_clean":"2024-04-25T05:48:08.141729+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:48:08.141729+0000","last_undegraded":"2024-04-25T05:48:08.141729+0000","last_fullsized":"2024-04-25T05:48:08.141729+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:15.018379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:15.018379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.914407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154116+0000","last_change":"2024-04-25T05:46:44.669771+0000","last_active":"2024-04-25T05:48:13.154116+0000","last_peered":"2024-04-25T05:48:13.154116+0000","last_clean":"2024-04-25T05:48:13.154116+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:48:13.154116+0000","last_undegraded":"2024-04-25T05:48:13.154116+0000","last_fullsized":"2024-04-25T05:48:13.154116+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.669615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.669615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228896+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":939,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151576+0000","last_change":"2024-04-25T05:47:19.766253+0000","last_active":"2024-04-25T05:48:13.151576+0000","last_peered":"2024-04-25T05:48:13.151576+0000","last_clean":"2024-04-25T05:48:13.151576+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:48:13.151576+0000","last_undegraded":"2024-04-25T05:48:13.151576+0000","last_fullsized":"2024-04-25T05:48:13.151576+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:19.766122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.766122+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:25.464800+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"418'7984","reported_seq":8059,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.148622+0000","last_change":"2024-04-25T05:46:37.986967+0000","last_active":"2024-04-25T05:48:13.148622+0000","last_peered":"2024-04-25T05:48:13.148622+0000","last_clean":"2024-04-25T05:48:13.148622+0000","last_became_active":"2024-04-25T05:46:37.986694+0000","last_became_peered":"2024-04-25T05:46:37.986694+0000","last_unstale":"2024-04-25T05:48:13.148622+0000","last_undegraded":"2024-04-25T05:48:13.148622+0000","last_fullsized":"2024-04-25T05:48:13.148622+0000","mapping_epoch":387,"log_start":"401'3000","ondisk_log_start":"401'3000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512294936,"num_objects":7818,"num_object_clones":0,"num_object_copies":23454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7984,"num_write_kb":505601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":514,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149914+0000","last_change":"2024-04-25T05:47:51.904861+0000","last_active":"2024-04-25T05:48:13.149914+0000","last_peered":"2024-04-25T05:48:13.149914+0000","last_clean":"2024-04-25T05:48:13.149914+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:48:13.149914+0000","last_undegraded":"2024-04-25T05:48:13.149914+0000","last_fullsized":"2024-04-25T05:48:13.149914+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:51.904568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.904568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.642706+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":122,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142685+0000","last_change":"2024-04-25T05:46:36.996561+0000","last_active":"2024-04-25T05:48:08.142685+0000","last_peered":"2024-04-25T05:48:08.142685+0000","last_clean":"2024-04-25T05:48:08.142685+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:48:08.142685+0000","last_undegraded":"2024-04-25T05:48:08.142685+0000","last_fullsized":"2024-04-25T05:48:08.142685+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:46:07.457049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.457049+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":790,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150222+0000","last_change":"2024-04-25T05:47:53.584048+0000","last_active":"2024-04-25T05:48:13.150222+0000","last_peered":"2024-04-25T05:48:13.150222+0000","last_clean":"2024-04-25T05:48:13.150222+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:48:13.150222+0000","last_undegraded":"2024-04-25T05:48:13.150222+0000","last_fullsized":"2024-04-25T05:48:13.150222+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:53.583930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:53.583930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.954029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":764,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151180+0000","last_change":"2024-04-25T05:46:47.388550+0000","last_active":"2024-04-25T05:48:13.151180+0000","last_peered":"2024-04-25T05:48:13.151180+0000","last_clean":"2024-04-25T05:48:13.151180+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:48:13.151180+0000","last_undegraded":"2024-04-25T05:48:13.151180+0000","last_fullsized":"2024-04-25T05:48:13.151180+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.388440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.388440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.581611+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":957,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149203+0000","last_change":"2024-04-25T05:46:52.649991+0000","last_active":"2024-04-25T05:48:13.149203+0000","last_peered":"2024-04-25T05:48:13.149203+0000","last_clean":"2024-04-25T05:48:13.149203+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:48:13.149203+0000","last_undegraded":"2024-04-25T05:48:13.149203+0000","last_fullsized":"2024-04-25T05:48:13.149203+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:52.649850+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.649850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"419'8125","reported_seq":8197,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.161368+0000","last_change":"2024-04-25T05:46:37.986465+0000","last_active":"2024-04-25T05:48:13.161368+0000","last_peered":"2024-04-25T05:48:13.161368+0000","last_clean":"2024-04-25T05:48:13.161368+0000","last_became_active":"2024-04-25T05:46:37.986157+0000","last_became_peered":"2024-04-25T05:46:37.986157+0000","last_unstale":"2024-04-25T05:48:13.161368+0000","last_undegraded":"2024-04-25T05:48:13.161368+0000","last_fullsized":"2024-04-25T05:48:13.161368+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517013504,"num_objects":7889,"num_object_clones":0,"num_object_copies":23667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144222+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:48:08.144222+0000","last_peered":"2024-04-25T05:48:08.144222+0000","last_clean":"2024-04-25T05:48:08.144222+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:48:08.144222+0000","last_undegraded":"2024-04-25T05:48:08.144222+0000","last_fullsized":"2024-04-25T05:48:08.144222+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:16.696528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":98,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145737+0000","last_change":"2024-04-25T05:47:55.201717+0000","last_active":"2024-04-25T05:48:08.145737+0000","last_peered":"2024-04-25T05:48:08.145737+0000","last_clean":"2024-04-25T05:48:08.145737+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:48:08.145737+0000","last_undegraded":"2024-04-25T05:48:08.145737+0000","last_fullsized":"2024-04-25T05:48:08.145737+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:55.201430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.201430+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:56.091558+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1646,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:11.630886+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:48:11.630886+0000","last_peered":"2024-04-25T05:48:11.630886+0000","last_clean":"2024-04-25T05:48:11.630886+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:48:11.630886+0000","last_undegraded":"2024-04-25T05:48:11.630886+0000","last_fullsized":"2024-04-25T05:48:11.630886+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.661889+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":836,"num_read_kb":831,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":383,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.132617+0000","last_change":"2024-04-25T05:47:25.731787+0000","last_active":"2024-04-25T05:48:08.132617+0000","last_peered":"2024-04-25T05:48:08.132617+0000","last_clean":"2024-04-25T05:48:08.132617+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:48:08.132617+0000","last_undegraded":"2024-04-25T05:48:08.132617+0000","last_fullsized":"2024-04-25T05:48:08.132617+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.731370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:45.013111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149935+0000","last_change":"2024-04-25T05:46:34.975022+0000","last_active":"2024-04-25T05:48:13.149935+0000","last_peered":"2024-04-25T05:48:13.149935+0000","last_clean":"2024-04-25T05:48:13.149935+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:48:13.149935+0000","last_undegraded":"2024-04-25T05:48:13.149935+0000","last_fullsized":"2024-04-25T05:48:13.149935+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:36.702085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.702085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594555+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":934,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.164038+0000","last_change":"2024-04-25T05:46:43.418452+0000","last_active":"2024-04-25T05:48:13.164038+0000","last_peered":"2024-04-25T05:48:13.164038+0000","last_clean":"2024-04-25T05:48:13.164038+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:48:13.164038+0000","last_undegraded":"2024-04-25T05:48:13.164038+0000","last_fullsized":"2024-04-25T05:48:13.164038+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:43.418024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.418024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594673+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"418'8160","reported_seq":8313,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149803+0000","last_change":"2024-04-25T05:48:01.589216+0000","last_active":"2024-04-25T05:48:13.149803+0000","last_peered":"2024-04-25T05:48:13.149803+0000","last_clean":"2024-04-25T05:48:13.149803+0000","last_became_active":"2024-04-25T05:46:37.985141+0000","last_became_peered":"2024-04-25T05:46:37.985141+0000","last_unstale":"2024-04-25T05:48:13.149803+0000","last_undegraded":"2024-04-25T05:48:13.149803+0000","last_fullsized":"2024-04-25T05:48:13.149803+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7377","last_scrub_stamp":"2024-04-25T05:48:01.588877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.588877+0000","objects_scrubbed":7283,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:04.287088+0000","scrub_duration":2528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522584064,"num_objects":7974,"num_object_clones":0,"num_object_copies":23922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":780,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139547+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:48:08.139547+0000","last_peered":"2024-04-25T05:48:08.139547+0000","last_clean":"2024-04-25T05:48:08.139547+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:48:08.139547+0000","last_undegraded":"2024-04-25T05:48:08.139547+0000","last_fullsized":"2024-04-25T05:48:08.139547+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:57.117658+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":616,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149058+0000","last_change":"2024-04-25T05:47:29.876796+0000","last_active":"2024-04-25T05:48:13.149058+0000","last_peered":"2024-04-25T05:48:13.149058+0000","last_clean":"2024-04-25T05:48:13.149058+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:48:13.149058+0000","last_undegraded":"2024-04-25T05:48:13.149058+0000","last_fullsized":"2024-04-25T05:48:13.149058+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:47:29.876498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:29.876498+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:37.200762+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":768,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155519+0000","last_change":"2024-04-25T05:46:34.970673+0000","last_active":"2024-04-25T05:48:13.155519+0000","last_peered":"2024-04-25T05:48:13.155519+0000","last_clean":"2024-04-25T05:48:13.155519+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:48:13.155519+0000","last_undegraded":"2024-04-25T05:48:13.155519+0000","last_fullsized":"2024-04-25T05:48:13.155519+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.811914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.811914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1068,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145836+0000","last_change":"2024-04-25T05:46:06.483237+0000","last_active":"2024-04-25T05:48:08.145836+0000","last_peered":"2024-04-25T05:48:08.145836+0000","last_clean":"2024-04-25T05:48:08.145836+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:48:08.145836+0000","last_undegraded":"2024-04-25T05:48:08.145836+0000","last_fullsized":"2024-04-25T05:48:08.145836+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:06.483108+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.483108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151140+0000","last_change":"2024-04-25T05:47:02.157593+0000","last_active":"2024-04-25T05:48:13.151140+0000","last_peered":"2024-04-25T05:48:13.151140+0000","last_clean":"2024-04-25T05:48:13.151140+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:48:13.151140+0000","last_undegraded":"2024-04-25T05:48:13.151140+0000","last_fullsized":"2024-04-25T05:48:13.151140+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:37.037435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.037435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"418'8079","reported_seq":8148,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.016570+0000","last_change":"2024-04-25T05:46:37.984020+0000","last_active":"2024-04-25T05:48:13.016570+0000","last_peered":"2024-04-25T05:48:13.016570+0000","last_clean":"2024-04-25T05:48:13.016570+0000","last_became_active":"2024-04-25T05:46:37.983253+0000","last_became_peered":"2024-04-25T05:46:37.983253+0000","last_unstale":"2024-04-25T05:48:13.016570+0000","last_undegraded":"2024-04-25T05:48:13.016570+0000","last_fullsized":"2024-04-25T05:48:13.016570+0000","mapping_epoch":387,"log_start":"401'3400","ondisk_log_start":"401'3400","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:14.106704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518455296,"num_objects":7911,"num_object_clones":0,"num_object_copies":23733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":108,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.158531+0000","last_change":"2024-04-25T05:47:33.788454+0000","last_active":"2024-04-25T05:48:13.158531+0000","last_peered":"2024-04-25T05:48:13.158531+0000","last_clean":"2024-04-25T05:48:13.158531+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:48:13.158531+0000","last_undegraded":"2024-04-25T05:48:13.158531+0000","last_fullsized":"2024-04-25T05:48:13.158531+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.788331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.788331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:38.579563+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":899,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150650+0000","last_change":"2024-04-25T05:48:02.601745+0000","last_active":"2024-04-25T05:48:13.150650+0000","last_peered":"2024-04-25T05:48:13.150650+0000","last_clean":"2024-04-25T05:48:13.150650+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:48:13.150650+0000","last_undegraded":"2024-04-25T05:48:13.150650+0000","last_fullsized":"2024-04-25T05:48:13.150650+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:48:02.601574+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.601574+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.802514+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.158096+0000","last_change":"2024-04-25T05:46:50.947463+0000","last_active":"2024-04-25T05:48:13.158096+0000","last_peered":"2024-04-25T05:48:13.158096+0000","last_clean":"2024-04-25T05:48:13.158096+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:48:13.158096+0000","last_undegraded":"2024-04-25T05:48:13.158096+0000","last_fullsized":"2024-04-25T05:48:13.158096+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.947345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.947345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":789,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150131+0000","last_change":"2024-04-25T05:46:19.992163+0000","last_active":"2024-04-25T05:48:13.150131+0000","last_peered":"2024-04-25T05:48:13.150131+0000","last_clean":"2024-04-25T05:48:13.150131+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:48:13.150131+0000","last_undegraded":"2024-04-25T05:48:13.150131+0000","last_fullsized":"2024-04-25T05:48:13.150131+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594339+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":758,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139409+0000","last_change":"2024-04-25T05:47:23.888942+0000","last_active":"2024-04-25T05:48:08.139409+0000","last_peered":"2024-04-25T05:48:08.139409+0000","last_clean":"2024-04-25T05:48:08.139409+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:48:08.139409+0000","last_undegraded":"2024-04-25T05:48:08.139409+0000","last_fullsized":"2024-04-25T05:48:08.139409+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:23.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.354969+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1075,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.138693+0000","last_change":"2024-04-25T05:46:23.137608+0000","last_active":"2024-04-25T05:48:08.138693+0000","last_peered":"2024-04-25T05:48:08.138693+0000","last_clean":"2024-04-25T05:48:08.138693+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:48:08.138693+0000","last_undegraded":"2024-04-25T05:48:08.138693+0000","last_fullsized":"2024-04-25T05:48:08.138693+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"418'8126","reported_seq":8200,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149462+0000","last_change":"2024-04-25T05:46:37.985580+0000","last_active":"2024-04-25T05:48:13.149462+0000","last_peered":"2024-04-25T05:48:13.149462+0000","last_clean":"2024-04-25T05:48:13.149462+0000","last_became_active":"2024-04-25T05:46:37.985031+0000","last_became_peered":"2024-04-25T05:46:37.985031+0000","last_unstale":"2024-04-25T05:48:13.149462+0000","last_undegraded":"2024-04-25T05:48:13.149462+0000","last_fullsized":"2024-04-25T05:48:13.149462+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522452992,"num_objects":7972,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8126,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":377,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150492+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:48:13.150492+0000","last_peered":"2024-04-25T05:48:13.150492+0000","last_clean":"2024-04-25T05:48:13.150492+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:48:13.150492+0000","last_undegraded":"2024-04-25T05:48:13.150492+0000","last_fullsized":"2024-04-25T05:48:13.150492+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594479+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1069,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150354+0000","last_change":"2024-04-25T05:46:52.415241+0000","last_active":"2024-04-25T05:48:13.150354+0000","last_peered":"2024-04-25T05:48:13.150354+0000","last_clean":"2024-04-25T05:48:13.150354+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:48:13.150354+0000","last_undegraded":"2024-04-25T05:48:13.150354+0000","last_fullsized":"2024-04-25T05:48:13.150354+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:46:52.415149+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.415149+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":547,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150132+0000","last_change":"2024-04-25T05:46:50.698635+0000","last_active":"2024-04-25T05:48:13.150132+0000","last_peered":"2024-04-25T05:48:13.150132+0000","last_clean":"2024-04-25T05:48:13.150132+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:48:13.150132+0000","last_undegraded":"2024-04-25T05:48:13.150132+0000","last_fullsized":"2024-04-25T05:48:13.150132+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.698507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.698507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":88,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141149+0000","last_change":"2024-04-25T05:46:49.868732+0000","last_active":"2024-04-25T05:48:08.141149+0000","last_peered":"2024-04-25T05:48:08.141149+0000","last_clean":"2024-04-25T05:48:08.141149+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:48:08.141149+0000","last_undegraded":"2024-04-25T05:48:08.141149+0000","last_fullsized":"2024-04-25T05:48:08.141149+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:46:49.868648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.868648+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.090373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":732,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155606+0000","last_change":"2024-04-25T05:46:16.748067+0000","last_active":"2024-04-25T05:48:13.155606+0000","last_peered":"2024-04-25T05:48:13.155606+0000","last_clean":"2024-04-25T05:48:13.155606+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:48:13.155606+0000","last_undegraded":"2024-04-25T05:48:13.155606+0000","last_fullsized":"2024-04-25T05:48:13.155606+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.747963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.747963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":776,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.146223+0000","last_change":"2024-04-25T05:46:37.982952+0000","last_active":"2024-04-25T05:48:08.146223+0000","last_peered":"2024-04-25T05:48:08.146223+0000","last_clean":"2024-04-25T05:48:08.146223+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:48:08.146223+0000","last_undegraded":"2024-04-25T05:48:08.146223+0000","last_fullsized":"2024-04-25T05:48:08.146223+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:11.595125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:11.595125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"418'8044","reported_seq":8116,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.158029+0000","last_change":"2024-04-25T05:46:37.986363+0000","last_active":"2024-04-25T05:48:13.158029+0000","last_peered":"2024-04-25T05:48:13.158029+0000","last_clean":"2024-04-25T05:48:13.158029+0000","last_became_active":"2024-04-25T05:46:37.986037+0000","last_became_peered":"2024-04-25T05:46:37.986037+0000","last_unstale":"2024-04-25T05:48:13.158029+0000","last_undegraded":"2024-04-25T05:48:13.158029+0000","last_fullsized":"2024-04-25T05:48:13.158029+0000","mapping_epoch":387,"log_start":"401'3200","ondisk_log_start":"401'3200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514588672,"num_objects":7852,"num_object_clones":0,"num_object_copies":23556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":470,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.163875+0000","last_change":"2024-04-25T05:47:23.436651+0000","last_active":"2024-04-25T05:48:13.163875+0000","last_peered":"2024-04-25T05:48:13.163875+0000","last_clean":"2024-04-25T05:48:13.163875+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:48:13.163875+0000","last_undegraded":"2024-04-25T05:48:13.163875+0000","last_fullsized":"2024-04-25T05:48:13.163875+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:23.436515+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.436515+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:52.391863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141501+0000","last_change":"2024-04-25T05:46:34.977874+0000","last_active":"2024-04-25T05:48:08.141501+0000","last_peered":"2024-04-25T05:48:08.141501+0000","last_clean":"2024-04-25T05:48:08.141501+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:48:08.141501+0000","last_undegraded":"2024-04-25T05:48:08.141501+0000","last_fullsized":"2024-04-25T05:48:08.141501+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.162412+0000","last_change":"2024-04-25T05:46:19.996324+0000","last_active":"2024-04-25T05:48:13.162412+0000","last_peered":"2024-04-25T05:48:13.162412+0000","last_clean":"2024-04-25T05:48:13.162412+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:48:13.162412+0000","last_undegraded":"2024-04-25T05:48:13.162412+0000","last_fullsized":"2024-04-25T05:48:13.162412+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":375,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142831+0000","last_change":"2024-04-25T05:47:19.913147+0000","last_active":"2024-04-25T05:48:08.142831+0000","last_peered":"2024-04-25T05:48:08.142831+0000","last_clean":"2024-04-25T05:48:08.142831+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:48:08.142831+0000","last_undegraded":"2024-04-25T05:48:08.142831+0000","last_fullsized":"2024-04-25T05:48:08.142831+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.913041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.913041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.287683+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"419'8248","reported_seq":8322,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.973754+0000","last_change":"2024-04-25T05:46:37.979066+0000","last_active":"2024-04-25T05:48:13.973754+0000","last_peered":"2024-04-25T05:48:13.973754+0000","last_clean":"2024-04-25T05:48:13.973754+0000","last_became_active":"2024-04-25T05:46:37.978316+0000","last_became_peered":"2024-04-25T05:46:37.978316+0000","last_unstale":"2024-04-25T05:48:13.973754+0000","last_undegraded":"2024-04-25T05:48:13.973754+0000","last_fullsized":"2024-04-25T05:48:13.973754+0000","mapping_epoch":387,"log_start":"403'4100","ondisk_log_start":"403'4100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519045120,"num_objects":7920,"num_object_clones":0,"num_object_copies":23760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8248,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":530,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149787+0000","last_change":"2024-04-25T05:47:31.914632+0000","last_active":"2024-04-25T05:48:13.149787+0000","last_peered":"2024-04-25T05:48:13.149787+0000","last_clean":"2024-04-25T05:48:13.149787+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:48:13.149787+0000","last_undegraded":"2024-04-25T05:48:13.149787+0000","last_fullsized":"2024-04-25T05:48:13.149787+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'41","last_scrub_stamp":"2024-04-25T05:47:31.914155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.914155+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:56.812560+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":804,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139301+0000","last_change":"2024-04-25T05:46:45.682570+0000","last_active":"2024-04-25T05:48:08.139301+0000","last_peered":"2024-04-25T05:48:08.139301+0000","last_clean":"2024-04-25T05:48:08.139301+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:48:08.139301+0000","last_undegraded":"2024-04-25T05:48:08.139301+0000","last_fullsized":"2024-04-25T05:48:08.139301+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:46:45.682441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.682441+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:12.440746+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":108,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156915+0000","last_change":"2024-04-25T05:47:59.817680+0000","last_active":"2024-04-25T05:48:13.156915+0000","last_peered":"2024-04-25T05:48:13.156915+0000","last_clean":"2024-04-25T05:48:13.156915+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:48:13.156915+0000","last_undegraded":"2024-04-25T05:48:13.156915+0000","last_fullsized":"2024-04-25T05:48:13.156915+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.817559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.817559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:17.472338+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":758,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145566+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:48:08.145566+0000","last_peered":"2024-04-25T05:48:08.145566+0000","last_clean":"2024-04-25T05:48:08.145566+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:48:08.145566+0000","last_undegraded":"2024-04-25T05:48:08.145566+0000","last_fullsized":"2024-04-25T05:48:08.145566+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.889858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"418'8112","reported_seq":8186,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151172+0000","last_change":"2024-04-25T05:46:37.986817+0000","last_active":"2024-04-25T05:48:13.151172+0000","last_peered":"2024-04-25T05:48:13.151172+0000","last_clean":"2024-04-25T05:48:13.151172+0000","last_became_active":"2024-04-25T05:46:37.983489+0000","last_became_peered":"2024-04-25T05:46:37.983489+0000","last_unstale":"2024-04-25T05:48:13.151172+0000","last_undegraded":"2024-04-25T05:48:13.151172+0000","last_fullsized":"2024-04-25T05:48:13.151172+0000","mapping_epoch":387,"log_start":"401'3100","ondisk_log_start":"401'3100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520093696,"num_objects":7936,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":513536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":108,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157463+0000","last_change":"2024-04-25T05:47:57.840420+0000","last_active":"2024-04-25T05:48:13.157463+0000","last_peered":"2024-04-25T05:48:13.157463+0000","last_clean":"2024-04-25T05:48:13.157463+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:48:13.157463+0000","last_undegraded":"2024-04-25T05:48:13.157463+0000","last_fullsized":"2024-04-25T05:48:13.157463+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:57.839912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:57.839912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:02.350285+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150819+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:48:13.150819+0000","last_peered":"2024-04-25T05:48:13.150819+0000","last_clean":"2024-04-25T05:48:13.150819+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:48:13.150819+0000","last_undegraded":"2024-04-25T05:48:13.150819+0000","last_fullsized":"2024-04-25T05:48:13.150819+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594424+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":758,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139302+0000","last_change":"2024-04-25T05:46:46.685480+0000","last_active":"2024-04-25T05:48:08.139302+0000","last_peered":"2024-04-25T05:48:08.139302+0000","last_clean":"2024-04-25T05:48:08.139302+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:48:08.139302+0000","last_undegraded":"2024-04-25T05:48:08.139302+0000","last_fullsized":"2024-04-25T05:48:08.139302+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.685365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.685365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":744,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157712+0000","last_change":"2024-04-25T05:46:52.779088+0000","last_active":"2024-04-25T05:48:13.157712+0000","last_peered":"2024-04-25T05:48:13.157712+0000","last_clean":"2024-04-25T05:48:13.157712+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:48:13.157712+0000","last_undegraded":"2024-04-25T05:48:13.157712+0000","last_fullsized":"2024-04-25T05:48:13.157712+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:52.779013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.779013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"418'8171","reported_seq":8240,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.004448+0000","last_change":"2024-04-25T05:46:37.981240+0000","last_active":"2024-04-25T05:48:13.004448+0000","last_peered":"2024-04-25T05:48:13.004448+0000","last_clean":"2024-04-25T05:48:13.004448+0000","last_became_active":"2024-04-25T05:46:37.980913+0000","last_became_peered":"2024-04-25T05:46:37.980913+0000","last_unstale":"2024-04-25T05:48:13.004448+0000","last_undegraded":"2024-04-25T05:48:13.004448+0000","last_fullsized":"2024-04-25T05:48:13.004448+0000","mapping_epoch":387,"log_start":"401'3500","ondisk_log_start":"401'3500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:14.106667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523567104,"num_objects":7989,"num_object_clones":0,"num_object_copies":23967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8171,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":953,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142393+0000","last_change":"2024-04-25T05:46:47.674917+0000","last_active":"2024-04-25T05:48:08.142393+0000","last_peered":"2024-04-25T05:48:08.142393+0000","last_clean":"2024-04-25T05:48:08.142393+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:48:08.142393+0000","last_undegraded":"2024-04-25T05:48:08.142393+0000","last_fullsized":"2024-04-25T05:48:08.142393+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:46:47.674789+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.674789+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":104,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144701+0000","last_change":"2024-04-25T05:47:23.979227+0000","last_active":"2024-04-25T05:48:08.144701+0000","last_peered":"2024-04-25T05:48:08.144701+0000","last_clean":"2024-04-25T05:48:08.144701+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:48:08.144701+0000","last_undegraded":"2024-04-25T05:48:08.144701+0000","last_fullsized":"2024-04-25T05:48:08.144701+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:47:23.979075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.979075+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:53.818392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":798,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.140543+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:48:08.140543+0000","last_peered":"2024-04-25T05:48:08.140543+0000","last_clean":"2024-04-25T05:48:08.140543+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:48:08.140543+0000","last_undegraded":"2024-04-25T05:48:08.140543+0000","last_fullsized":"2024-04-25T05:48:08.140543+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141249+0000","last_change":"2024-04-25T05:46:34.973794+0000","last_active":"2024-04-25T05:48:08.141249+0000","last_peered":"2024-04-25T05:48:08.141249+0000","last_clean":"2024-04-25T05:48:08.141249+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:48:08.141249+0000","last_undegraded":"2024-04-25T05:48:08.141249+0000","last_fullsized":"2024-04-25T05:48:08.141249+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:10.692888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.692888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":777,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.166736+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:48:13.166736+0000","last_peered":"2024-04-25T05:48:13.166736+0000","last_clean":"2024-04-25T05:48:13.166736+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:48:13.166736+0000","last_undegraded":"2024-04-25T05:48:13.166736+0000","last_fullsized":"2024-04-25T05:48:13.166736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:47.643410+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":505,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149064+0000","last_change":"2024-04-25T05:46:47.655135+0000","last_active":"2024-04-25T05:48:13.149064+0000","last_peered":"2024-04-25T05:48:13.149064+0000","last_clean":"2024-04-25T05:48:13.149064+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:48:13.149064+0000","last_undegraded":"2024-04-25T05:48:13.149064+0000","last_fullsized":"2024-04-25T05:48:13.149064+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.655013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.655013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097434+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145657+0000","last_change":"2024-04-25T05:47:56.163151+0000","last_active":"2024-04-25T05:48:08.145657+0000","last_peered":"2024-04-25T05:48:08.145657+0000","last_clean":"2024-04-25T05:48:08.145657+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:48:08.145657+0000","last_undegraded":"2024-04-25T05:48:08.145657+0000","last_fullsized":"2024-04-25T05:48:08.145657+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:56.163019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.163019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:01.308762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":757,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139080+0000","last_change":"2024-04-25T05:47:32.863388+0000","last_active":"2024-04-25T05:48:08.139080+0000","last_peered":"2024-04-25T05:48:08.139080+0000","last_clean":"2024-04-25T05:48:08.139080+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:48:08.139080+0000","last_undegraded":"2024-04-25T05:48:08.139080+0000","last_fullsized":"2024-04-25T05:48:08.139080+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:32.863250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.863250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:46.805628+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":458,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155492+0000","last_change":"2024-04-25T05:46:20.806968+0000","last_active":"2024-04-25T05:48:13.155492+0000","last_peered":"2024-04-25T05:48:13.155492+0000","last_clean":"2024-04-25T05:48:13.155492+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:48:13.155492+0000","last_undegraded":"2024-04-25T05:48:13.155492+0000","last_fullsized":"2024-04-25T05:48:13.155492+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:46:20.806869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:20.806869+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":776,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142571+0000","last_change":"2024-04-25T05:47:31.715369+0000","last_active":"2024-04-25T05:48:08.142571+0000","last_peered":"2024-04-25T05:48:08.142571+0000","last_clean":"2024-04-25T05:48:08.142571+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:48:08.142571+0000","last_undegraded":"2024-04-25T05:48:08.142571+0000","last_fullsized":"2024-04-25T05:48:08.142571+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.715261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.794511+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":735,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.166471+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:48:13.166471+0000","last_peered":"2024-04-25T05:48:13.166471+0000","last_clean":"2024-04-25T05:48:13.166471+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:48:13.166471+0000","last_undegraded":"2024-04-25T05:48:13.166471+0000","last_fullsized":"2024-04-25T05:48:13.166471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:06.785637+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":94,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157043+0000","last_change":"2024-04-25T05:46:19.984536+0000","last_active":"2024-04-25T05:48:13.157043+0000","last_peered":"2024-04-25T05:48:13.157043+0000","last_clean":"2024-04-25T05:48:13.157043+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:48:13.157043+0000","last_undegraded":"2024-04-25T05:48:13.157043+0000","last_fullsized":"2024-04-25T05:48:13.157043+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.695090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.695090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":88,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141415+0000","last_change":"2024-04-25T05:46:35.006236+0000","last_active":"2024-04-25T05:48:08.141415+0000","last_peered":"2024-04-25T05:48:08.141415+0000","last_clean":"2024-04-25T05:48:08.141415+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:48:08.141415+0000","last_undegraded":"2024-04-25T05:48:08.141415+0000","last_fullsized":"2024-04-25T05:48:08.141415+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:37.317498+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.317498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":369,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.138446+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:48:08.138446+0000","last_peered":"2024-04-25T05:48:08.138446+0000","last_clean":"2024-04-25T05:48:08.138446+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:48:08.138446+0000","last_undegraded":"2024-04-25T05:48:08.138446+0000","last_fullsized":"2024-04-25T05:48:08.138446+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:24.171124+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":936,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156222+0000","last_change":"2024-04-25T05:47:18.482166+0000","last_active":"2024-04-25T05:48:13.156222+0000","last_peered":"2024-04-25T05:48:13.156222+0000","last_clean":"2024-04-25T05:48:13.156222+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:48:13.156222+0000","last_undegraded":"2024-04-25T05:48:13.156222+0000","last_fullsized":"2024-04-25T05:48:13.156222+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:47:18.482041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.482041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.135532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":108,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.160124+0000","last_change":"2024-04-25T05:48:01.987948+0000","last_active":"2024-04-25T05:48:13.160124+0000","last_peered":"2024-04-25T05:48:13.160124+0000","last_clean":"2024-04-25T05:48:13.160124+0000","last_became_active":"2024-04-25T05:46:34.971901+0000","last_became_peered":"2024-04-25T05:46:34.971901+0000","last_unstale":"2024-04-25T05:48:13.160124+0000","last_undegraded":"2024-04-25T05:48:13.160124+0000","last_fullsized":"2024-04-25T05:48:13.160124+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.987808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.987808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:21.635305+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":802,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155718+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:48:13.155718+0000","last_peered":"2024-04-25T05:48:13.155718+0000","last_clean":"2024-04-25T05:48:13.155718+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:48:13.155718+0000","last_undegraded":"2024-04-25T05:48:13.155718+0000","last_fullsized":"2024-04-25T05:48:13.155718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:10.540049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":770,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139850+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:48:08.139850+0000","last_peered":"2024-04-25T05:48:08.139850+0000","last_clean":"2024-04-25T05:48:08.139850+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:48:08.139850+0000","last_undegraded":"2024-04-25T05:48:08.139850+0000","last_fullsized":"2024-04-25T05:48:08.139850+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.690045+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":99,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141665+0000","last_change":"2024-04-25T05:46:34.997195+0000","last_active":"2024-04-25T05:48:08.141665+0000","last_peered":"2024-04-25T05:48:08.141665+0000","last_clean":"2024-04-25T05:48:08.141665+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:48:08.141665+0000","last_undegraded":"2024-04-25T05:48:08.141665+0000","last_fullsized":"2024-04-25T05:48:08.141665+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:46:19.516621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:19.516621+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":369,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139094+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:48:08.139094+0000","last_peered":"2024-04-25T05:48:08.139094+0000","last_clean":"2024-04-25T05:48:08.139094+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:48:08.139094+0000","last_undegraded":"2024-04-25T05:48:08.139094+0000","last_fullsized":"2024-04-25T05:48:08.139094+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.718367+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":896,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.132979+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:48:08.132979+0000","last_peered":"2024-04-25T05:48:08.132979+0000","last_clean":"2024-04-25T05:48:08.132979+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:48:08.132979+0000","last_undegraded":"2024-04-25T05:48:08.132979+0000","last_fullsized":"2024-04-25T05:48:08.132979+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.466214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":429,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.138882+0000","last_change":"2024-04-25T05:45:33.294954+0000","last_active":"2024-04-25T05:48:08.138882+0000","last_peered":"2024-04-25T05:48:08.138882+0000","last_clean":"2024-04-25T05:48:08.138882+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:48:08.138882+0000","last_undegraded":"2024-04-25T05:48:08.138882+0000","last_fullsized":"2024-04-25T05:48:08.138882+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151660+0000","last_change":"2024-04-25T05:47:30.502986+0000","last_active":"2024-04-25T05:48:13.151660+0000","last_peered":"2024-04-25T05:48:13.151660+0000","last_clean":"2024-04-25T05:48:13.151660+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:48:13.151660+0000","last_undegraded":"2024-04-25T05:48:13.151660+0000","last_fullsized":"2024-04-25T05:48:13.151660+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:30.502873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.502873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.898108+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":776,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155924+0000","last_change":"2024-04-25T05:46:53.519364+0000","last_active":"2024-04-25T05:48:13.155924+0000","last_peered":"2024-04-25T05:48:13.155924+0000","last_clean":"2024-04-25T05:48:13.155924+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:48:13.155924+0000","last_undegraded":"2024-04-25T05:48:13.155924+0000","last_fullsized":"2024-04-25T05:48:13.155924+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.519223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.519223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":306,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.152103+0000","last_change":"2024-04-25T05:46:20.039638+0000","last_active":"2024-04-25T05:48:13.152103+0000","last_peered":"2024-04-25T05:48:13.152103+0000","last_clean":"2024-04-25T05:48:13.152103+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:48:13.152103+0000","last_undegraded":"2024-04-25T05:48:13.152103+0000","last_fullsized":"2024-04-25T05:48:13.152103+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:46:07.665439+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.665439+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594585+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":757,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139194+0000","last_change":"2024-04-25T05:46:44.682768+0000","last_active":"2024-04-25T05:48:08.139194+0000","last_peered":"2024-04-25T05:48:08.139194+0000","last_clean":"2024-04-25T05:48:08.139194+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:48:08.139194+0000","last_undegraded":"2024-04-25T05:48:08.139194+0000","last_fullsized":"2024-04-25T05:48:08.139194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.682645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.682645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:10.110610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":760,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157522+0000","last_change":"2024-04-25T05:47:25.459410+0000","last_active":"2024-04-25T05:48:13.157522+0000","last_peered":"2024-04-25T05:48:13.157522+0000","last_clean":"2024-04-25T05:48:13.157522+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:48:13.157522+0000","last_undegraded":"2024-04-25T05:48:13.157522+0000","last_fullsized":"2024-04-25T05:48:13.157522+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:25.459222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.459222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.040578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144507+0000","last_change":"2024-04-25T05:46:53.839614+0000","last_active":"2024-04-25T05:48:08.144507+0000","last_peered":"2024-04-25T05:48:08.144507+0000","last_clean":"2024-04-25T05:48:08.144507+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:48:08.144507+0000","last_undegraded":"2024-04-25T05:48:08.144507+0000","last_fullsized":"2024-04-25T05:48:08.144507+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.839498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.839498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:11.237992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":837,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156253+0000","last_change":"2024-04-25T05:47:30.478629+0000","last_active":"2024-04-25T05:48:13.156253+0000","last_peered":"2024-04-25T05:48:13.156253+0000","last_clean":"2024-04-25T05:48:13.156253+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:48:13.156253+0000","last_undegraded":"2024-04-25T05:48:13.156253+0000","last_fullsized":"2024-04-25T05:48:13.156253+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.478462+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:30.722214+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.145490+0000","last_change":"2024-04-25T05:47:26.965285+0000","last_active":"2024-04-25T05:48:08.145490+0000","last_peered":"2024-04-25T05:48:08.145490+0000","last_clean":"2024-04-25T05:48:08.145490+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:48:08.145490+0000","last_undegraded":"2024-04-25T05:48:08.145490+0000","last_fullsized":"2024-04-25T05:48:08.145490+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:26.965135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:26.965135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:27.629847+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":768,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.146110+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:48:08.146110+0000","last_peered":"2024-04-25T05:48:08.146110+0000","last_clean":"2024-04-25T05:48:08.146110+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:48:08.146110+0000","last_undegraded":"2024-04-25T05:48:08.146110+0000","last_fullsized":"2024-04-25T05:48:08.146110+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.640824+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":768,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157808+0000","last_change":"2024-04-25T05:46:19.986880+0000","last_active":"2024-04-25T05:48:13.157808+0000","last_peered":"2024-04-25T05:48:13.157808+0000","last_clean":"2024-04-25T05:48:13.157808+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:48:13.157808+0000","last_undegraded":"2024-04-25T05:48:13.157808+0000","last_fullsized":"2024-04-25T05:48:13.157808+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:17.841621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:46:17.841621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228746+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":776,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154424+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:48:13.154424+0000","last_peered":"2024-04-25T05:48:13.154424+0000","last_clean":"2024-04-25T05:48:13.154424+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:48:13.154424+0000","last_undegraded":"2024-04-25T05:48:13.154424+0000","last_fullsized":"2024-04-25T05:48:13.154424+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228497+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.144130+0000","last_change":"2024-04-25T05:47:32.942934+0000","last_active":"2024-04-25T05:48:08.144130+0000","last_peered":"2024-04-25T05:48:08.144130+0000","last_clean":"2024-04-25T05:48:08.144130+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:48:08.144130+0000","last_undegraded":"2024-04-25T05:48:08.144130+0000","last_fullsized":"2024-04-25T05:48:08.144130+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:32.942795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.942795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.052701+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":475,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.148389+0000","last_change":"2024-04-25T05:47:28.867384+0000","last_active":"2024-04-25T05:48:13.148389+0000","last_peered":"2024-04-25T05:48:13.148389+0000","last_clean":"2024-04-25T05:48:13.148389+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:48:13.148389+0000","last_undegraded":"2024-04-25T05:48:13.148389+0000","last_fullsized":"2024-04-25T05:48:13.148389+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.867273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.867273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.337979+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":759,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.138787+0000","last_change":"2024-04-25T05:46:20.027611+0000","last_active":"2024-04-25T05:48:08.138787+0000","last_peered":"2024-04-25T05:48:08.138787+0000","last_clean":"2024-04-25T05:48:08.138787+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:48:08.138787+0000","last_undegraded":"2024-04-25T05:48:08.138787+0000","last_fullsized":"2024-04-25T05:48:08.138787+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":588,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151668+0000","last_change":"2024-04-25T05:46:44.699964+0000","last_active":"2024-04-25T05:48:13.151668+0000","last_peered":"2024-04-25T05:48:13.151668+0000","last_clean":"2024-04-25T05:48:13.151668+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:48:13.151668+0000","last_undegraded":"2024-04-25T05:48:13.151668+0000","last_fullsized":"2024-04-25T05:48:13.151668+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:46:44.699808+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.699808+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097651+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":775,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.154308+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:48:13.154308+0000","last_peered":"2024-04-25T05:48:13.154308+0000","last_clean":"2024-04-25T05:48:13.154308+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:48:13.154308+0000","last_undegraded":"2024-04-25T05:48:13.154308+0000","last_fullsized":"2024-04-25T05:48:13.154308+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.724783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":367,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139349+0000","last_change":"2024-04-25T05:46:19.988527+0000","last_active":"2024-04-25T05:48:08.139349+0000","last_peered":"2024-04-25T05:48:08.139349+0000","last_clean":"2024-04-25T05:48:08.139349+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:48:08.139349+0000","last_undegraded":"2024-04-25T05:48:08.139349+0000","last_fullsized":"2024-04-25T05:48:08.139349+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":757,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150185+0000","last_change":"2024-04-25T05:47:59.671965+0000","last_active":"2024-04-25T05:48:13.150185+0000","last_peered":"2024-04-25T05:48:13.150185+0000","last_clean":"2024-04-25T05:48:13.150185+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:48:13.150185+0000","last_undegraded":"2024-04-25T05:48:13.150185+0000","last_fullsized":"2024-04-25T05:48:13.150185+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.671833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.671833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:07.546615+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":442,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139565+0000","last_change":"2024-04-25T05:47:21.817524+0000","last_active":"2024-04-25T05:48:08.139565+0000","last_peered":"2024-04-25T05:48:08.139565+0000","last_clean":"2024-04-25T05:48:08.139565+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:48:08.139565+0000","last_undegraded":"2024-04-25T05:48:08.139565+0000","last_fullsized":"2024-04-25T05:48:08.139565+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:47:21.817370+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:47:21.817370+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.100421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":752,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150921+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:48:13.150921+0000","last_peered":"2024-04-25T05:48:13.150921+0000","last_clean":"2024-04-25T05:48:13.150921+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:48:13.150921+0000","last_undegraded":"2024-04-25T05:48:13.150921+0000","last_fullsized":"2024-04-25T05:48:13.150921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594454+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":592,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151443+0000","last_change":"2024-04-25T05:47:56.016055+0000","last_active":"2024-04-25T05:48:13.151443+0000","last_peered":"2024-04-25T05:48:13.151443+0000","last_clean":"2024-04-25T05:48:13.151443+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:48:13.151443+0000","last_undegraded":"2024-04-25T05:48:13.151443+0000","last_fullsized":"2024-04-25T05:48:13.151443+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:47:56.015913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.015913+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.422541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":786,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.139068+0000","last_change":"2024-04-25T05:46:34.991626+0000","last_active":"2024-04-25T05:48:08.139068+0000","last_peered":"2024-04-25T05:48:08.139068+0000","last_clean":"2024-04-25T05:48:08.139068+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:48:08.139068+0000","last_undegraded":"2024-04-25T05:48:08.139068+0000","last_fullsized":"2024-04-25T05:48:08.139068+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:46:09.493093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.493093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":783,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.159159+0000","last_change":"2024-04-25T05:47:16.525921+0000","last_active":"2024-04-25T05:48:13.159159+0000","last_peered":"2024-04-25T05:48:13.159159+0000","last_clean":"2024-04-25T05:48:13.159159+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:48:13.159159+0000","last_undegraded":"2024-04-25T05:48:13.159159+0000","last_fullsized":"2024-04-25T05:48:13.159159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:16.525560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:16.525560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.665516+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":791,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.135499+0000","last_change":"2024-04-25T05:48:00.830523+0000","last_active":"2024-04-25T05:48:08.135499+0000","last_peered":"2024-04-25T05:48:08.135499+0000","last_clean":"2024-04-25T05:48:08.135499+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:48:08.135499+0000","last_undegraded":"2024-04-25T05:48:08.135499+0000","last_fullsized":"2024-04-25T05:48:08.135499+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.830408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.175161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":435,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155886+0000","last_change":"2024-04-25T05:46:47.610533+0000","last_active":"2024-04-25T05:48:13.155886+0000","last_peered":"2024-04-25T05:48:13.155886+0000","last_clean":"2024-04-25T05:48:13.155886+0000","last_became_active":"2024-04-25T05:46:20.009022+0000","last_became_peered":"2024-04-25T05:46:20.009022+0000","last_unstale":"2024-04-25T05:48:13.155886+0000","last_undegraded":"2024-04-25T05:48:13.155886+0000","last_fullsized":"2024-04-25T05:48:13.155886+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.610387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.610387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228803+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":756,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.166645+0000","last_change":"2024-04-25T05:46:49.931810+0000","last_active":"2024-04-25T05:48:13.166645+0000","last_peered":"2024-04-25T05:48:13.166645+0000","last_clean":"2024-04-25T05:48:13.166645+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:48:13.166645+0000","last_undegraded":"2024-04-25T05:48:13.166645+0000","last_fullsized":"2024-04-25T05:48:13.166645+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:49.930039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.930039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228829+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":118,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156870+0000","last_change":"2024-04-25T05:47:25.921271+0000","last_active":"2024-04-25T05:48:13.156870+0000","last_peered":"2024-04-25T05:48:13.156870+0000","last_clean":"2024-04-25T05:48:13.156870+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:48:13.156870+0000","last_undegraded":"2024-04-25T05:48:13.156870+0000","last_fullsized":"2024-04-25T05:48:13.156870+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:47:25.921121+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.921121+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.176425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":245,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.140050+0000","last_change":"2024-04-25T05:48:05.703358+0000","last_active":"2024-04-25T05:48:08.140050+0000","last_peered":"2024-04-25T05:48:08.140050+0000","last_clean":"2024-04-25T05:48:08.140050+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:48:08.140050+0000","last_undegraded":"2024-04-25T05:48:08.140050+0000","last_fullsized":"2024-04-25T05:48:08.140050+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:11.734140+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":767,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150400+0000","last_change":"2024-04-25T05:46:34.974900+0000","last_active":"2024-04-25T05:48:13.150400+0000","last_peered":"2024-04-25T05:48:13.150400+0000","last_clean":"2024-04-25T05:48:13.150400+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:48:13.150400+0000","last_undegraded":"2024-04-25T05:48:13.150400+0000","last_fullsized":"2024-04-25T05:48:13.150400+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594368+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":787,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.135860+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:48:08.135860+0000","last_peered":"2024-04-25T05:48:08.135860+0000","last_clean":"2024-04-25T05:48:08.135860+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:48:08.135860+0000","last_undegraded":"2024-04-25T05:48:08.135860+0000","last_fullsized":"2024-04-25T05:48:08.135860+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.067934+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":494,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157319+0000","last_change":"2024-04-25T05:46:05.752045+0000","last_active":"2024-04-25T05:48:13.157319+0000","last_peered":"2024-04-25T05:48:13.157319+0000","last_clean":"2024-04-25T05:48:13.157319+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:48:13.157319+0000","last_undegraded":"2024-04-25T05:48:13.157319+0000","last_fullsized":"2024-04-25T05:48:13.157319+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:46:05.751904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:05.751904+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141593+0000","last_change":"2024-04-25T05:48:04.097751+0000","last_active":"2024-04-25T05:48:08.141593+0000","last_peered":"2024-04-25T05:48:08.141593+0000","last_clean":"2024-04-25T05:48:08.141593+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:48:08.141593+0000","last_undegraded":"2024-04-25T05:48:08.141593+0000","last_fullsized":"2024-04-25T05:48:08.141593+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:04.097613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.097613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:13.875974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":747,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.138606+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:48:08.138606+0000","last_peered":"2024-04-25T05:48:08.138606+0000","last_clean":"2024-04-25T05:48:08.138606+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:48:08.138606+0000","last_undegraded":"2024-04-25T05:48:08.138606+0000","last_fullsized":"2024-04-25T05:48:08.138606+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.146317+0000","last_change":"2024-04-25T05:46:34.985777+0000","last_active":"2024-04-25T05:48:08.146317+0000","last_peered":"2024-04-25T05:48:08.146317+0000","last_clean":"2024-04-25T05:48:08.146317+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:48:08.146317+0000","last_undegraded":"2024-04-25T05:48:08.146317+0000","last_fullsized":"2024-04-25T05:48:08.146317+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:38.767074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.767074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":551,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155696+0000","last_change":"2024-04-25T05:46:20.046230+0000","last_active":"2024-04-25T05:48:13.155696+0000","last_peered":"2024-04-25T05:48:13.155696+0000","last_clean":"2024-04-25T05:48:13.155696+0000","last_became_active":"2024-04-25T05:46:20.009817+0000","last_became_peered":"2024-04-25T05:46:20.009817+0000","last_unstale":"2024-04-25T05:48:13.155696+0000","last_undegraded":"2024-04-25T05:48:13.155696+0000","last_fullsized":"2024-04-25T05:48:13.155696+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'28","last_scrub_stamp":"2024-04-25T05:46:10.858811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.858811+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141523+0000","last_change":"2024-04-25T05:48:00.061365+0000","last_active":"2024-04-25T05:48:08.141523+0000","last_peered":"2024-04-25T05:48:08.141523+0000","last_clean":"2024-04-25T05:48:08.141523+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:48:08.141523+0000","last_undegraded":"2024-04-25T05:48:08.141523+0000","last_fullsized":"2024-04-25T05:48:08.141523+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.061240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.061240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:07.003582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.142552+0000","last_change":"2024-04-25T05:46:34.979661+0000","last_active":"2024-04-25T05:48:08.142552+0000","last_peered":"2024-04-25T05:48:08.142552+0000","last_clean":"2024-04-25T05:48:08.142552+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:48:08.142552+0000","last_undegraded":"2024-04-25T05:48:08.142552+0000","last_fullsized":"2024-04-25T05:48:08.142552+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":377,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.150597+0000","last_change":"2024-04-25T05:47:13.375334+0000","last_active":"2024-04-25T05:48:13.150597+0000","last_peered":"2024-04-25T05:48:13.150597+0000","last_clean":"2024-04-25T05:48:13.150597+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:48:13.150597+0000","last_undegraded":"2024-04-25T05:48:13.150597+0000","last_fullsized":"2024-04-25T05:48:13.150597+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:13.375091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:13.375091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:19.270626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1040,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.156355+0000","last_change":"2024-04-25T05:46:48.610335+0000","last_active":"2024-04-25T05:48:13.156355+0000","last_peered":"2024-04-25T05:48:13.156355+0000","last_clean":"2024-04-25T05:48:13.156355+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:48:13.156355+0000","last_undegraded":"2024-04-25T05:48:13.156355+0000","last_fullsized":"2024-04-25T05:48:13.156355+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:46:48.610157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.610157+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":777,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.155262+0000","last_change":"2024-04-25T05:47:57.543231+0000","last_active":"2024-04-25T05:48:13.155262+0000","last_peered":"2024-04-25T05:48:13.155262+0000","last_clean":"2024-04-25T05:48:13.155262+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:48:13.155262+0000","last_undegraded":"2024-04-25T05:48:13.155262+0000","last_fullsized":"2024-04-25T05:48:13.155262+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:57.543118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:47:57.543118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:58.141942+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":727,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.149306+0000","last_change":"2024-04-25T05:47:18.658184+0000","last_active":"2024-04-25T05:48:13.149306+0000","last_peered":"2024-04-25T05:48:13.149306+0000","last_clean":"2024-04-25T05:48:13.149306+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:48:13.149306+0000","last_undegraded":"2024-04-25T05:48:13.149306+0000","last_fullsized":"2024-04-25T05:48:13.149306+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:18.657935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.657935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:31.709365+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":743,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.157613+0000","last_change":"2024-04-25T05:46:46.830806+0000","last_active":"2024-04-25T05:48:13.157613+0000","last_peered":"2024-04-25T05:48:13.157613+0000","last_clean":"2024-04-25T05:48:13.157613+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:48:13.157613+0000","last_undegraded":"2024-04-25T05:48:13.157613+0000","last_fullsized":"2024-04-25T05:48:13.157613+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.830676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.830676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228934+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":847,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.134477+0000","last_change":"2024-04-25T05:47:55.921938+0000","last_active":"2024-04-25T05:48:08.134477+0000","last_peered":"2024-04-25T05:48:08.134477+0000","last_clean":"2024-04-25T05:48:08.134477+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:48:08.134477+0000","last_undegraded":"2024-04-25T05:48:08.134477+0000","last_fullsized":"2024-04-25T05:48:08.134477+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:55.921801+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.921801+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:03.833450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.143982+0000","last_change":"2024-04-25T05:46:44.971898+0000","last_active":"2024-04-25T05:48:08.143982+0000","last_peered":"2024-04-25T05:48:08.143982+0000","last_clean":"2024-04-25T05:48:08.143982+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:48:08.143982+0000","last_undegraded":"2024-04-25T05:48:08.143982+0000","last_fullsized":"2024-04-25T05:48:08.143982+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.971755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.971755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":761,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-04-25T05:48:13.151088+0000","last_change":"2024-04-25T05:46:34.974443+0000","last_active":"2024-04-25T05:48:13.151088+0000","last_peered":"2024-04-25T05:48:13.151088+0000","last_clean":"2024-04-25T05:48:13.151088+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:48:13.151088+0000","last_undegraded":"2024-04-25T05:48:13.151088+0000","last_fullsized":"2024-04-25T05:48:13.151088+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:13.594394+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":76,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-04-25T05:48:08.141438+0000","last_change":"2024-04-25T05:46:34.988586+0000","last_active":"2024-04-25T05:48:08.141438+0000","last_peered":"2024-04-25T05:48:08.141438+0000","last_clean":"2024-04-25T05:48:08.141438+0000","last_became_active":"2024-04-25T05:46:34.986413+0000","last_became_peered":"2024-04-25T05:46:34.986413+0000","last_unstale":"2024-04-25T05:48:08.141438+0000","last_undegraded":"2024-04-25T05:48:08.141438+0000","last_fullsized":"2024-04-25T05:48:08.141438+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:39.734373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.734373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8305049624,"num_objects":126726,"num_object_clones":0,"num_object_copies":380178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129970,"num_write_kb":8214209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24888619008,"data_stored":24888606792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76370,"ondisk_log_size":76370,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":836,"num_read_kb":831,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1394,"num_read_kb":1119,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":10814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":964,"ondisk_log_size":964,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070640,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3251344,"kb_used_data":2552328,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":90493808,"statfs":{"total":95995035648,"available":92665659392,"internally_reserved":0,"allocated":2613583872,"data_stored":2612972946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364980,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4164072,"kb_used_data":3559520,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":89581080,"statfs":{"total":95995035648,"available":91731025920,"internally_reserved":0,"allocated":3644948480,"data_stored":3644376954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3898348,"kb_used_data":3046508,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":89846804,"statfs":{"total":95995035648,"available":92003127296,"internally_reserved":0,"allocated":3119624192,"data_stored":3119070425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738579,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411528,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4071996,"kb_used_data":3553396,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":89673156,"statfs":{"total":95995035648,"available":91825311744,"internally_reserved":0,"allocated":3638677504,"data_stored":3638085520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":530905541},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149890,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4988448,"kb_used_data":4036576,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":88756704,"statfs":{"total":95995035648,"available":90886864896,"internally_reserved":0,"allocated":4133453824,"data_stored":4132912646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":974453191},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986365,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5204676,"kb_used_data":4605500,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":88540476,"statfs":{"total":95995035648,"available":90665447424,"internally_reserved":0,"allocated":4716032000,"data_stored":4715431984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692025,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3509680,"kb_used_data":3073128,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90235472,"statfs":{"total":95995035648,"available":92401123328,"internally_reserved":0,"allocated":3146883072,"data_stored":3146303460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3106144256,"data_stored":3106144256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3622371328,"data_stored":3622371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4100329472,"data_stored":4100325400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4701818880,"data_stored":4701814808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3114598400,"data_stored":3114598400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3641315328,"data_stored":3641311256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2602041344,"data_stored":2602041344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:48:17.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:48:17.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:17.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:17.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.662 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:17.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:17.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:17.800 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:17.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:17.934 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:17.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:18.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:18.070 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:18.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:18.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:18.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:18.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:18.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:18.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:18.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:18.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:18.721 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:18.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:18.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:18.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:18.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:18.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:19.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:19.164 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:19.201 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:48:19.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[27010]: pgmap v758: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 742 op/s 2024-04-25T05:48:19.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:19.206 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:19.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[37492]: pgmap v758: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 742 op/s 2024-04-25T05:48:19.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:19.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:19.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:19.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:19.394 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:19 smithi060 ceph-mon[30898]: pgmap v758: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 742 op/s 2024-04-25T05:48:19.394 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:19.394 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:19.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:19.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:19.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:19.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:19.666 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:48:19.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:19.677 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:48:19.706 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:19.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:19.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:19.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:19.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:48:19.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:19.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:20.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:20.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:20.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:20.238 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:48:20.238 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:48:20.238 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:48:20.238 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:48:20.238 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:48:20.247 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:20.349 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:20.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4131695506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[27010]: osdmap e421: 8 total, 8 up, 7 in 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4131695506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:20.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:20 smithi043 ceph-mon[37492]: osdmap e421: 8 total, 8 up, 7 in 2024-04-25T05:48:20.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4131695506' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:20 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:20 smithi060 ceph-mon[30898]: osdmap e421: 8 total, 8 up, 7 in 2024-04-25T05:48:20.516 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:20.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:20.649 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:20.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:20.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:20.879 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:20.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:20.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:21.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:21.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:21.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:21.172 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:21.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:21.276 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2024-04-25T05:48:19.180450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7916821043,"num_objects":121023,"num_object_clones":0,"num_object_copies":363069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121023,"num_whiteouts":0,"num_read":2253,"num_read_kb":1973,"num_write":136817,"num_write_kb":8214254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":1196524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78490,"ondisk_log_size":78490,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":27920164,"kb_used_data":23259180,"kb_used_omap":10,"kb_used_meta":4660725,"kb_avail":628295900,"statfs":{"total":671965249536,"available":643375001600,"internally_reserved":0,"allocated":23817400320,"data_stored":23814222360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4772582507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,4,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-212598760,"num_objects":-3243,"num_object_clones":0,"num_object_copies":-9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3243,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":11867,"num_write_kb":275905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.220149"},"pg_stats":[{"pgid":"10.2","version":"420'8496","reported_seq":8570,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.947407+0000","last_change":"2024-04-25T05:46:37.979403+0000","last_active":"2024-04-25T05:48:18.947407+0000","last_peered":"2024-04-25T05:48:18.947407+0000","last_clean":"2024-04-25T05:48:18.947407+0000","last_became_active":"2024-04-25T05:46:37.979097+0000","last_became_peered":"2024-04-25T05:46:37.979097+0000","last_unstale":"2024-04-25T05:48:18.947407+0000","last_undegraded":"2024-04-25T05:48:18.947407+0000","last_fullsized":"2024-04-25T05:48:18.947407+0000","mapping_epoch":387,"log_start":"403'4300","ondisk_log_start":"403'4300","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4196,"log_dups_size":0,"ondisk_log_size":4196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:20.560130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483262464,"num_objects":7374,"num_object_clones":0,"num_object_copies":22122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8496,"num_write_kb":507840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":473,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189065+0000","last_change":"2024-04-25T05:46:09.856702+0000","last_active":"2024-04-25T05:48:14.189065+0000","last_peered":"2024-04-25T05:48:14.189065+0000","last_clean":"2024-04-25T05:48:14.189065+0000","last_became_active":"2024-04-25T05:40:09.730973+0000","last_became_peered":"2024-04-25T05:40:09.730973+0000","last_unstale":"2024-04-25T05:48:14.189065+0000","last_undegraded":"2024-04-25T05:48:14.189065+0000","last_fullsized":"2024-04-25T05:48:14.189065+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:46:09.856439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.856439+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228693+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":764,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190416+0000","last_change":"2024-04-25T05:46:03.753399+0000","last_active":"2024-04-25T05:48:14.190416+0000","last_peered":"2024-04-25T05:48:14.190416+0000","last_clean":"2024-04-25T05:48:14.190416+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:48:14.190416+0000","last_undegraded":"2024-04-25T05:48:14.190416+0000","last_fullsized":"2024-04-25T05:48:14.190416+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:03.753274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:03.753274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":133,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179968+0000","last_change":"2024-04-25T05:46:35.013019+0000","last_active":"2024-04-25T05:48:14.179968+0000","last_peered":"2024-04-25T05:48:14.179968+0000","last_clean":"2024-04-25T05:48:14.179968+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:48:14.179968+0000","last_undegraded":"2024-04-25T05:48:14.179968+0000","last_fullsized":"2024-04-25T05:48:14.179968+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:45:36.284279+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.284279+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":442,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.197382+0000","last_change":"2024-04-25T05:47:28.424745+0000","last_active":"2024-04-25T05:48:14.197382+0000","last_peered":"2024-04-25T05:48:14.197382+0000","last_clean":"2024-04-25T05:48:14.197382+0000","last_became_active":"2024-04-25T05:46:19.990605+0000","last_became_peered":"2024-04-25T05:46:19.990605+0000","last_unstale":"2024-04-25T05:48:14.197382+0000","last_undegraded":"2024-04-25T05:48:14.197382+0000","last_fullsized":"2024-04-25T05:48:14.197382+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.424618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.424618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:55.358144+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1079,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183526+0000","last_change":"2024-04-25T05:46:23.149014+0000","last_active":"2024-04-25T05:48:14.183526+0000","last_peered":"2024-04-25T05:48:14.183526+0000","last_clean":"2024-04-25T05:48:14.183526+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:48:14.183526+0000","last_undegraded":"2024-04-25T05:48:14.183526+0000","last_fullsized":"2024-04-25T05:48:14.183526+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"420'8358","reported_seq":8432,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.186473+0000","last_change":"2024-04-25T05:46:37.983434+0000","last_active":"2024-04-25T05:48:18.186473+0000","last_peered":"2024-04-25T05:48:18.186473+0000","last_clean":"2024-04-25T05:48:18.186473+0000","last_became_active":"2024-04-25T05:46:37.983078+0000","last_became_peered":"2024-04-25T05:46:37.983078+0000","last_unstale":"2024-04-25T05:48:18.186473+0000","last_undegraded":"2024-04-25T05:48:18.186473+0000","last_fullsized":"2024-04-25T05:48:18.186473+0000","mapping_epoch":387,"log_start":"401'3500","ondisk_log_start":"401'3500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485883904,"num_objects":7414,"num_object_clones":0,"num_object_copies":22242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":146,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180076+0000","last_change":"2024-04-25T05:46:48.976027+0000","last_active":"2024-04-25T05:48:14.180076+0000","last_peered":"2024-04-25T05:48:14.180076+0000","last_clean":"2024-04-25T05:48:14.180076+0000","last_became_active":"2024-04-25T05:46:36.984563+0000","last_became_peered":"2024-04-25T05:46:36.984563+0000","last_unstale":"2024-04-25T05:48:14.180076+0000","last_undegraded":"2024-04-25T05:48:14.180076+0000","last_fullsized":"2024-04-25T05:48:14.180076+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:46:48.975826+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.975826+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.768088+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":478,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184483+0000","last_change":"2024-04-25T05:46:42.728489+0000","last_active":"2024-04-25T05:48:14.184483+0000","last_peered":"2024-04-25T05:48:14.184483+0000","last_clean":"2024-04-25T05:48:14.184483+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:48:14.184483+0000","last_undegraded":"2024-04-25T05:48:14.184483+0000","last_fullsized":"2024-04-25T05:48:14.184483+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:42.728367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:46:42.728367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097616+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195808+0000","last_change":"2024-04-25T05:45:33.765309+0000","last_active":"2024-04-25T05:48:14.195808+0000","last_peered":"2024-04-25T05:48:14.195808+0000","last_clean":"2024-04-25T05:48:14.195808+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:48:14.195808+0000","last_undegraded":"2024-04-25T05:48:14.195808+0000","last_fullsized":"2024-04-25T05:48:14.195808+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:33.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":104,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179632+0000","last_change":"2024-04-25T05:48:03.971319+0000","last_active":"2024-04-25T05:48:14.179632+0000","last_peered":"2024-04-25T05:48:14.179632+0000","last_clean":"2024-04-25T05:48:14.179632+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:48:14.179632+0000","last_undegraded":"2024-04-25T05:48:14.179632+0000","last_fullsized":"2024-04-25T05:48:14.179632+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.971192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.971192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.239829+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":677,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183950+0000","last_change":"2024-04-25T05:47:25.988415+0000","last_active":"2024-04-25T05:48:14.183950+0000","last_peered":"2024-04-25T05:48:14.183950+0000","last_clean":"2024-04-25T05:48:14.183950+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:48:14.183950+0000","last_undegraded":"2024-04-25T05:48:14.183950+0000","last_fullsized":"2024-04-25T05:48:14.183950+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.988268+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:28.271995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"420'8532","reported_seq":8668,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.157850+0000","last_change":"2024-04-25T05:48:08.191377+0000","last_active":"2024-04-25T05:48:18.157850+0000","last_peered":"2024-04-25T05:48:18.157850+0000","last_clean":"2024-04-25T05:48:18.157850+0000","last_became_active":"2024-04-25T05:46:37.984393+0000","last_became_peered":"2024-04-25T05:46:37.984393+0000","last_unstale":"2024-04-25T05:48:18.157850+0000","last_undegraded":"2024-04-25T05:48:18.157850+0000","last_fullsized":"2024-04-25T05:48:18.157850+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":5549,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.110277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499122176,"num_objects":7616,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.186423+0000","last_change":"2024-04-25T05:47:24.775449+0000","last_active":"2024-04-25T05:48:14.186423+0000","last_peered":"2024-04-25T05:48:14.186423+0000","last_clean":"2024-04-25T05:48:14.186423+0000","last_became_active":"2024-04-25T05:40:09.729621+0000","last_became_peered":"2024-04-25T05:40:09.729621+0000","last_unstale":"2024-04-25T05:48:14.186423+0000","last_undegraded":"2024-04-25T05:48:14.186423+0000","last_fullsized":"2024-04-25T05:48:14.186423+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:47:24.775084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.775084+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.201360+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.175514+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:48:14.175514+0000","last_peered":"2024-04-25T05:48:14.175514+0000","last_clean":"2024-04-25T05:48:14.175514+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:48:14.175514+0000","last_undegraded":"2024-04-25T05:48:14.175514+0000","last_fullsized":"2024-04-25T05:48:14.175514+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.578308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183146+0000","last_change":"2024-04-25T05:46:34.982029+0000","last_active":"2024-04-25T05:48:14.183146+0000","last_peered":"2024-04-25T05:48:14.183146+0000","last_clean":"2024-04-25T05:48:14.183146+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:48:14.183146+0000","last_undegraded":"2024-04-25T05:48:14.183146+0000","last_fullsized":"2024-04-25T05:48:14.183146+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:18.666459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:46:18.666459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":758,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188636+0000","last_change":"2024-04-25T05:46:51.568352+0000","last_active":"2024-04-25T05:48:14.188636+0000","last_peered":"2024-04-25T05:48:14.188636+0000","last_clean":"2024-04-25T05:48:14.188636+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:48:14.188636+0000","last_undegraded":"2024-04-25T05:48:14.188636+0000","last_fullsized":"2024-04-25T05:48:14.188636+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:51.568255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:51.568255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228961+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"420'8600","reported_seq":8676,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.053001+0000","last_change":"2024-04-25T05:46:37.983353+0000","last_active":"2024-04-25T05:48:18.053001+0000","last_peered":"2024-04-25T05:48:18.053001+0000","last_clean":"2024-04-25T05:48:18.053001+0000","last_became_active":"2024-04-25T05:46:37.983037+0000","last_became_peered":"2024-04-25T05:46:37.983037+0000","last_unstale":"2024-04-25T05:48:18.053001+0000","last_undegraded":"2024-04-25T05:48:18.053001+0000","last_fullsized":"2024-04-25T05:48:18.053001+0000","mapping_epoch":387,"log_start":"403'4000","ondisk_log_start":"403'4000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:22.884988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502398976,"num_objects":7666,"num_object_clones":0,"num_object_copies":22998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":800,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196954+0000","last_change":"2024-04-25T05:47:18.459728+0000","last_active":"2024-04-25T05:48:14.196954+0000","last_peered":"2024-04-25T05:48:14.196954+0000","last_clean":"2024-04-25T05:48:14.196954+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:48:14.196954+0000","last_undegraded":"2024-04-25T05:48:14.196954+0000","last_fullsized":"2024-04-25T05:48:14.196954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:47:18.459587+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.459587+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:21.437539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187493+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:48:14.187493+0000","last_peered":"2024-04-25T05:48:14.187493+0000","last_clean":"2024-04-25T05:48:14.187493+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:48:14.187493+0000","last_undegraded":"2024-04-25T05:48:14.187493+0000","last_fullsized":"2024-04-25T05:48:14.187493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.434575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"388'15","reported_seq":104,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182297+0000","last_change":"2024-04-25T05:46:35.014060+0000","last_active":"2024-04-25T05:48:14.182297+0000","last_peered":"2024-04-25T05:48:14.182297+0000","last_clean":"2024-04-25T05:48:14.182297+0000","last_became_active":"2024-04-25T05:46:34.989992+0000","last_became_peered":"2024-04-25T05:46:34.989992+0000","last_unstale":"2024-04-25T05:48:14.182297+0000","last_undegraded":"2024-04-25T05:48:14.182297+0000","last_fullsized":"2024-04-25T05:48:14.182297+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'13","last_scrub_stamp":"2024-04-25T05:45:28.975922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:28.975922+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181818+0000","last_change":"2024-04-25T05:46:34.990045+0000","last_active":"2024-04-25T05:48:14.181818+0000","last_peered":"2024-04-25T05:48:14.181818+0000","last_clean":"2024-04-25T05:48:14.181818+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:48:14.181818+0000","last_undegraded":"2024-04-25T05:48:14.181818+0000","last_fullsized":"2024-04-25T05:48:14.181818+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.796754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":641,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180785+0000","last_change":"2024-04-25T05:47:51.226427+0000","last_active":"2024-04-25T05:48:14.180785+0000","last_peered":"2024-04-25T05:48:14.180785+0000","last_clean":"2024-04-25T05:48:14.180785+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:48:14.180785+0000","last_undegraded":"2024-04-25T05:48:14.180785+0000","last_fullsized":"2024-04-25T05:48:14.180785+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:51.226126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.226126+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:11.223819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"420'8522","reported_seq":8678,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.147861+0000","last_change":"2024-04-25T05:47:59.467502+0000","last_active":"2024-04-25T05:48:18.147861+0000","last_peered":"2024-04-25T05:48:18.147861+0000","last_clean":"2024-04-25T05:48:18.147861+0000","last_became_active":"2024-04-25T05:46:37.983600+0000","last_became_peered":"2024-04-25T05:46:37.983600+0000","last_unstale":"2024-04-25T05:48:18.147861+0000","last_undegraded":"2024-04-25T05:48:18.147861+0000","last_fullsized":"2024-04-25T05:48:18.147861+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7199","last_scrub_stamp":"2024-04-25T05:47:59.467288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.467288+0000","objects_scrubbed":7112,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:03.703400+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500957184,"num_objects":7644,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8522,"num_write_kb":517312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.175338+0000","last_change":"2024-04-25T05:47:19.690057+0000","last_active":"2024-04-25T05:48:14.175338+0000","last_peered":"2024-04-25T05:48:14.175338+0000","last_clean":"2024-04-25T05:48:14.175338+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:48:14.175338+0000","last_undegraded":"2024-04-25T05:48:14.175338+0000","last_fullsized":"2024-04-25T05:48:14.175338+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.689810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.689810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:37.489431+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":909,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187709+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:48:14.187709+0000","last_peered":"2024-04-25T05:48:14.187709+0000","last_clean":"2024-04-25T05:48:14.187709+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:48:14.187709+0000","last_undegraded":"2024-04-25T05:48:14.187709+0000","last_fullsized":"2024-04-25T05:48:14.187709+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:00.492808+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":786,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195251+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:48:14.195251+0000","last_peered":"2024-04-25T05:48:14.195251+0000","last_clean":"2024-04-25T05:48:14.195251+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:48:14.195251+0000","last_undegraded":"2024-04-25T05:48:14.195251+0000","last_fullsized":"2024-04-25T05:48:14.195251+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.712277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":436,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188432+0000","last_change":"2024-04-25T05:47:17.513777+0000","last_active":"2024-04-25T05:48:14.188432+0000","last_peered":"2024-04-25T05:48:14.188432+0000","last_clean":"2024-04-25T05:48:14.188432+0000","last_became_active":"2024-04-25T05:46:19.997163+0000","last_became_peered":"2024-04-25T05:46:19.997163+0000","last_unstale":"2024-04-25T05:48:14.188432+0000","last_undegraded":"2024-04-25T05:48:14.188432+0000","last_fullsized":"2024-04-25T05:48:14.188432+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:17.513668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:17.513668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.470303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":991,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.176475+0000","last_change":"2024-04-25T05:47:28.786464+0000","last_active":"2024-04-25T05:48:14.176475+0000","last_peered":"2024-04-25T05:48:14.176475+0000","last_clean":"2024-04-25T05:48:14.176475+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:48:14.176475+0000","last_undegraded":"2024-04-25T05:48:14.176475+0000","last_fullsized":"2024-04-25T05:48:14.176475+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:28.786340+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.786340+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.786586+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"420'8470","reported_seq":8705,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.168994+0000","last_change":"2024-04-25T05:48:08.175544+0000","last_active":"2024-04-25T05:48:18.168994+0000","last_peered":"2024-04-25T05:48:18.168994+0000","last_clean":"2024-04-25T05:48:18.168994+0000","last_became_active":"2024-04-25T05:46:37.982701+0000","last_became_peered":"2024-04-25T05:46:37.982701+0000","last_unstale":"2024-04-25T05:48:18.168994+0000","last_undegraded":"2024-04-25T05:48:18.168994+0000","last_fullsized":"2024-04-25T05:48:18.168994+0000","mapping_epoch":387,"log_start":"401'3500","ondisk_log_start":"401'3500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7494","last_scrub_stamp":"2024-04-25T05:48:05.024302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.024302+0000","objects_scrubbed":7249,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.164610+0000","scrub_duration":2970,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492175360,"num_objects":7510,"num_object_clones":0,"num_object_copies":22530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":930,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187868+0000","last_change":"2024-04-25T05:45:36.207163+0000","last_active":"2024-04-25T05:48:14.187868+0000","last_peered":"2024-04-25T05:48:14.187868+0000","last_clean":"2024-04-25T05:48:14.187868+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:48:14.187868+0000","last_undegraded":"2024-04-25T05:48:14.187868+0000","last_fullsized":"2024-04-25T05:48:14.187868+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T05:45:34.969209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:34.969209+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:21.176016+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":106,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182484+0000","last_change":"2024-04-25T05:48:05.149694+0000","last_active":"2024-04-25T05:48:14.182484+0000","last_peered":"2024-04-25T05:48:14.182484+0000","last_clean":"2024-04-25T05:48:14.182484+0000","last_became_active":"2024-04-25T05:46:34.980031+0000","last_became_peered":"2024-04-25T05:46:34.980031+0000","last_unstale":"2024-04-25T05:48:14.182484+0000","last_undegraded":"2024-04-25T05:48:14.182484+0000","last_fullsized":"2024-04-25T05:48:14.182484+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:48:05.148321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.148321+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:33.058689+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":458,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.197542+0000","last_change":"2024-04-25T05:48:04.608424+0000","last_active":"2024-04-25T05:48:14.197542+0000","last_peered":"2024-04-25T05:48:14.197542+0000","last_clean":"2024-04-25T05:48:14.197542+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:48:14.197542+0000","last_undegraded":"2024-04-25T05:48:14.197542+0000","last_fullsized":"2024-04-25T05:48:14.197542+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.608284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:05.704760+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":136,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196407+0000","last_change":"2024-04-25T05:47:14.571538+0000","last_active":"2024-04-25T05:48:14.196407+0000","last_peered":"2024-04-25T05:48:14.196407+0000","last_clean":"2024-04-25T05:48:14.196407+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:48:14.196407+0000","last_undegraded":"2024-04-25T05:48:14.196407+0000","last_fullsized":"2024-04-25T05:48:14.196407+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:14.571417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:14.571417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.666298+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":947,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188234+0000","last_change":"2024-04-25T05:46:37.983775+0000","last_active":"2024-04-25T05:48:14.188234+0000","last_peered":"2024-04-25T05:48:14.188234+0000","last_clean":"2024-04-25T05:48:14.188234+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:48:14.188234+0000","last_undegraded":"2024-04-25T05:48:14.188234+0000","last_fullsized":"2024-04-25T05:48:14.188234+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.882669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.882669+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"420'8564","reported_seq":8639,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.082263+0000","last_change":"2024-04-25T05:46:37.986037+0000","last_active":"2024-04-25T05:48:18.082263+0000","last_peered":"2024-04-25T05:48:18.082263+0000","last_clean":"2024-04-25T05:48:18.082263+0000","last_became_active":"2024-04-25T05:46:37.985670+0000","last_became_peered":"2024-04-25T05:46:37.985670+0000","last_unstale":"2024-04-25T05:48:18.082263+0000","last_undegraded":"2024-04-25T05:48:18.082263+0000","last_fullsized":"2024-04-25T05:48:18.082263+0000","mapping_epoch":387,"log_start":"403'4000","ondisk_log_start":"403'4000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:22.885020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499384320,"num_objects":7620,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":519,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.185846+0000","last_change":"2024-04-25T05:46:46.694407+0000","last_active":"2024-04-25T05:48:14.185846+0000","last_peered":"2024-04-25T05:48:14.185846+0000","last_clean":"2024-04-25T05:48:14.185846+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:48:14.185846+0000","last_undegraded":"2024-04-25T05:48:14.185846+0000","last_fullsized":"2024-04-25T05:48:14.185846+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:46:46.694273+0000","last_deep_scrub":"321'14","last_deep_scrub_stamp":"2024-04-25T05:44:11.494293+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.694273+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097566+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":496,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196285+0000","last_change":"2024-04-25T05:46:45.354890+0000","last_active":"2024-04-25T05:48:14.196285+0000","last_peered":"2024-04-25T05:48:14.196285+0000","last_clean":"2024-04-25T05:48:14.196285+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:48:14.196285+0000","last_undegraded":"2024-04-25T05:48:14.196285+0000","last_fullsized":"2024-04-25T05:48:14.196285+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:46:45.354712+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.354712+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181921+0000","last_change":"2024-04-25T05:47:15.018494+0000","last_active":"2024-04-25T05:48:14.181921+0000","last_peered":"2024-04-25T05:48:14.181921+0000","last_clean":"2024-04-25T05:48:14.181921+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:48:14.181921+0000","last_undegraded":"2024-04-25T05:48:14.181921+0000","last_fullsized":"2024-04-25T05:48:14.181921+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:15.018379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:15.018379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.914407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":392,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187169+0000","last_change":"2024-04-25T05:46:44.669771+0000","last_active":"2024-04-25T05:48:14.187169+0000","last_peered":"2024-04-25T05:48:14.187169+0000","last_clean":"2024-04-25T05:48:14.187169+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:48:14.187169+0000","last_undegraded":"2024-04-25T05:48:14.187169+0000","last_fullsized":"2024-04-25T05:48:14.187169+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.669615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.669615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228896+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":941,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184157+0000","last_change":"2024-04-25T05:47:19.766253+0000","last_active":"2024-04-25T05:48:14.184157+0000","last_peered":"2024-04-25T05:48:14.184157+0000","last_clean":"2024-04-25T05:48:14.184157+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:48:14.184157+0000","last_undegraded":"2024-04-25T05:48:14.184157+0000","last_fullsized":"2024-04-25T05:48:14.184157+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:19.766122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.766122+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:25.464800+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"420'8319","reported_seq":8396,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.094277+0000","last_change":"2024-04-25T05:46:37.986967+0000","last_active":"2024-04-25T05:48:18.094277+0000","last_peered":"2024-04-25T05:48:18.094277+0000","last_clean":"2024-04-25T05:48:18.094277+0000","last_became_active":"2024-04-25T05:46:37.986694+0000","last_became_peered":"2024-04-25T05:46:37.986694+0000","last_unstale":"2024-04-25T05:48:18.094277+0000","last_undegraded":"2024-04-25T05:48:18.094277+0000","last_fullsized":"2024-04-25T05:48:18.094277+0000","mapping_epoch":387,"log_start":"401'3400","ondisk_log_start":"401'3400","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490340376,"num_objects":7483,"num_object_clones":0,"num_object_copies":22449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8319,"num_write_kb":505601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":516,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184598+0000","last_change":"2024-04-25T05:47:51.904861+0000","last_active":"2024-04-25T05:48:14.184598+0000","last_peered":"2024-04-25T05:48:14.184598+0000","last_clean":"2024-04-25T05:48:14.184598+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:48:14.184598+0000","last_undegraded":"2024-04-25T05:48:14.184598+0000","last_fullsized":"2024-04-25T05:48:14.184598+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:51.904568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.904568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.642706+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":126,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180809+0000","last_change":"2024-04-25T05:46:36.996561+0000","last_active":"2024-04-25T05:48:14.180809+0000","last_peered":"2024-04-25T05:48:14.180809+0000","last_clean":"2024-04-25T05:48:14.180809+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:48:14.180809+0000","last_undegraded":"2024-04-25T05:48:14.180809+0000","last_fullsized":"2024-04-25T05:48:14.180809+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:46:07.457049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.457049+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":792,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195586+0000","last_change":"2024-04-25T05:47:53.584048+0000","last_active":"2024-04-25T05:48:14.195586+0000","last_peered":"2024-04-25T05:48:14.195586+0000","last_clean":"2024-04-25T05:48:14.195586+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:48:14.195586+0000","last_undegraded":"2024-04-25T05:48:14.195586+0000","last_fullsized":"2024-04-25T05:48:14.195586+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:53.583930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:53.583930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.954029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":766,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194648+0000","last_change":"2024-04-25T05:46:47.388550+0000","last_active":"2024-04-25T05:48:14.194648+0000","last_peered":"2024-04-25T05:48:14.194648+0000","last_clean":"2024-04-25T05:48:14.194648+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:48:14.194648+0000","last_undegraded":"2024-04-25T05:48:14.194648+0000","last_fullsized":"2024-04-25T05:48:14.194648+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.388440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.388440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.581611+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":959,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.186110+0000","last_change":"2024-04-25T05:46:52.649991+0000","last_active":"2024-04-25T05:48:14.186110+0000","last_peered":"2024-04-25T05:48:14.186110+0000","last_clean":"2024-04-25T05:48:14.186110+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:48:14.186110+0000","last_undegraded":"2024-04-25T05:48:14.186110+0000","last_fullsized":"2024-04-25T05:48:14.186110+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:52.649850+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.649850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"420'8479","reported_seq":8553,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.178330+0000","last_change":"2024-04-25T05:46:37.986465+0000","last_active":"2024-04-25T05:48:18.178330+0000","last_peered":"2024-04-25T05:48:18.178330+0000","last_clean":"2024-04-25T05:48:18.178330+0000","last_became_active":"2024-04-25T05:46:37.986157+0000","last_became_peered":"2024-04-25T05:46:37.986157+0000","last_unstale":"2024-04-25T05:48:18.178330+0000","last_undegraded":"2024-04-25T05:48:18.178330+0000","last_fullsized":"2024-04-25T05:48:18.178330+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493813760,"num_objects":7535,"num_object_clones":0,"num_object_copies":22605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183235+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:48:14.183235+0000","last_peered":"2024-04-25T05:48:14.183235+0000","last_clean":"2024-04-25T05:48:14.183235+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:48:14.183235+0000","last_undegraded":"2024-04-25T05:48:14.183235+0000","last_fullsized":"2024-04-25T05:48:14.183235+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:16.696528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":102,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182493+0000","last_change":"2024-04-25T05:47:55.201717+0000","last_active":"2024-04-25T05:48:14.182493+0000","last_peered":"2024-04-25T05:48:14.182493+0000","last_clean":"2024-04-25T05:48:14.182493+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:48:14.182493+0000","last_undegraded":"2024-04-25T05:48:14.182493+0000","last_fullsized":"2024-04-25T05:48:14.182493+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:55.201430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.201430+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:56.091558+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1655,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:16.642262+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:48:16.642262+0000","last_peered":"2024-04-25T05:48:16.642262+0000","last_clean":"2024-04-25T05:48:16.642262+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:48:16.642262+0000","last_undegraded":"2024-04-25T05:48:16.642262+0000","last_fullsized":"2024-04-25T05:48:16.642262+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.661889+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":841,"num_read_kb":836,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":387,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.175604+0000","last_change":"2024-04-25T05:47:25.731787+0000","last_active":"2024-04-25T05:48:14.175604+0000","last_peered":"2024-04-25T05:48:14.175604+0000","last_clean":"2024-04-25T05:48:14.175604+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:48:14.175604+0000","last_undegraded":"2024-04-25T05:48:14.175604+0000","last_fullsized":"2024-04-25T05:48:14.175604+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.731370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:45.013111+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195304+0000","last_change":"2024-04-25T05:46:34.975022+0000","last_active":"2024-04-25T05:48:14.195304+0000","last_peered":"2024-04-25T05:48:14.195304+0000","last_clean":"2024-04-25T05:48:14.195304+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:48:14.195304+0000","last_undegraded":"2024-04-25T05:48:14.195304+0000","last_fullsized":"2024-04-25T05:48:14.195304+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:36.702085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:36.702085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":936,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196306+0000","last_change":"2024-04-25T05:46:43.418452+0000","last_active":"2024-04-25T05:48:14.196306+0000","last_peered":"2024-04-25T05:48:14.196306+0000","last_clean":"2024-04-25T05:48:14.196306+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:48:14.196306+0000","last_undegraded":"2024-04-25T05:48:14.196306+0000","last_fullsized":"2024-04-25T05:48:14.196306+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:43.418024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:43.418024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"420'8530","reported_seq":8686,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.160781+0000","last_change":"2024-04-25T05:48:01.589216+0000","last_active":"2024-04-25T05:48:18.160781+0000","last_peered":"2024-04-25T05:48:18.160781+0000","last_clean":"2024-04-25T05:48:18.160781+0000","last_became_active":"2024-04-25T05:46:37.985141+0000","last_became_peered":"2024-04-25T05:46:37.985141+0000","last_unstale":"2024-04-25T05:48:18.160781+0000","last_undegraded":"2024-04-25T05:48:18.160781+0000","last_fullsized":"2024-04-25T05:48:18.160781+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7377","last_scrub_stamp":"2024-04-25T05:48:01.588877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.588877+0000","objects_scrubbed":7283,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:04.287088+0000","scrub_duration":2528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498335744,"num_objects":7604,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":784,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181614+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:48:14.181614+0000","last_peered":"2024-04-25T05:48:14.181614+0000","last_clean":"2024-04-25T05:48:14.181614+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:48:14.181614+0000","last_undegraded":"2024-04-25T05:48:14.181614+0000","last_fullsized":"2024-04-25T05:48:14.181614+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:57.117658+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":619,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.186894+0000","last_change":"2024-04-25T05:47:29.876796+0000","last_active":"2024-04-25T05:48:14.186894+0000","last_peered":"2024-04-25T05:48:14.186894+0000","last_clean":"2024-04-25T05:48:14.186894+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:48:14.186894+0000","last_undegraded":"2024-04-25T05:48:14.186894+0000","last_fullsized":"2024-04-25T05:48:14.186894+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:47:29.876498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:29.876498+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:37.200762+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188101+0000","last_change":"2024-04-25T05:46:34.970673+0000","last_active":"2024-04-25T05:48:14.188101+0000","last_peered":"2024-04-25T05:48:14.188101+0000","last_clean":"2024-04-25T05:48:14.188101+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:48:14.188101+0000","last_undegraded":"2024-04-25T05:48:14.188101+0000","last_fullsized":"2024-04-25T05:48:14.188101+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.811914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.811914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1072,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180353+0000","last_change":"2024-04-25T05:46:06.483237+0000","last_active":"2024-04-25T05:48:14.180353+0000","last_peered":"2024-04-25T05:48:14.180353+0000","last_clean":"2024-04-25T05:48:14.180353+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:48:14.180353+0000","last_undegraded":"2024-04-25T05:48:14.180353+0000","last_fullsized":"2024-04-25T05:48:14.180353+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:06.483108+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.483108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722425+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.197051+0000","last_change":"2024-04-25T05:47:02.157593+0000","last_active":"2024-04-25T05:48:14.197051+0000","last_peered":"2024-04-25T05:48:14.197051+0000","last_clean":"2024-04-25T05:48:14.197051+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:48:14.197051+0000","last_undegraded":"2024-04-25T05:48:14.197051+0000","last_fullsized":"2024-04-25T05:48:14.197051+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:37.037435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.037435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"420'8464","reported_seq":8538,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.009885+0000","last_change":"2024-04-25T05:46:37.984020+0000","last_active":"2024-04-25T05:48:18.009885+0000","last_peered":"2024-04-25T05:48:18.009885+0000","last_clean":"2024-04-25T05:48:18.009885+0000","last_became_active":"2024-04-25T05:46:37.983253+0000","last_became_peered":"2024-04-25T05:46:37.983253+0000","last_unstale":"2024-04-25T05:48:18.009885+0000","last_undegraded":"2024-04-25T05:48:18.009885+0000","last_fullsized":"2024-04-25T05:48:18.009885+0000","mapping_epoch":387,"log_start":"403'3800","ondisk_log_start":"403'3800","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493223936,"num_objects":7526,"num_object_clones":0,"num_object_copies":22578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8464,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":110,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180226+0000","last_change":"2024-04-25T05:47:33.788454+0000","last_active":"2024-04-25T05:48:14.180226+0000","last_peered":"2024-04-25T05:48:14.180226+0000","last_clean":"2024-04-25T05:48:14.180226+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:48:14.180226+0000","last_undegraded":"2024-04-25T05:48:14.180226+0000","last_fullsized":"2024-04-25T05:48:14.180226+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.788331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.788331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:38.579563+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":901,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196845+0000","last_change":"2024-04-25T05:48:02.601745+0000","last_active":"2024-04-25T05:48:14.196845+0000","last_peered":"2024-04-25T05:48:14.196845+0000","last_clean":"2024-04-25T05:48:14.196845+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:48:14.196845+0000","last_undegraded":"2024-04-25T05:48:14.196845+0000","last_fullsized":"2024-04-25T05:48:14.196845+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:48:02.601574+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.601574+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.802514+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179142+0000","last_change":"2024-04-25T05:46:50.947463+0000","last_active":"2024-04-25T05:48:14.179142+0000","last_peered":"2024-04-25T05:48:14.179142+0000","last_clean":"2024-04-25T05:48:14.179142+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:48:14.179142+0000","last_undegraded":"2024-04-25T05:48:14.179142+0000","last_fullsized":"2024-04-25T05:48:14.179142+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.947345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.947345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194827+0000","last_change":"2024-04-25T05:46:19.992163+0000","last_active":"2024-04-25T05:48:14.194827+0000","last_peered":"2024-04-25T05:48:14.194827+0000","last_clean":"2024-04-25T05:48:14.194827+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:48:14.194827+0000","last_undegraded":"2024-04-25T05:48:14.194827+0000","last_fullsized":"2024-04-25T05:48:14.194827+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:13.364530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:13.364530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181419+0000","last_change":"2024-04-25T05:47:23.888942+0000","last_active":"2024-04-25T05:48:14.181419+0000","last_peered":"2024-04-25T05:48:14.181419+0000","last_clean":"2024-04-25T05:48:14.181419+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:48:14.181419+0000","last_undegraded":"2024-04-25T05:48:14.181419+0000","last_fullsized":"2024-04-25T05:48:14.181419+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:23.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.354969+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1079,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183341+0000","last_change":"2024-04-25T05:46:23.137608+0000","last_active":"2024-04-25T05:48:14.183341+0000","last_peered":"2024-04-25T05:48:14.183341+0000","last_clean":"2024-04-25T05:48:14.183341+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:48:14.183341+0000","last_undegraded":"2024-04-25T05:48:14.183341+0000","last_fullsized":"2024-04-25T05:48:14.183341+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"420'8488","reported_seq":8564,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.158566+0000","last_change":"2024-04-25T05:46:37.985580+0000","last_active":"2024-04-25T05:48:18.158566+0000","last_peered":"2024-04-25T05:48:18.158566+0000","last_clean":"2024-04-25T05:48:18.158566+0000","last_became_active":"2024-04-25T05:46:37.985031+0000","last_became_peered":"2024-04-25T05:46:37.985031+0000","last_unstale":"2024-04-25T05:48:18.158566+0000","last_undegraded":"2024-04-25T05:48:18.158566+0000","last_fullsized":"2024-04-25T05:48:18.158566+0000","mapping_epoch":387,"log_start":"401'3500","ondisk_log_start":"401'3500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.110318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498728960,"num_objects":7610,"num_object_clones":0,"num_object_copies":22830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":379,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195808+0000","last_change":"2024-04-25T05:45:22.680176+0000","last_active":"2024-04-25T05:48:14.195808+0000","last_peered":"2024-04-25T05:48:14.195808+0000","last_clean":"2024-04-25T05:48:14.195808+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:48:14.195808+0000","last_undegraded":"2024-04-25T05:48:14.195808+0000","last_fullsized":"2024-04-25T05:48:14.195808+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:22.679850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:22.679850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1072,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195710+0000","last_change":"2024-04-25T05:46:52.415241+0000","last_active":"2024-04-25T05:48:14.195710+0000","last_peered":"2024-04-25T05:48:14.195710+0000","last_clean":"2024-04-25T05:48:14.195710+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:48:14.195710+0000","last_undegraded":"2024-04-25T05:48:14.195710+0000","last_fullsized":"2024-04-25T05:48:14.195710+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:46:52.415149+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.415149+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:20.560193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":549,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.185696+0000","last_change":"2024-04-25T05:46:50.698635+0000","last_active":"2024-04-25T05:48:14.185696+0000","last_peered":"2024-04-25T05:48:14.185696+0000","last_clean":"2024-04-25T05:48:14.185696+0000","last_became_active":"2024-04-25T05:44:28.159852+0000","last_became_peered":"2024-04-25T05:44:28.159852+0000","last_unstale":"2024-04-25T05:48:14.185696+0000","last_undegraded":"2024-04-25T05:48:14.185696+0000","last_fullsized":"2024-04-25T05:48:14.185696+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:50.698507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:46:50.698507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":92,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180575+0000","last_change":"2024-04-25T05:46:49.868732+0000","last_active":"2024-04-25T05:48:14.180575+0000","last_peered":"2024-04-25T05:48:14.180575+0000","last_clean":"2024-04-25T05:48:14.180575+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:48:14.180575+0000","last_undegraded":"2024-04-25T05:48:14.180575+0000","last_fullsized":"2024-04-25T05:48:14.180575+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:46:49.868648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.868648+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:16.090373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":734,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189834+0000","last_change":"2024-04-25T05:46:16.748067+0000","last_active":"2024-04-25T05:48:14.189834+0000","last_peered":"2024-04-25T05:48:14.189834+0000","last_clean":"2024-04-25T05:48:14.189834+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:48:14.189834+0000","last_undegraded":"2024-04-25T05:48:14.189834+0000","last_fullsized":"2024-04-25T05:48:14.189834+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.747963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.747963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182597+0000","last_change":"2024-04-25T05:46:37.982952+0000","last_active":"2024-04-25T05:48:14.182597+0000","last_peered":"2024-04-25T05:48:14.182597+0000","last_clean":"2024-04-25T05:48:14.182597+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:48:14.182597+0000","last_undegraded":"2024-04-25T05:48:14.182597+0000","last_fullsized":"2024-04-25T05:48:14.182597+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:11.595125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:46:11.595125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"420'8454","reported_seq":8527,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.185771+0000","last_change":"2024-04-25T05:46:37.986363+0000","last_active":"2024-04-25T05:48:18.185771+0000","last_peered":"2024-04-25T05:48:18.185771+0000","last_clean":"2024-04-25T05:48:18.185771+0000","last_became_active":"2024-04-25T05:46:37.986037+0000","last_became_peered":"2024-04-25T05:46:37.986037+0000","last_unstale":"2024-04-25T05:48:18.185771+0000","last_undegraded":"2024-04-25T05:48:18.185771+0000","last_fullsized":"2024-04-25T05:48:18.185771+0000","mapping_epoch":387,"log_start":"401'3600","ondisk_log_start":"401'3600","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487718912,"num_objects":7442,"num_object_clones":0,"num_object_copies":22326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":472,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196143+0000","last_change":"2024-04-25T05:47:23.436651+0000","last_active":"2024-04-25T05:48:14.196143+0000","last_peered":"2024-04-25T05:48:14.196143+0000","last_clean":"2024-04-25T05:48:14.196143+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:48:14.196143+0000","last_undegraded":"2024-04-25T05:48:14.196143+0000","last_fullsized":"2024-04-25T05:48:14.196143+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:23.436515+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.436515+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:52.391863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182455+0000","last_change":"2024-04-25T05:46:34.977874+0000","last_active":"2024-04-25T05:48:14.182455+0000","last_peered":"2024-04-25T05:48:14.182455+0000","last_clean":"2024-04-25T05:48:14.182455+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:48:14.182455+0000","last_undegraded":"2024-04-25T05:48:14.182455+0000","last_fullsized":"2024-04-25T05:48:14.182455+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:27.996019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:45:27.996019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":96,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180011+0000","last_change":"2024-04-25T05:46:19.996324+0000","last_active":"2024-04-25T05:48:14.180011+0000","last_peered":"2024-04-25T05:48:14.180011+0000","last_clean":"2024-04-25T05:48:14.180011+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:48:14.180011+0000","last_undegraded":"2024-04-25T05:48:14.180011+0000","last_fullsized":"2024-04-25T05:48:14.180011+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:45:32.512005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":379,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183333+0000","last_change":"2024-04-25T05:47:19.913147+0000","last_active":"2024-04-25T05:48:14.183333+0000","last_peered":"2024-04-25T05:48:14.183333+0000","last_clean":"2024-04-25T05:48:14.183333+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:48:14.183333+0000","last_undegraded":"2024-04-25T05:48:14.183333+0000","last_fullsized":"2024-04-25T05:48:14.183333+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.913041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.913041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:39.287683+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"420'8606","reported_seq":8682,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.969172+0000","last_change":"2024-04-25T05:46:37.979066+0000","last_active":"2024-04-25T05:48:18.969172+0000","last_peered":"2024-04-25T05:48:18.969172+0000","last_clean":"2024-04-25T05:48:18.969172+0000","last_became_active":"2024-04-25T05:46:37.978316+0000","last_became_peered":"2024-04-25T05:46:37.978316+0000","last_unstale":"2024-04-25T05:48:18.969172+0000","last_undegraded":"2024-04-25T05:48:18.969172+0000","last_fullsized":"2024-04-25T05:48:18.969172+0000","mapping_epoch":387,"log_start":"405'4500","ondisk_log_start":"405'4500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:21.176371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495583232,"num_objects":7562,"num_object_clones":0,"num_object_copies":22686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8606,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":532,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184478+0000","last_change":"2024-04-25T05:47:31.914632+0000","last_active":"2024-04-25T05:48:14.184478+0000","last_peered":"2024-04-25T05:48:14.184478+0000","last_clean":"2024-04-25T05:48:14.184478+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:48:14.184478+0000","last_undegraded":"2024-04-25T05:48:14.184478+0000","last_fullsized":"2024-04-25T05:48:14.184478+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'41","last_scrub_stamp":"2024-04-25T05:47:31.914155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.914155+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:56.812560+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":808,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181269+0000","last_change":"2024-04-25T05:46:45.682570+0000","last_active":"2024-04-25T05:48:14.181269+0000","last_peered":"2024-04-25T05:48:14.181269+0000","last_clean":"2024-04-25T05:48:14.181269+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:48:14.181269+0000","last_undegraded":"2024-04-25T05:48:14.181269+0000","last_fullsized":"2024-04-25T05:48:14.181269+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:46:45.682441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.682441+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:18.760645+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":110,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182774+0000","last_change":"2024-04-25T05:47:59.817680+0000","last_active":"2024-04-25T05:48:14.182774+0000","last_peered":"2024-04-25T05:48:14.182774+0000","last_clean":"2024-04-25T05:48:14.182774+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:48:14.182774+0000","last_undegraded":"2024-04-25T05:48:14.182774+0000","last_fullsized":"2024-04-25T05:48:14.182774+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.817559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.817559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:17.472338+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181042+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:48:14.181042+0000","last_peered":"2024-04-25T05:48:14.181042+0000","last_clean":"2024-04-25T05:48:14.181042+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:48:14.181042+0000","last_undegraded":"2024-04-25T05:48:14.181042+0000","last_fullsized":"2024-04-25T05:48:14.181042+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.889858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"420'8481","reported_seq":8556,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:18.168300+0000","last_change":"2024-04-25T05:46:37.986817+0000","last_active":"2024-04-25T05:48:18.168300+0000","last_peered":"2024-04-25T05:48:18.168300+0000","last_clean":"2024-04-25T05:48:18.168300+0000","last_became_active":"2024-04-25T05:46:37.983489+0000","last_became_peered":"2024-04-25T05:46:37.983489+0000","last_unstale":"2024-04-25T05:48:18.168300+0000","last_undegraded":"2024-04-25T05:48:18.168300+0000","last_fullsized":"2024-04-25T05:48:18.168300+0000","mapping_epoch":387,"log_start":"401'3500","ondisk_log_start":"401'3500","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4981,"log_dups_size":0,"ondisk_log_size":4981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.117841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495910912,"num_objects":7567,"num_object_clones":0,"num_object_copies":22701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8481,"num_write_kb":513536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":110,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179137+0000","last_change":"2024-04-25T05:47:57.840420+0000","last_active":"2024-04-25T05:48:14.179137+0000","last_peered":"2024-04-25T05:48:14.179137+0000","last_clean":"2024-04-25T05:48:14.179137+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:48:14.179137+0000","last_undegraded":"2024-04-25T05:48:14.179137+0000","last_fullsized":"2024-04-25T05:48:14.179137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:57.839912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:57.839912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:02.350285+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":404,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194279+0000","last_change":"2024-04-25T05:44:12.362275+0000","last_active":"2024-04-25T05:48:14.194279+0000","last_peered":"2024-04-25T05:48:14.194279+0000","last_clean":"2024-04-25T05:48:14.194279+0000","last_became_active":"2024-04-25T05:40:55.981581+0000","last_became_peered":"2024-04-25T05:40:55.981581+0000","last_unstale":"2024-04-25T05:48:14.194279+0000","last_undegraded":"2024-04-25T05:48:14.194279+0000","last_fullsized":"2024-04-25T05:48:14.194279+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:44:12.362144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:44:12.362144+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184189+0000","last_change":"2024-04-25T05:46:46.685480+0000","last_active":"2024-04-25T05:48:14.184189+0000","last_peered":"2024-04-25T05:48:14.184189+0000","last_clean":"2024-04-25T05:48:14.184189+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:48:14.184189+0000","last_undegraded":"2024-04-25T05:48:14.184189+0000","last_fullsized":"2024-04-25T05:48:14.184189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.685365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.685365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.726677+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":746,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190813+0000","last_change":"2024-04-25T05:46:52.779088+0000","last_active":"2024-04-25T05:48:14.190813+0000","last_peered":"2024-04-25T05:48:14.190813+0000","last_clean":"2024-04-25T05:48:14.190813+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:48:14.190813+0000","last_undegraded":"2024-04-25T05:48:14.190813+0000","last_fullsized":"2024-04-25T05:48:14.190813+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:52.779013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.779013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"420'8540","reported_seq":8614,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:17.989592+0000","last_change":"2024-04-25T05:46:37.981240+0000","last_active":"2024-04-25T05:48:17.989592+0000","last_peered":"2024-04-25T05:48:17.989592+0000","last_clean":"2024-04-25T05:48:17.989592+0000","last_became_active":"2024-04-25T05:46:37.980913+0000","last_became_peered":"2024-04-25T05:46:37.980913+0000","last_unstale":"2024-04-25T05:48:17.989592+0000","last_undegraded":"2024-04-25T05:48:17.989592+0000","last_fullsized":"2024-04-25T05:48:17.989592+0000","mapping_epoch":387,"log_start":"403'3900","ondisk_log_start":"403'3900","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499384320,"num_objects":7620,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8540,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":958,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182885+0000","last_change":"2024-04-25T05:46:47.674917+0000","last_active":"2024-04-25T05:48:14.182885+0000","last_peered":"2024-04-25T05:48:14.182885+0000","last_clean":"2024-04-25T05:48:14.182885+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:48:14.182885+0000","last_undegraded":"2024-04-25T05:48:14.182885+0000","last_fullsized":"2024-04-25T05:48:14.182885+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:46:47.674789+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.674789+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.726643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":108,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184058+0000","last_change":"2024-04-25T05:47:23.979227+0000","last_active":"2024-04-25T05:48:14.184058+0000","last_peered":"2024-04-25T05:48:14.184058+0000","last_clean":"2024-04-25T05:48:14.184058+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:48:14.184058+0000","last_undegraded":"2024-04-25T05:48:14.184058+0000","last_fullsized":"2024-04-25T05:48:14.184058+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:47:23.979075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.979075+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:53.818392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":802,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182583+0000","last_change":"2024-04-25T05:44:28.205600+0000","last_active":"2024-04-25T05:48:14.182583+0000","last_peered":"2024-04-25T05:48:14.182583+0000","last_clean":"2024-04-25T05:48:14.182583+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:48:14.182583+0000","last_undegraded":"2024-04-25T05:48:14.182583+0000","last_fullsized":"2024-04-25T05:48:14.182583+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:44:07.061592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:07.061592+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180683+0000","last_change":"2024-04-25T05:46:34.973794+0000","last_active":"2024-04-25T05:48:14.180683+0000","last_peered":"2024-04-25T05:48:14.180683+0000","last_clean":"2024-04-25T05:48:14.180683+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:48:14.180683+0000","last_undegraded":"2024-04-25T05:48:14.180683+0000","last_fullsized":"2024-04-25T05:48:14.180683+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:10.692888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.692888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195889+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:48:14.195889+0000","last_peered":"2024-04-25T05:48:14.195889+0000","last_clean":"2024-04-25T05:48:14.195889+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:48:14.195889+0000","last_undegraded":"2024-04-25T05:48:14.195889+0000","last_fullsized":"2024-04-25T05:48:14.195889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:47.643410+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":507,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.185531+0000","last_change":"2024-04-25T05:46:47.655135+0000","last_active":"2024-04-25T05:48:14.185531+0000","last_peered":"2024-04-25T05:48:14.185531+0000","last_clean":"2024-04-25T05:48:14.185531+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:48:14.185531+0000","last_undegraded":"2024-04-25T05:48:14.185531+0000","last_fullsized":"2024-04-25T05:48:14.185531+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.655013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.655013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097434+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182399+0000","last_change":"2024-04-25T05:47:56.163151+0000","last_active":"2024-04-25T05:48:14.182399+0000","last_peered":"2024-04-25T05:48:14.182399+0000","last_clean":"2024-04-25T05:48:14.182399+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:48:14.182399+0000","last_undegraded":"2024-04-25T05:48:14.182399+0000","last_fullsized":"2024-04-25T05:48:14.182399+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:56.163019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.163019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:01.308762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":761,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181002+0000","last_change":"2024-04-25T05:47:32.863388+0000","last_active":"2024-04-25T05:48:14.181002+0000","last_peered":"2024-04-25T05:48:14.181002+0000","last_clean":"2024-04-25T05:48:14.181002+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:48:14.181002+0000","last_undegraded":"2024-04-25T05:48:14.181002+0000","last_fullsized":"2024-04-25T05:48:14.181002+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:32.863250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.863250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:46.805628+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":460,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189747+0000","last_change":"2024-04-25T05:46:20.806968+0000","last_active":"2024-04-25T05:48:14.189747+0000","last_peered":"2024-04-25T05:48:14.189747+0000","last_clean":"2024-04-25T05:48:14.189747+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:48:14.189747+0000","last_undegraded":"2024-04-25T05:48:14.189747+0000","last_fullsized":"2024-04-25T05:48:14.189747+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:46:20.806869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:20.806869+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":780,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183095+0000","last_change":"2024-04-25T05:47:31.715369+0000","last_active":"2024-04-25T05:48:14.183095+0000","last_peered":"2024-04-25T05:48:14.183095+0000","last_clean":"2024-04-25T05:48:14.183095+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:48:14.183095+0000","last_undegraded":"2024-04-25T05:48:14.183095+0000","last_fullsized":"2024-04-25T05:48:14.183095+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.715261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:33.794511+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":737,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195641+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:48:14.195641+0000","last_peered":"2024-04-25T05:48:14.195641+0000","last_clean":"2024-04-25T05:48:14.195641+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:48:14.195641+0000","last_undegraded":"2024-04-25T05:48:14.195641+0000","last_fullsized":"2024-04-25T05:48:14.195641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:06.785637+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":96,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179654+0000","last_change":"2024-04-25T05:46:19.984536+0000","last_active":"2024-04-25T05:48:14.179654+0000","last_peered":"2024-04-25T05:48:14.179654+0000","last_clean":"2024-04-25T05:48:14.179654+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:48:14.179654+0000","last_undegraded":"2024-04-25T05:48:14.179654+0000","last_fullsized":"2024-04-25T05:48:14.179654+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:07.695090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.695090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.760065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":92,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181534+0000","last_change":"2024-04-25T05:46:35.006236+0000","last_active":"2024-04-25T05:48:14.181534+0000","last_peered":"2024-04-25T05:48:14.181534+0000","last_clean":"2024-04-25T05:48:14.181534+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:48:14.181534+0000","last_undegraded":"2024-04-25T05:48:14.181534+0000","last_fullsized":"2024-04-25T05:48:14.181534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:45:37.317498+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:45:37.317498+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":373,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183092+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:48:14.183092+0000","last_peered":"2024-04-25T05:48:14.183092+0000","last_clean":"2024-04-25T05:48:14.183092+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:48:14.183092+0000","last_undegraded":"2024-04-25T05:48:14.183092+0000","last_fullsized":"2024-04-25T05:48:14.183092+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:24.171124+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":939,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189301+0000","last_change":"2024-04-25T05:47:18.482166+0000","last_active":"2024-04-25T05:48:14.189301+0000","last_peered":"2024-04-25T05:48:14.189301+0000","last_clean":"2024-04-25T05:48:14.189301+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:48:14.189301+0000","last_undegraded":"2024-04-25T05:48:14.189301+0000","last_fullsized":"2024-04-25T05:48:14.189301+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:47:18.482041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.482041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.135532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":110,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179532+0000","last_change":"2024-04-25T05:48:01.987948+0000","last_active":"2024-04-25T05:48:14.179532+0000","last_peered":"2024-04-25T05:48:14.179532+0000","last_clean":"2024-04-25T05:48:14.179532+0000","last_became_active":"2024-04-25T05:46:34.971901+0000","last_became_peered":"2024-04-25T05:46:34.971901+0000","last_unstale":"2024-04-25T05:48:14.179532+0000","last_undegraded":"2024-04-25T05:48:14.179532+0000","last_fullsized":"2024-04-25T05:48:14.179532+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.987808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.987808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:21.635305+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":804,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188817+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:48:14.188817+0000","last_peered":"2024-04-25T05:48:14.188817+0000","last_clean":"2024-04-25T05:48:14.188817+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:48:14.188817+0000","last_undegraded":"2024-04-25T05:48:14.188817+0000","last_fullsized":"2024-04-25T05:48:14.188817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:10.540049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":774,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183566+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:48:14.183566+0000","last_peered":"2024-04-25T05:48:14.183566+0000","last_clean":"2024-04-25T05:48:14.183566+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:48:14.183566+0000","last_undegraded":"2024-04-25T05:48:14.183566+0000","last_fullsized":"2024-04-25T05:48:14.183566+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.690045+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":104,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181720+0000","last_change":"2024-04-25T05:46:34.997195+0000","last_active":"2024-04-25T05:48:14.181720+0000","last_peered":"2024-04-25T05:48:14.181720+0000","last_clean":"2024-04-25T05:48:14.181720+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:48:14.181720+0000","last_undegraded":"2024-04-25T05:48:14.181720+0000","last_fullsized":"2024-04-25T05:48:14.181720+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-04-25T05:46:19.516621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:46:19.516621+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":373,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184035+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:48:14.184035+0000","last_peered":"2024-04-25T05:48:14.184035+0000","last_clean":"2024-04-25T05:48:14.184035+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:48:14.184035+0000","last_undegraded":"2024-04-25T05:48:14.184035+0000","last_fullsized":"2024-04-25T05:48:14.184035+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.718367+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":900,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.176308+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:48:14.176308+0000","last_peered":"2024-04-25T05:48:14.176308+0000","last_clean":"2024-04-25T05:48:14.176308+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:48:14.176308+0000","last_undegraded":"2024-04-25T05:48:14.176308+0000","last_fullsized":"2024-04-25T05:48:14.176308+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.466214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":433,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181283+0000","last_change":"2024-04-25T05:45:33.294954+0000","last_active":"2024-04-25T05:48:14.181283+0000","last_peered":"2024-04-25T05:48:14.181283+0000","last_clean":"2024-04-25T05:48:14.181283+0000","last_became_active":"2024-04-25T05:40:09.727730+0000","last_became_peered":"2024-04-25T05:40:09.727730+0000","last_unstale":"2024-04-25T05:48:14.181283+0000","last_undegraded":"2024-04-25T05:48:14.181283+0000","last_fullsized":"2024-04-25T05:48:14.181283+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":761,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194933+0000","last_change":"2024-04-25T05:47:30.502986+0000","last_active":"2024-04-25T05:48:14.194933+0000","last_peered":"2024-04-25T05:48:14.194933+0000","last_clean":"2024-04-25T05:48:14.194933+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:48:14.194933+0000","last_undegraded":"2024-04-25T05:48:14.194933+0000","last_fullsized":"2024-04-25T05:48:14.194933+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:30.502873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.502873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:44.898108+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":778,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188602+0000","last_change":"2024-04-25T05:46:53.519364+0000","last_active":"2024-04-25T05:48:14.188602+0000","last_peered":"2024-04-25T05:48:14.188602+0000","last_clean":"2024-04-25T05:48:14.188602+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:48:14.188602+0000","last_undegraded":"2024-04-25T05:48:14.188602+0000","last_fullsized":"2024-04-25T05:48:14.188602+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.519223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.519223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":308,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.196750+0000","last_change":"2024-04-25T05:46:20.039638+0000","last_active":"2024-04-25T05:48:14.196750+0000","last_peered":"2024-04-25T05:48:14.196750+0000","last_clean":"2024-04-25T05:48:14.196750+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:48:14.196750+0000","last_undegraded":"2024-04-25T05:48:14.196750+0000","last_fullsized":"2024-04-25T05:48:14.196750+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:46:07.665439+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:46:07.665439+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":761,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184521+0000","last_change":"2024-04-25T05:46:44.682768+0000","last_active":"2024-04-25T05:48:14.184521+0000","last_peered":"2024-04-25T05:48:14.184521+0000","last_clean":"2024-04-25T05:48:14.184521+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:48:14.184521+0000","last_undegraded":"2024-04-25T05:48:14.184521+0000","last_fullsized":"2024-04-25T05:48:14.184521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.682645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.682645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.726706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":762,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190526+0000","last_change":"2024-04-25T05:47:25.459410+0000","last_active":"2024-04-25T05:48:14.190526+0000","last_peered":"2024-04-25T05:48:14.190526+0000","last_clean":"2024-04-25T05:48:14.190526+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:48:14.190526+0000","last_undegraded":"2024-04-25T05:48:14.190526+0000","last_fullsized":"2024-04-25T05:48:14.190526+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:25.459222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.459222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.040578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183847+0000","last_change":"2024-04-25T05:46:53.839614+0000","last_active":"2024-04-25T05:48:14.183847+0000","last_peered":"2024-04-25T05:48:14.183847+0000","last_clean":"2024-04-25T05:48:14.183847+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:48:14.183847+0000","last_undegraded":"2024-04-25T05:48:14.183847+0000","last_fullsized":"2024-04-25T05:48:14.183847+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.839498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.839498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.080601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":839,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188935+0000","last_change":"2024-04-25T05:47:30.478629+0000","last_active":"2024-04-25T05:48:14.188935+0000","last_peered":"2024-04-25T05:48:14.188935+0000","last_clean":"2024-04-25T05:48:14.188935+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:48:14.188935+0000","last_undegraded":"2024-04-25T05:48:14.188935+0000","last_fullsized":"2024-04-25T05:48:14.188935+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.478462+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:30.722214+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181171+0000","last_change":"2024-04-25T05:47:26.965285+0000","last_active":"2024-04-25T05:48:14.181171+0000","last_peered":"2024-04-25T05:48:14.181171+0000","last_clean":"2024-04-25T05:48:14.181171+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:48:14.181171+0000","last_undegraded":"2024-04-25T05:48:14.181171+0000","last_fullsized":"2024-04-25T05:48:14.181171+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:26.965135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:26.965135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:27.629847+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":772,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180955+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:48:14.180955+0000","last_peered":"2024-04-25T05:48:14.180955+0000","last_clean":"2024-04-25T05:48:14.180955+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:48:14.180955+0000","last_undegraded":"2024-04-25T05:48:14.180955+0000","last_fullsized":"2024-04-25T05:48:14.180955+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.640824+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":770,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190934+0000","last_change":"2024-04-25T05:46:19.986880+0000","last_active":"2024-04-25T05:48:14.190934+0000","last_peered":"2024-04-25T05:48:14.190934+0000","last_clean":"2024-04-25T05:48:14.190934+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:48:14.190934+0000","last_undegraded":"2024-04-25T05:48:14.190934+0000","last_fullsized":"2024-04-25T05:48:14.190934+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:17.841621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:46:17.841621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228746+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":778,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187387+0000","last_change":"2024-04-25T05:45:23.020898+0000","last_active":"2024-04-25T05:48:14.187387+0000","last_peered":"2024-04-25T05:48:14.187387+0000","last_clean":"2024-04-25T05:48:14.187387+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:48:14.187387+0000","last_undegraded":"2024-04-25T05:48:14.187387+0000","last_fullsized":"2024-04-25T05:48:14.187387+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:45:23.020614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.020614+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228497+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182994+0000","last_change":"2024-04-25T05:47:32.942934+0000","last_active":"2024-04-25T05:48:14.182994+0000","last_peered":"2024-04-25T05:48:14.182994+0000","last_clean":"2024-04-25T05:48:14.182994+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:48:14.182994+0000","last_undegraded":"2024-04-25T05:48:14.182994+0000","last_fullsized":"2024-04-25T05:48:14.182994+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:32.942795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.942795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:42.052701+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":477,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182832+0000","last_change":"2024-04-25T05:47:28.867384+0000","last_active":"2024-04-25T05:48:14.182832+0000","last_peered":"2024-04-25T05:48:14.182832+0000","last_clean":"2024-04-25T05:48:14.182832+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:48:14.182832+0000","last_undegraded":"2024-04-25T05:48:14.182832+0000","last_fullsized":"2024-04-25T05:48:14.182832+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.867273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.867273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.337979+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":763,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183438+0000","last_change":"2024-04-25T05:46:20.027611+0000","last_active":"2024-04-25T05:48:14.183438+0000","last_peered":"2024-04-25T05:48:14.183438+0000","last_clean":"2024-04-25T05:48:14.183438+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:48:14.183438+0000","last_undegraded":"2024-04-25T05:48:14.183438+0000","last_fullsized":"2024-04-25T05:48:14.183438+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722191+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":590,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184302+0000","last_change":"2024-04-25T05:46:44.699964+0000","last_active":"2024-04-25T05:48:14.184302+0000","last_peered":"2024-04-25T05:48:14.184302+0000","last_clean":"2024-04-25T05:48:14.184302+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:48:14.184302+0000","last_undegraded":"2024-04-25T05:48:14.184302+0000","last_fullsized":"2024-04-25T05:48:14.184302+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:46:44.699808+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.699808+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:17.097651+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":777,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.187286+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:48:14.187286+0000","last_peered":"2024-04-25T05:48:14.187286+0000","last_clean":"2024-04-25T05:48:14.187286+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:48:14.187286+0000","last_undegraded":"2024-04-25T05:48:14.187286+0000","last_fullsized":"2024-04-25T05:48:14.187286+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:08.724783+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":371,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184275+0000","last_change":"2024-04-25T05:46:19.988527+0000","last_active":"2024-04-25T05:48:14.184275+0000","last_peered":"2024-04-25T05:48:14.184275+0000","last_clean":"2024-04-25T05:48:14.184275+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:48:14.184275+0000","last_undegraded":"2024-04-25T05:48:14.184275+0000","last_fullsized":"2024-04-25T05:48:14.184275+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":759,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195475+0000","last_change":"2024-04-25T05:47:59.671965+0000","last_active":"2024-04-25T05:48:14.195475+0000","last_peered":"2024-04-25T05:48:14.195475+0000","last_clean":"2024-04-25T05:48:14.195475+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:48:14.195475+0000","last_undegraded":"2024-04-25T05:48:14.195475+0000","last_fullsized":"2024-04-25T05:48:14.195475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.671833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.671833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:07.546615+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":446,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184406+0000","last_change":"2024-04-25T05:47:21.817524+0000","last_active":"2024-04-25T05:48:14.184406+0000","last_peered":"2024-04-25T05:48:14.184406+0000","last_clean":"2024-04-25T05:48:14.184406+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:48:14.184406+0000","last_undegraded":"2024-04-25T05:48:14.184406+0000","last_fullsized":"2024-04-25T05:48:14.184406+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:47:21.817370+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:47:21.817370+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:23.100421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":754,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194397+0000","last_change":"2024-04-25T05:44:15.347952+0000","last_active":"2024-04-25T05:48:14.194397+0000","last_peered":"2024-04-25T05:48:14.194397+0000","last_clean":"2024-04-25T05:48:14.194397+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:48:14.194397+0000","last_undegraded":"2024-04-25T05:48:14.194397+0000","last_fullsized":"2024-04-25T05:48:14.194397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:44:15.347805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:44:15.347805+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":595,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.184061+0000","last_change":"2024-04-25T05:47:56.016055+0000","last_active":"2024-04-25T05:48:14.184061+0000","last_peered":"2024-04-25T05:48:14.184061+0000","last_clean":"2024-04-25T05:48:14.184061+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:48:14.184061+0000","last_undegraded":"2024-04-25T05:48:14.184061+0000","last_fullsized":"2024-04-25T05:48:14.184061+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:47:56.015913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.015913+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.422541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":790,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181428+0000","last_change":"2024-04-25T05:46:34.991626+0000","last_active":"2024-04-25T05:48:14.181428+0000","last_peered":"2024-04-25T05:48:14.181428+0000","last_clean":"2024-04-25T05:48:14.181428+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:48:14.181428+0000","last_undegraded":"2024-04-25T05:48:14.181428+0000","last_fullsized":"2024-04-25T05:48:14.181428+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:46:09.493093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.493093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":785,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194473+0000","last_change":"2024-04-25T05:47:16.525921+0000","last_active":"2024-04-25T05:48:14.194473+0000","last_peered":"2024-04-25T05:48:14.194473+0000","last_clean":"2024-04-25T05:48:14.194473+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:48:14.194473+0000","last_undegraded":"2024-04-25T05:48:14.194473+0000","last_fullsized":"2024-04-25T05:48:14.194473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:16.525560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:16.525560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:41.665516+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":795,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.175569+0000","last_change":"2024-04-25T05:48:00.830523+0000","last_active":"2024-04-25T05:48:14.175569+0000","last_peered":"2024-04-25T05:48:14.175569+0000","last_clean":"2024-04-25T05:48:14.175569+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:48:14.175569+0000","last_undegraded":"2024-04-25T05:48:14.175569+0000","last_fullsized":"2024-04-25T05:48:14.175569+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.830408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.175161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":437,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188974+0000","last_change":"2024-04-25T05:46:47.610533+0000","last_active":"2024-04-25T05:48:14.188974+0000","last_peered":"2024-04-25T05:48:14.188974+0000","last_clean":"2024-04-25T05:48:14.188974+0000","last_became_active":"2024-04-25T05:46:20.009022+0000","last_became_peered":"2024-04-25T05:46:20.009022+0000","last_unstale":"2024-04-25T05:48:14.188974+0000","last_undegraded":"2024-04-25T05:48:14.188974+0000","last_fullsized":"2024-04-25T05:48:14.188974+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.610387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.610387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228803+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":758,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195786+0000","last_change":"2024-04-25T05:46:49.931810+0000","last_active":"2024-04-25T05:48:14.195786+0000","last_peered":"2024-04-25T05:48:14.195786+0000","last_clean":"2024-04-25T05:48:14.195786+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:48:14.195786+0000","last_undegraded":"2024-04-25T05:48:14.195786+0000","last_fullsized":"2024-04-25T05:48:14.195786+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:49.930039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.930039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228829+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":120,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.179449+0000","last_change":"2024-04-25T05:47:25.921271+0000","last_active":"2024-04-25T05:48:14.179449+0000","last_peered":"2024-04-25T05:48:14.179449+0000","last_clean":"2024-04-25T05:48:14.179449+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:48:14.179449+0000","last_undegraded":"2024-04-25T05:48:14.179449+0000","last_fullsized":"2024-04-25T05:48:14.179449+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:47:25.921121+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.921121+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:26.176425+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":249,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183742+0000","last_change":"2024-04-25T05:48:05.703358+0000","last_active":"2024-04-25T05:48:14.183742+0000","last_peered":"2024-04-25T05:48:14.183742+0000","last_clean":"2024-04-25T05:48:14.183742+0000","last_became_active":"2024-04-25T05:44:28.166774+0000","last_became_peered":"2024-04-25T05:44:28.166774+0000","last_unstale":"2024-04-25T05:48:14.183742+0000","last_undegraded":"2024-04-25T05:48:14.183742+0000","last_fullsized":"2024-04-25T05:48:14.183742+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:11.734140+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":769,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.195709+0000","last_change":"2024-04-25T05:46:34.974900+0000","last_active":"2024-04-25T05:48:14.195709+0000","last_peered":"2024-04-25T05:48:14.195709+0000","last_clean":"2024-04-25T05:48:14.195709+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:48:14.195709+0000","last_undegraded":"2024-04-25T05:48:14.195709+0000","last_fullsized":"2024-04-25T05:48:14.195709+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:02.270844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:02.270844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":791,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.178503+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:48:14.178503+0000","last_peered":"2024-04-25T05:48:14.178503+0000","last_clean":"2024-04-25T05:48:14.178503+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:48:14.178503+0000","last_undegraded":"2024-04-25T05:48:14.178503+0000","last_fullsized":"2024-04-25T05:48:14.178503+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.067934+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":496,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190320+0000","last_change":"2024-04-25T05:46:05.752045+0000","last_active":"2024-04-25T05:48:14.190320+0000","last_peered":"2024-04-25T05:48:14.190320+0000","last_clean":"2024-04-25T05:48:14.190320+0000","last_became_active":"2024-04-25T05:43:35.349454+0000","last_became_peered":"2024-04-25T05:43:35.349454+0000","last_unstale":"2024-04-25T05:48:14.190320+0000","last_undegraded":"2024-04-25T05:48:14.190320+0000","last_fullsized":"2024-04-25T05:48:14.190320+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:46:05.751904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:05.751904+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182558+0000","last_change":"2024-04-25T05:48:04.097751+0000","last_active":"2024-04-25T05:48:14.182558+0000","last_peered":"2024-04-25T05:48:14.182558+0000","last_clean":"2024-04-25T05:48:14.182558+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:48:14.182558+0000","last_undegraded":"2024-04-25T05:48:14.182558+0000","last_fullsized":"2024-04-25T05:48:14.182558+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:04.097613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.097613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:13.875974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":751,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.183251+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:48:14.183251+0000","last_peered":"2024-04-25T05:48:14.183251+0000","last_clean":"2024-04-25T05:48:14.183251+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:48:14.183251+0000","last_undegraded":"2024-04-25T05:48:14.183251+0000","last_fullsized":"2024-04-25T05:48:14.183251+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.722283+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182707+0000","last_change":"2024-04-25T05:46:34.985777+0000","last_active":"2024-04-25T05:48:14.182707+0000","last_peered":"2024-04-25T05:48:14.182707+0000","last_clean":"2024-04-25T05:48:14.182707+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:48:14.182707+0000","last_undegraded":"2024-04-25T05:48:14.182707+0000","last_fullsized":"2024-04-25T05:48:14.182707+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:38.767074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:38.767074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"320'28","reported_seq":553,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.188269+0000","last_change":"2024-04-25T05:46:20.046230+0000","last_active":"2024-04-25T05:48:14.188269+0000","last_peered":"2024-04-25T05:48:14.188269+0000","last_clean":"2024-04-25T05:48:14.188269+0000","last_became_active":"2024-04-25T05:46:20.009817+0000","last_became_peered":"2024-04-25T05:46:20.009817+0000","last_unstale":"2024-04-25T05:48:14.188269+0000","last_undegraded":"2024-04-25T05:48:14.188269+0000","last_fullsized":"2024-04-25T05:48:14.188269+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'28","last_scrub_stamp":"2024-04-25T05:46:10.858811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:10.858811+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.181631+0000","last_change":"2024-04-25T05:48:00.061365+0000","last_active":"2024-04-25T05:48:14.181631+0000","last_peered":"2024-04-25T05:48:14.181631+0000","last_clean":"2024-04-25T05:48:14.181631+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:48:14.181631+0000","last_undegraded":"2024-04-25T05:48:14.181631+0000","last_fullsized":"2024-04-25T05:48:14.181631+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.061240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.061240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:07.003582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180717+0000","last_change":"2024-04-25T05:46:34.979661+0000","last_active":"2024-04-25T05:48:14.180717+0000","last_peered":"2024-04-25T05:48:14.180717+0000","last_clean":"2024-04-25T05:48:14.180717+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:48:14.180717+0000","last_undegraded":"2024-04-25T05:48:14.180717+0000","last_fullsized":"2024-04-25T05:48:14.180717+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:23.299183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:23.299183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":379,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.199326+0000","last_change":"2024-04-25T05:47:13.375334+0000","last_active":"2024-04-25T05:48:14.199326+0000","last_peered":"2024-04-25T05:48:14.199326+0000","last_clean":"2024-04-25T05:48:14.199326+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:48:14.199326+0000","last_undegraded":"2024-04-25T05:48:14.199326+0000","last_fullsized":"2024-04-25T05:48:14.199326+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:13.375091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:13.375091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:19.270626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1042,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189036+0000","last_change":"2024-04-25T05:46:48.610335+0000","last_active":"2024-04-25T05:48:14.189036+0000","last_peered":"2024-04-25T05:48:14.189036+0000","last_clean":"2024-04-25T05:48:14.189036+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:48:14.189036+0000","last_undegraded":"2024-04-25T05:48:14.189036+0000","last_fullsized":"2024-04-25T05:48:14.189036+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:46:48.610157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.610157+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":779,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.189557+0000","last_change":"2024-04-25T05:47:57.543231+0000","last_active":"2024-04-25T05:48:14.189557+0000","last_peered":"2024-04-25T05:48:14.189557+0000","last_clean":"2024-04-25T05:48:14.189557+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:48:14.189557+0000","last_undegraded":"2024-04-25T05:48:14.189557+0000","last_fullsized":"2024-04-25T05:48:14.189557+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:57.543118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:47:57.543118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:58.141942+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":729,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.185765+0000","last_change":"2024-04-25T05:47:18.658184+0000","last_active":"2024-04-25T05:48:14.185765+0000","last_peered":"2024-04-25T05:48:14.185765+0000","last_clean":"2024-04-25T05:48:14.185765+0000","last_became_active":"2024-04-25T05:40:09.736537+0000","last_became_peered":"2024-04-25T05:40:09.736537+0000","last_unstale":"2024-04-25T05:48:14.185765+0000","last_undegraded":"2024-04-25T05:48:14.185765+0000","last_fullsized":"2024-04-25T05:48:14.185765+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:18.657935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.657935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:31.709365+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":745,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.190600+0000","last_change":"2024-04-25T05:46:46.830806+0000","last_active":"2024-04-25T05:48:14.190600+0000","last_peered":"2024-04-25T05:48:14.190600+0000","last_clean":"2024-04-25T05:48:14.190600+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:48:14.190600+0000","last_undegraded":"2024-04-25T05:48:14.190600+0000","last_fullsized":"2024-04-25T05:48:14.190600+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.830676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.830676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:48:15.228934+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":851,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.176054+0000","last_change":"2024-04-25T05:47:55.921938+0000","last_active":"2024-04-25T05:48:14.176054+0000","last_peered":"2024-04-25T05:48:14.176054+0000","last_clean":"2024-04-25T05:48:14.176054+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:48:14.176054+0000","last_undegraded":"2024-04-25T05:48:14.176054+0000","last_fullsized":"2024-04-25T05:48:14.176054+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:55.921801+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.921801+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:03.833450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":88,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.182833+0000","last_change":"2024-04-25T05:46:44.971898+0000","last_active":"2024-04-25T05:48:14.182833+0000","last_peered":"2024-04-25T05:48:14.182833+0000","last_clean":"2024-04-25T05:48:14.182833+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:48:14.182833+0000","last_undegraded":"2024-04-25T05:48:14.182833+0000","last_fullsized":"2024-04-25T05:48:14.182833+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.971755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.971755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":763,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.194562+0000","last_change":"2024-04-25T05:46:34.974443+0000","last_active":"2024-04-25T05:48:14.194562+0000","last_peered":"2024-04-25T05:48:14.194562+0000","last_clean":"2024-04-25T05:48:14.194562+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:48:14.194562+0000","last_undegraded":"2024-04-25T05:48:14.194562+0000","last_fullsized":"2024-04-25T05:48:14.194562+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:05.242375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:05.242375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":80,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-04-25T05:48:14.180883+0000","last_change":"2024-04-25T05:46:34.988586+0000","last_active":"2024-04-25T05:48:14.180883+0000","last_peered":"2024-04-25T05:48:14.180883+0000","last_clean":"2024-04-25T05:48:14.180883+0000","last_became_active":"2024-04-25T05:46:34.986413+0000","last_became_peered":"2024-04-25T05:46:34.986413+0000","last_unstale":"2024-04-25T05:48:14.180883+0000","last_undegraded":"2024-04-25T05:48:14.180883+0000","last_fullsized":"2024-04-25T05:48:14.180883+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:39.734373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.734373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7916224536,"num_objects":120793,"num_object_clones":0,"num_object_copies":362379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120793,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135903,"num_write_kb":8214209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23728566272,"data_stored":23728554056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76003,"ondisk_log_size":76003,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":841,"num_read_kb":836,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1394,"num_read_kb":1119,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":10814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":964,"ondisk_log_size":964,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070641,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3133184,"kb_used_data":2425912,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90611968,"statfs":{"total":95995035648,"available":92786655232,"internally_reserved":0,"allocated":2484133888,"data_stored":2483667627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":724236746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364981,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4010328,"kb_used_data":3397456,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":89734824,"statfs":{"total":95995035648,"available":91888459776,"internally_reserved":0,"allocated":3478994944,"data_stored":3478568083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627571142},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575054,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3749388,"kb_used_data":2905988,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":89995764,"statfs":{"total":95995035648,"available":92155662336,"internally_reserved":0,"allocated":2975731712,"data_stored":2975251807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738580,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411529,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3910252,"kb_used_data":3383460,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":89834900,"statfs":{"total":95995035648,"available":91990937600,"internally_reserved":0,"allocated":3464663040,"data_stored":3464215721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149891,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4797120,"kb_used_data":3849592,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":88948032,"statfs":{"total":95995035648,"available":91082784768,"internally_reserved":0,"allocated":3941982208,"data_stored":3941514892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":970193351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986366,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4987380,"kb_used_data":4392492,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":88757772,"statfs":{"total":95995035648,"available":90887958528,"internally_reserved":0,"allocated":4497911808,"data_stored":4497456457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692026,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3332512,"kb_used_data":2904280,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90412640,"statfs":{"total":95995035648,"available":92582543360,"internally_reserved":0,"allocated":2973982720,"data_stored":2973547773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2962751488,"data_stored":2962751488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3451715584,"data_stored":3451715584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3904835584,"data_stored":3904831512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4485484544,"data_stored":4485480472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2970877952,"data_stored":2970877952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3475509248,"data_stored":3475505176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2477391872,"data_stored":2477391872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:48:21.282 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:48:21.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:48:21.283 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-25T05:48:21.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-04-25T05:48:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:21.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:21.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:21.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:48:21.420 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:21.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:21 smithi060 ceph-mon[30898]: from='client.25660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:21.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:21 smithi060 ceph-mon[30898]: pgmap v760: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 901 B/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-25T05:48:21.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:21 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:48:21.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:21 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:48:21.462 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:21.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:21.564 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[27010]: from='client.25660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:21.564 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[27010]: pgmap v760: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 901 B/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-25T05:48:21.564 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:48:21.564 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:48:21.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[37492]: from='client.25660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:48:21.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[37492]: pgmap v760: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 901 B/s rd, 29 MiB/s wr, 1.52k op/s 2024-04-25T05:48:21.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:48:21.565 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:21 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:48:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:21.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:21.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.755 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:21.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:21.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:21.879 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:21.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:22.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:22.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:22.144 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:22.181 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:22.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:48:22.191 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:22.230 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:22.274 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:22.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:22.312 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:22 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:48:22.312 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:22 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:48:22.312 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/321993223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:22.313 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:22.353 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:22.393 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:22.421 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:22.432 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:22.435 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:22.436 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:22.451 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:48:22.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:48:22.452 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/321993223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:22.452 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:48:22.452 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:48:22.452 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/321993223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:22.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:22.537 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:22.537 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:22.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:22.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:22.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:22.726 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:22.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:22.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:22.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:22.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:23.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:23.112 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:23.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:23.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:23.281 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:23.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: pgmap v761: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 3.4 MiB/s wr, 839 op/s 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: osdmap e422: 8 total, 8 up, 7 in 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:48:23.324 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:48:23.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: pgmap v761: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 3.4 MiB/s wr, 839 op/s 2024-04-25T05:48:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: osdmap e422: 8 total, 8 up, 7 in 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:48:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: pgmap v761: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 3.4 MiB/s wr, 839 op/s 2024-04-25T05:48:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: osdmap e422: 8 total, 8 up, 7 in 2024-04-25T05:48:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:48:23.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:23 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:48:23.445 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:23.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:23.599 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:23.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.748 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:23.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:23.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:23.805 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:23.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:23.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:23.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:24.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:24.124 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:24.162 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:24.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:24.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:24.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.425 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:24.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:48:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:24 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:48:24.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:24.526 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:24.526 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:24.541 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.561 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:24.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:24.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:48:24.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:48:24.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:48:24.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:24 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:48:24.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:24.693 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:24.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:24.832 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:24.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:24.970 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:24.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:25.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:25.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:25.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:25.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:25.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-04-25T05:48:25.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:25.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:25.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: pgmap v763: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:48:25.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: pgmap v763: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:48:25.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:25 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: pgmap v763: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:48:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:25 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:48:25.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:25.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:25.639 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:25.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:25.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:25.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:25.887 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:25.915 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:25.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:26.017 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:26.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.061 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:26.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:26.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:48:26.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:26.208 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:26.259 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:48:26.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:48:26.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:26.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:26.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:26.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:26.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:26 smithi060 ceph-mon[30898]: osdmap e423: 8 total, 8 up, 7 in 2024-04-25T05:48:26.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4077117919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.507 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:26.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:26.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[27010]: osdmap e423: 8 total, 8 up, 7 in 2024-04-25T05:48:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4077117919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[37492]: osdmap e423: 8 total, 8 up, 7 in 2024-04-25T05:48:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4077117919' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:26.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.767 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:26.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:26.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:26.899 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:26.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:27.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:27.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:27.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:27.171 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:27.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:27.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:27.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:27.470 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: pgmap v765: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 880 op/s 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:27.471 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:27 smithi060 ceph-mon[30898]: osdmap e424: 8 total, 8 up, 7 in 2024-04-25T05:48:27.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.590 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:27.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:27.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:48:27.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: pgmap v765: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 880 op/s 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[37492]: osdmap e424: 8 total, 8 up, 7 in 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: pgmap v765: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 880 op/s 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:48:27.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:48:27.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:48:27.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:48:27.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:27.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:27 smithi043 ceph-mon[27010]: osdmap e424: 8 total, 8 up, 7 in 2024-04-25T05:48:27.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:27.692 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:27.692 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:27.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:27.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:27.873 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:27.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:27.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:28.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:28.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:28.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:28.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:28.297 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:28.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:28.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:28.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:28.478 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:28 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:48:28.478 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:28 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:48:28.478 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:28 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:48:28.478 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:28 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:48:28.478 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:28 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:48:28.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:48:28.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:28.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:48:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:48:28.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:48:28.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:28 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:48:28.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:28.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:28.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:28.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:28.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:28.943 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:28.990 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:28.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:29.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:29.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:29.226 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.264 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:29.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:29.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:29.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:29.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:29.545 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:29.621 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:29.621 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:29.621 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:29 smithi060 ceph-mon[30898]: 10.3 scrub starts 2024-04-25T05:48:29.621 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:29 smithi060 ceph-mon[30898]: pgmap v767: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-04-25T05:48:29.622 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:29 smithi060 ceph-mon[30898]: 10.0 scrub starts 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[37492]: 10.3 scrub starts 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[37492]: pgmap v767: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[37492]: 10.0 scrub starts 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[27010]: 10.3 scrub starts 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[27010]: pgmap v767: 158 pgs: 158 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 995 op/s 2024-04-25T05:48:29.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-mon[27010]: 10.0 scrub starts 2024-04-25T05:48:29.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:29.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:29.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:29.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:29.944 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:29.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:29.979 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:30.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:30.083 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:30.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:30.217 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:30.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:30.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:48:30.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:30.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:30.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:30.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:48:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:48:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:48:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:48:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:48:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:48:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:48:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:30 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:48:30.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:30 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:48:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:30 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:48:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:30 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:48:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:30 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:30 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:48:30.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:30.764 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:30.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:30.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:30.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:31.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:31.048 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:31.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:31.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:31.405 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:31.405 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:31.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:31.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 10.3 scrub ok 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: pgmap v768: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:48:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:31 smithi060 ceph-mon[30898]: 10.0 scrub ok 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 10.3 scrub ok 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: pgmap v768: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:48:31.524 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[27010]: 10.0 scrub ok 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 10.3 scrub ok 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: pgmap v768: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:48:31.525 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:31 smithi043 ceph-mon[37492]: 10.0 scrub ok 2024-04-25T05:48:31.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:31.600 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:31.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:31.734 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:31.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:31.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:31.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.005 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:32.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:32.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:32.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:32.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:32.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:32.460 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:32 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:48:32.461 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:32 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:48:32.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:32.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:48:32.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:32.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:32 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:48:32.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:32 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:48:32.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:32 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:48:32.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:32 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:48:32.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:32.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-25T05:48:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:32.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.817 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:32.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:32.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:33.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:33.066 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:33.107 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:33.110 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:33.113 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:33.148 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:33.193 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:33.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:33.235 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:33.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:48:33.252 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:33.281 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:33.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:33.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:33.382 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:33.387 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:33.388 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:33.388 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:33.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:33.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:48:33.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:48:33.421 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 827 op/s 2024-04-25T05:48:33.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/565385509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:33.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:33.422 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:33.500 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:48:33.500 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:48:33.500 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 827 op/s 2024-04-25T05:48:33.500 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/565385509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 827 op/s 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/565385509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:33.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:33.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:33.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:33.634 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:48:33.634 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:48:33.635 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:48:33.635 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:48:33.635 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:48:33.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:33.686 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:33.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:33.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:33.823 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:33.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:33.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:33.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:34.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:34.097 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:34.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.239 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:34.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:34.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.377 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:34.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[37492]: 10.8 scrub starts 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[37492]: 6.7 scrub starts 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[37492]: 6.7 scrub ok 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[27010]: 10.8 scrub starts 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[27010]: 6.7 scrub starts 2024-04-25T05:48:34.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:34 smithi043 ceph-mon[27010]: 6.7 scrub ok 2024-04-25T05:48:34.479 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:34 smithi060 ceph-mon[30898]: 10.8 scrub starts 2024-04-25T05:48:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:34 smithi060 ceph-mon[30898]: 6.7 scrub starts 2024-04-25T05:48:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:34 smithi060 ceph-mon[30898]: 6.7 scrub ok 2024-04-25T05:48:34.515 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:34.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:48:34.617 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:34.655 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:34.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.793 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:34.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:34.894 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:34.924 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:34.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:35.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-25T05:48:35.018 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:48:35.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:35.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:35.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:35.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:35.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:35.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:35.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:35 smithi060 ceph-mon[30898]: 10.5 scrub starts 2024-04-25T05:48:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:35 smithi060 ceph-mon[30898]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:35 smithi060 ceph-mon[30898]: 10.a deep-scrub starts 2024-04-25T05:48:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:35.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:35.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:35.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:35.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[37492]: 10.5 scrub starts 2024-04-25T05:48:35.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[37492]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:35.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[37492]: 10.a deep-scrub starts 2024-04-25T05:48:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[27010]: 10.5 scrub starts 2024-04-25T05:48:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[27010]: pgmap v770: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:35 smithi043 ceph-mon[27010]: 10.a deep-scrub starts 2024-04-25T05:48:35.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:35.747 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:35.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.910 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228018 8.17 8.878e0737 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 353.930456042, 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0050960120000001, 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228018 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.911 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975924+0000", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.711e-06 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975930+0000", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1469999999999998e-06 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975947+0000", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7601999999999999e-05 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975956+0000", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9400000000000008e-06 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976889+0000", 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093282800000000002 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976898+0000", 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8769999999999999e-06 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980999+0000", 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.00410116 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981018+0000", 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8746999999999999e-05 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228040 8.17 8.3c35da57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 353.91300175200001, 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98798208499999995, 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228040 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993377+0000", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.012e-06 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993383+0000", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9769999999999997e-06 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993394+0000", 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0370999999999999e-05 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993402+0000", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1729999999999995e-06 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993458+0000", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6025e-05 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993465+0000", 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2339999999999997e-06 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981341+0000", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98787640899999996 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981358+0000", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6884e-05 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228041 8.17 8.7bebe17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:48:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 353.91284644400002, 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98792126800000002, 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228041 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993532+0000", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8400000000000005e-07 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993535+0000", 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6070000000000002e-06 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993542+0000", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9619999999999996e-06 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993548+0000", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8190000000000003e-06 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993585+0000", 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6328000000000002e-05 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993588+0000", 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4319999999999999e-06 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981443+0000", 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98785475499999997 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981453+0000", 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8810000000000008e-06 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:6 9.7 9:e9b4c7c6:::benchmark_data_smithi043_129820_object5:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 320.70582856700003, 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.006128809, 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200550+0000", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3300000000000002e-07 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200583+0000", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2659e-05 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200590+0000", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9890000000000003e-06 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200595+0000", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7899999999999996e-06 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200646+0000", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0822000000000002e-05 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200656+0000", 2024-04-25T05:48:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5109999999999992e-06 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192142+0000", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99148672699999996 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192281+0000", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139018 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192427+0000", 2024-04-25T05:48:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014579999999999999 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.194168+0000", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017404429999999999 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205031+0000", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010862966 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206577+0000", 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015465780000000001 2024-04-25T05:48:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206635+0000", 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7998000000000003e-05 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206678+0000", 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2975000000000001e-05 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:12 9.d 9:b543e130:::benchmark_data_smithi043_129820_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 320.70567481699999, 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0180756520000001, 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200704+0000", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2099999999999997e-07 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212291+0000", 2024-04-25T05:48:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011587573 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212312+0000", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0627000000000001e-05 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212324+0000", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.185e-05 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212430+0000", 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010610199999999999 2024-04-25T05:48:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212444+0000", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4188999999999999e-05 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205046+0000", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99260219400000005 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205178+0000", 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013207299999999999 2024-04-25T05:48:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205330+0000", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000151438 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.209820+0000", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044903390000000003 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210175+0000", 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00035483800000000001 2024-04-25T05:48:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218678+0000", 2024-04-25T05:48:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0085029509999999999 2024-04-25T05:48:35.936 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218737+0000", 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9036000000000002e-05 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218779+0000", 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1921000000000001e-05 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:20 9.d 9:b555abe5:::benchmark_data_smithi043_129820_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 320.69163718300001, 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0041661829999999, 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 20 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2600000000000002e-07 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214807+0000", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5287e-05 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214816+0000", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7630000000000006e-06 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214821+0000", 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214856+0000", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5147000000000001e-05 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214861+0000", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205852+0000", 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99099027500000003 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206034+0000", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000182628 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206177+0000", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014306599999999999 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210096+0000", 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039190930000000002 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210182+0000", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6057000000000006e-05 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218819+0000", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086365039999999997 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218868+0000", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9302000000000003e-05 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218907+0000", 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8674999999999998e-05 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:34 9.7 9:e321d650:::benchmark_data_smithi043_129820_object33:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 320.68753718300002, 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.988632811, 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 34 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4099999999999998e-07 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218902+0000", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0786999999999999e-05 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218912+0000", 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4930000000000004e-06 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218920+0000", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1890000000000007e-06 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218967+0000", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6733000000000001e-05 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218984+0000", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7153999999999999e-05 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192758+0000", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97377451299999995 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192893+0000", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013458900000000001 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.193067+0000", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017409899999999999 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.195263+0000", 2024-04-25T05:48:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021959700000000002 2024-04-25T05:48:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:35.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206451+0000", 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011187704 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207381+0000", 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093041 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207433+0000", 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2145000000000003e-05 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207474+0000", 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0284000000000003e-05 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122647 9.9 9:946f8a80:::benchmark_data_smithi043_129820_object1333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 229.638071486, 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98571332099999998, 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122647 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268307+0000", 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5600000000000004e-07 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268313+0000", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9560000000000002e-06 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268323+0000", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3479999999999993e-06 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268330+0000", 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.861e-06 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268387+0000", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7014e-05 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268396+0000", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.842e-06 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249234+0000", 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98083744799999995 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249334+0000", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9856000000000001e-05 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249425+0000", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1697e-05 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253800+0000", 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043740819999999996 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253847+0000", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7231999999999998e-05 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253904+0000", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7262000000000002e-05 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253938+0000", 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4394000000000003e-05 2024-04-25T05:48:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254020+0000", 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1372999999999995e-05 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122655 9.9 9:91969b48:::benchmark_data_smithi043_129820_object1341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 229.63374558300001, 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98173263399999999, 2024-04-25T05:48:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122655 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272633+0000", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.39e-07 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272636+0000", 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4369999999999999e-06 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272641+0000", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0069999999999997e-06 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272644+0000", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1300000000000001e-06 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272681+0000", 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7063000000000001e-05 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272687+0000", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2059999999999998e-06 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251540+0000", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97885334999999996 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251625+0000", 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5487000000000003e-05 2024-04-25T05:48:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251692+0000", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6076999999999998e-05 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254167+0000", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002475929 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254218+0000", 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1023999999999998e-05 2024-04-25T05:48:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254276+0000", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7348e-05 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254326+0000", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0139999999999998e-05 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254365+0000", 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8996999999999999e-05 2024-04-25T05:48:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122657 9.9 9:97065b42:::benchmark_data_smithi043_129820_object1343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 229.633276819, 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98447252200000002, 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122657 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273102+0000", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1399999999999995e-07 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273106+0000", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4240000000000003e-06 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273116+0000", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1109999999999997e-06 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273123+0000", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2089999999999999e-06 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273170+0000", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7475999999999997e-05 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273178+0000", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5789999999999998e-06 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252726+0000", 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97954808900000001 2024-04-25T05:48:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252818+0000", 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2471000000000003e-05 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252938+0000", 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119312 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254376+0000", 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001438745 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254418+0000", 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1127000000000001e-05 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257471+0000", 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003053596 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257528+0000", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7151000000000002e-05 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257574+0000", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5318000000000002e-05 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234262 9.13 9:c8a64bff:::benchmark_data_smithi043_129820_object112948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.486855228, 2024-04-25T05:48:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044775727, 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234262 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419524+0000", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.114e-06 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419529+0000", 2024-04-25T05:48:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1320000000000002e-06 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419541+0000", 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2517000000000001e-05 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419551+0000", 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.55e-06 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419626+0000", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5401999999999994e-05 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419639+0000", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3045e-05 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458481+0000", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0388411980000001 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458596+0000", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114869 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458847+0000", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000251838 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463718+0000", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048708349999999996 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463779+0000", 2024-04-25T05:48:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0671000000000002e-05 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464065+0000", 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028592400000000001 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464124+0000", 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8913000000000003e-05 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464180+0000", 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6740000000000002e-05 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464298+0000", 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117979 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234266 9.13 9:cde636a1:::benchmark_data_smithi043_129820_object112952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:48:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 142.48309403600001, 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.043099716, 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234266 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423285+0000", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.47e-07 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423290+0000", 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-06 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423301+0000", 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1249e-05 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423309+0000", 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423370+0000", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1496999999999998e-05 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423380+0000", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.459982+0000", 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.036602386 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460104+0000", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121814 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460347+0000", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024315799999999999 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464108+0000", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003761347 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464306+0000", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019793999999999999 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466211+0000", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001904836 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466254+0000", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2517e-05 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466302+0000", 2024-04-25T05:48:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8791000000000001e-05 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466384+0000", 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1110000000000001e-05 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237263 9.10 9:09fb57ed:::benchmark_data_smithi043_129820_object115949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:48:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.81018665400001, 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0207323559999999, 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237263 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:48:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096192+0000", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8400000000000004e-07 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096195+0000", 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1750000000000001e-06 2024-04-25T05:48:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096202+0000", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8600000000000004e-06 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096206+0000", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096257+0000", 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0695000000000002e-05 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096262+0000", 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5300000000000004e-06 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113813+0000", 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0175506910000001 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114018+0000", 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000205124 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114193+0000", 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000174981 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116636+0000", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024427059999999998 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116671+0000", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5380999999999999e-05 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116825+0000", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000153993 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116873+0000", 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7308000000000002e-05 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116924+0000", 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1087000000000001e-05 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237265 9.13 9:ced551e7:::benchmark_data_smithi043_129820_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.80896320700001, 2024-04-25T05:48:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.029123689, 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237265 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3300000000000001e-07 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097420+0000", 2024-04-25T05:48:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0969999999999999e-06 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097429+0000", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0380000000000005e-06 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097436+0000", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7449999999999995e-06 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097483+0000", 2024-04-25T05:48:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6343e-05 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097489+0000", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6830000000000002e-06 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115159+0000", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0176693729999999 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115269+0000", 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109979 2024-04-25T05:48:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115548+0000", 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027962100000000003 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118288+0000", 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027397879999999999 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120356+0000", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020682280000000001 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126451+0000", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060945569999999996 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126495+0000", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3942e-05 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126538+0000", 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3562000000000001e-05 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237270 9.13 9:ccb5a786:::benchmark_data_smithi043_129820_object115956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.805597627, 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.025924235, 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237270 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100781+0000", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1200000000000002e-07 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100789+0000", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7179999999999996e-06 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100798+0000", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8599999999999999e-06 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100804+0000", 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0009999999999998e-06 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100867+0000", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3851999999999998e-05 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100877+0000", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1940000000000004e-06 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117090+0000", 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016213622 2024-04-25T05:48:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117202+0000", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111869 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117405+0000", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020259799999999999 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126115+0000", 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0087104629999999999 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126211+0000", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5789000000000001e-05 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126619+0000", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00040798499999999998 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126663+0000", 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3543e-05 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126705+0000", 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2029e-05 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237273 9.10 9:0de19dcd:::benchmark_data_smithi043_129820_object115959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.80382634399999, 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0209831819999999, 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237273 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102553+0000", 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.034e-06 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102559+0000", 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9089999999999997e-06 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102570+0000", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.112e-05 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102578+0000", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2709999999999992e-06 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102617+0000", 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8695999999999997e-05 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102625+0000", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9929999999999995e-06 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115511+0000", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0128864559999999 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115643+0000", 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131746 2024-04-25T05:48:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115827+0000", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018383200000000001 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118880+0000", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030529400000000001 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119838+0000", 2024-04-25T05:48:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00095799800000000005 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123433+0000", 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035950230000000001 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123491+0000", 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8242000000000002e-05 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123535+0000", 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3921999999999997e-05 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237293 9.13 9:cf509854:::benchmark_data_smithi043_129820_object115979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.795707521, 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016106186, 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237293 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110671+0000", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110676+0000", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6029999999999998e-06 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110685+0000", 2024-04-25T05:48:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6149999999999997e-06 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110691+0000", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8569999999999997e-06 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110748+0000", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6978999999999998e-05 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110757+0000", 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7949999999999996e-06 2024-04-25T05:48:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118323+0000", 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007566237 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118419+0000", 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5356e-05 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118567+0000", 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014814099999999999 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126167+0000", 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.007599522 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126302+0000", 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013507600000000001 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126712+0000", 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041020800000000001 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126745+0000", 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3250999999999997e-05 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126777+0000", 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1547000000000001e-05 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237310 9.13 9:cc1e0a4c:::benchmark_data_smithi043_129820_object115996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.78647947100001, 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007006198, 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237310 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119899+0000", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9100000000000003e-07 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119903+0000", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5090000000000002e-06 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119912+0000", 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0440000000000001e-06 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119917+0000", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2959999999999998e-06 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119983+0000", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5779999999999997e-05 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119993+0000", 2024-04-25T05:48:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8179999999999999e-06 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119292+0000", 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99929933400000004 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119417+0000", 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125449 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119591+0000", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017395799999999999 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126261+0000", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066693810000000003 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126390+0000", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000129164 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126784+0000", 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039376400000000001 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126863+0000", 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8963999999999998e-05 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126905+0000", 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2045999999999997e-05 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237325 9.13 9:cbb3ea78:::benchmark_data_smithi043_129820_object116011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.77688265899999, 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99763348100000004, 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237325 2024-04-25T05:48:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129496+0000", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4600000000000004e-07 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129500+0000", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1790000000000001e-06 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129511+0000", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0611000000000001e-05 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129518+0000", 2024-04-25T05:48:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6020000000000002e-06 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129565+0000", 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6158000000000001e-05 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129573+0000", 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6240000000000008e-06 2024-04-25T05:48:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120395+0000", 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99082194099999998 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120502+0000", 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010669 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120678+0000", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175976 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126345+0000", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005667025 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126911+0000", 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056641199999999997 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127019+0000", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107938 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127082+0000", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2848999999999994e-05 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127129+0000", 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6730000000000002e-05 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237346 9.10 9:0cf8b6b3:::benchmark_data_smithi043_129820_object116032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 139.76509371, 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98237561100000004, 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237346 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.033 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141285+0000", 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3699999999999999e-07 2024-04-25T05:48:36.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141290+0000", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6820000000000002e-06 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141300+0000", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9890000000000004e-06 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141309+0000", 2024-04-25T05:48:36.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9840000000000007e-06 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141364+0000", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4886999999999998e-05 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141373+0000", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2369999999999998e-06 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116981+0000", 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97560839600000004 2024-04-25T05:48:36.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117101+0000", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119911 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117270+0000", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016895300000000001 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119880+0000", 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026103020000000001 2024-04-25T05:48:36.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.121458+0000", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001577422 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123574+0000", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021161970000000002 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123625+0000", 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0562000000000001e-05 2024-04-25T05:48:36.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123660+0000", 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5151999999999998e-05 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:48:36.039 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:48:36.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:36.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:36.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:36.215 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:36.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:36.360 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:36.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.498 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:36.499 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 10.8 scrub ok 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:48:36.500 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 10.8 scrub ok 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:48:36.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:36 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:48:36.581 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:36.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:36.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:36.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 10.8 scrub ok 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:48:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:36 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:48:36.766 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:36.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:36.901 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:36.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:37.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:37.034 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:37.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:37.137 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:48:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:37.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:37.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:37.374 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:48:37.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:37.449 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:37.550 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:37.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:48:37.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:37.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[27010]: 10.5 scrub ok 2024-04-25T05:48:37.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[27010]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:37.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:37.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[37492]: 10.5 scrub ok 2024-04-25T05:48:37.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[37492]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:37.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:37.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:37.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:37 smithi060 ceph-mon[30898]: 10.5 scrub ok 2024-04-25T05:48:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:37 smithi060 ceph-mon[30898]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:48:37.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:37.876 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:37.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:37.978 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:38.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:38.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-04-25T05:48:38.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:38.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:38.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:38.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:38.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:38.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:38.532 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.561 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:38.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[27010]: osdmap e425: 8 total, 8 up, 7 in 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:38.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:38.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:38 smithi043 ceph-mon[37492]: osdmap e425: 8 total, 8 up, 7 in 2024-04-25T05:48:38.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:48:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:48:38.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:38.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2003067648' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:48:38.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:38 smithi060 ceph-mon[30898]: osdmap e425: 8 total, 8 up, 7 in 2024-04-25T05:48:38.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:38.831 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:38.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:38.986 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:38.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:39.034 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:39.037 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:39.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.121 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:39.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:39.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:39.254 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:39.356 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:39.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:39.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[37492]: pgmap v773: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[27010]: pgmap v773: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:39.553 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:39.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:39.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:39.696 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:39 smithi060 ceph-mon[30898]: pgmap v773: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:48:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:39 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:39 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:39.752 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:48:39.753 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:48:39.753 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:48:39.753 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:48:39.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.797 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:39.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:39.835 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:39.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:39.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:39.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:40.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:40.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:40.199 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:40.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:40.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:40.473 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:40.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:40.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:40.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:40 smithi043 ceph-mon[27010]: osdmap e426: 8 total, 8 up, 7 in 2024-04-25T05:48:40.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:40 smithi043 ceph-mon[37492]: osdmap e426: 8 total, 8 up, 7 in 2024-04-25T05:48:40.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.740 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:40.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:40 smithi060 ceph-mon[30898]: osdmap e426: 8 total, 8 up, 7 in 2024-04-25T05:48:40.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:48:40.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:40.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:41.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:41.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:48:41.045 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:41.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:41.145 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:41.147 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:41.192 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:41.221 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:41.229 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:41.268 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:41.317 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:41.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:41.361 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:41.399 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:41.404 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:41.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:41.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:41 smithi043 ceph-mon[27010]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:48:41.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/31863311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:41.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:41 smithi043 ceph-mon[37492]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:48:41.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/31863311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:41.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:41 smithi060 ceph-mon[30898]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:48:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/31863311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:41.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:41.853 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:41.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:41.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:41.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:42.023 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:42.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:42.157 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:42.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:42.391 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:42.424 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:42.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:42.563 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:42.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:42.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:42.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:42.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:42.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:42.969 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:42.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:43.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.099 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:43.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:43.232 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:43.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:43.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:43.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:43.499 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:43.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:43.632 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:43.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:43 smithi060 ceph-mon[30898]: pgmap v776: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s 2024-04-25T05:48:43.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:43.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:43.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:43.894 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:43 smithi043 ceph-mon[27010]: pgmap v776: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s 2024-04-25T05:48:43.895 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:43 smithi043 ceph-mon[37492]: pgmap v776: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s 2024-04-25T05:48:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:43.900 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:43.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:43.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-04-25T05:48:44.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:44.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:44.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:44.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:44.345 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:44.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:44.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:44.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:44.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:44.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:44.746 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:44.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:44.886 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:44.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:44.967 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:48:44.967 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:48:44.988 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:45.027 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:45.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:48:45.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:45.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:45.291 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:45.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:45.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:45.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:45.554 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:45.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:45.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:45 smithi060 ceph-mon[30898]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-04-25T05:48:45.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.824 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:45.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:45.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:45 smithi043 ceph-mon[27010]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-04-25T05:48:45.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:45.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:45 smithi043 ceph-mon[37492]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-04-25T05:48:45.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:45.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:45.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:46.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:48:46.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:46.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:46.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:46.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:46.367 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:46.390 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:46.391 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:46.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:46.513 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:46.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:46.650 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:46.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:46.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:46.921 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:46.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:47.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:47.062 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:47.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:47.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:47.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:47.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:47.433 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:48:47.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:47.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:47.505 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:47.542 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:47.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:47.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:47.615 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:47.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:47 smithi043 ceph-mon[27010]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:47 smithi043 ceph-mon[37492]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:47.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:47.674 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:47.707 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:47.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:47.748 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:47 smithi060 ceph-mon[30898]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-04-25T05:48:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:48:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:47.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:47.785 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:47.787 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:47.787 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:47.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:47.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:47.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:47.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:47.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:48.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:48.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:48.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:48.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:48.289 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:48.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:48.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:48.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:48.559 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: osdmap e427: 8 total, 8 up, 7 in 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4104323101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: osdmap e427: 8 total, 8 up, 7 in 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4104323101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:48.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3217946858' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: osdmap e427: 8 total, 8 up, 7 in 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4104323101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:48:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:48:48.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:48.842 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:48.889 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:48.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:48.975 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:48.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:49.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:49.112 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:49.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:49.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:49.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:49.379 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:49.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:49.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:49.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:49 smithi060 ceph-mon[30898]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 596 B/s rd, 0 B/s wr, 635 op/s 2024-04-25T05:48:49.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:49.779 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:49.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:49.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:49 smithi043 ceph-mon[37492]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 596 B/s rd, 0 B/s wr, 635 op/s 2024-04-25T05:48:49.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:49 smithi043 ceph-mon[27010]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 596 B/s rd, 0 B/s wr, 635 op/s 2024-04-25T05:48:49.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:49.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:49.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:50.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:50.039 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:50.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:50.165 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:50.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.304 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:50.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:50.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:48:50.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:50.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:50.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.571 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:50.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:50.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:50.705 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:50.807 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:50.938 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:50.965 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:50.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:51.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:51.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:51.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:51.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:51.361 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:51.463 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:51.495 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:51.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:51.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:51.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:51.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:51 smithi060 ceph-mon[30898]: pgmap v781: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:51.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:51.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:51.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:51 smithi043 ceph-mon[27010]: pgmap v781: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:51.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:51 smithi043 ceph-mon[37492]: pgmap v781: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:51.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:51.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:51.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:52.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:52.168 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:52.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:52.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:52.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:52.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:52.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:52.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:52.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:52.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:52.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:52 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:52.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:52 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:52.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:52.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:52.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:52 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:52.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:52 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:52.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:52 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:52.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:52 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:52.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:52.969 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:52.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:53.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:53.104 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:53.206 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:53.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:53.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:53.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:53.473 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:53.495 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:53.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:53.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:53.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:53.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:53 smithi043 ceph-mon[27010]: pgmap v782: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:53.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:53 smithi043 ceph-mon[27010]: osdmap e428: 8 total, 8 up, 7 in 2024-04-25T05:48:53.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:53 smithi043 ceph-mon[37492]: pgmap v782: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:53.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:53 smithi043 ceph-mon[37492]: osdmap e428: 8 total, 8 up, 7 in 2024-04-25T05:48:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:53 smithi060 ceph-mon[30898]: pgmap v782: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-04-25T05:48:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:53 smithi060 ceph-mon[30898]: osdmap e428: 8 total, 8 up, 7 in 2024-04-25T05:48:53.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:48:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:48:53.761 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:53.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:53.901 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:53.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:54.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.040 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:54.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:54.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:54.175 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:54.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:48:54.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-04-25T05:48:54.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:54.308 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:54.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.455 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:54.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:54.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:54.586 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:54.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:54.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:54.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:54.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:54.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:54.873 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:54.944 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:54.952 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184730 8.c 8:3895024e:::benchmark_data_smithi043_104001_object60606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 409.05290196099998, 2024-04-25T05:48:54.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0106298600000001, 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184730 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902411+0000", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.483e-06 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902418+0000", 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3269999999999997e-06 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902437+0000", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9673999999999998e-05 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902446+0000", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902550+0000", 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104485 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902561+0000", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0351e-05 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906055+0000", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.003494664 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906151+0000", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5391999999999997e-05 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:48:54.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906246+0000", 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5552999999999998e-05 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912838+0000", 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006591675 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912925+0000", 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7159999999999999e-05 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912941+0000", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5968999999999999e-05 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912992+0000", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0841000000000003e-05 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913040+0000", 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7842e-05 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184731 8.c 8:3f348624:::benchmark_data_smithi043_104001_object60607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 409.05284787599999, 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.010906069, 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184731 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3099999999999998e-07 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902468+0000", 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.337e-06 2024-04-25T05:48:54.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902473+0000", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5310000000000001e-06 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902479+0000", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4700000000000001e-06 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902585+0000", 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106054 2024-04-25T05:48:54.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902591+0000", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3790000000000003e-06 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908540+0000", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.00594837 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908663+0000", 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123437 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908776+0000", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112724 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913122+0000", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043464560000000003 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913170+0000", 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7431e-05 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.968 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913290+0000", 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120698 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913330+0000", 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0070000000000001e-05 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:54.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913370+0000", 2024-04-25T05:48:54.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9580999999999999e-05 2024-04-25T05:48:54.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184780 8.c 8:3fdf1b59:::benchmark_data_smithi043_104001_object60656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 408.99797710799999, 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95616667600000005, 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184780 2024-04-25T05:48:54.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957336+0000", 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5799999999999999e-06 2024-04-25T05:48:54.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957344+0000", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.807e-06 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957356+0000", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2404e-05 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957365+0000", 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5040000000000002e-06 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957437+0000", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350999999999999e-05 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957448+0000", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0669999999999999e-05 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910672+0000", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95322447200000004 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:54.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910797+0000", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124442 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910926+0000", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128849 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913223+0000", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022977219999999999 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913420+0000", 2024-04-25T05:48:54.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019708099999999999 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913431+0000", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0312000000000001e-05 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913459+0000", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8577000000000001e-05 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913501+0000", 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1904999999999999e-05 2024-04-25T05:48:54.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184787 8.c 8:3320ee50:::benchmark_data_smithi043_104001_object60663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 408.99064588099998, 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95582188800000001, 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184787 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.977 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964667+0000", 2024-04-25T05:48:54.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964673+0000", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964690+0000", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7629999999999999e-05 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964699+0000", 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1390000000000001e-06 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964760+0000", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0217999999999998e-05 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964769+0000", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7860000000000008e-06 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913564+0000", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94879425100000003 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:54.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913643+0000", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9556999999999997e-05 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913717+0000", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4209999999999996e-05 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920254+0000", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065365060000000001 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920304+0000", 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9512000000000003e-05 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920385+0000", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1979999999999998e-05 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920439+0000", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3928000000000003e-05 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920488+0000", 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8301999999999999e-05 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226338 8.6 8:6585b86d:::benchmark_data_smithi043_104001_object102214:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 375.05847537099999, 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.103761714, 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226338 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896840+0000", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2289999999999999e-06 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896861+0000", 2024-04-25T05:48:54.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1603999999999998e-05 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896888+0000", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7047999999999999e-05 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896903+0000", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4888999999999999e-05 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019552+0000", 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12264891999999999 2024-04-25T05:48:54.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019561+0000", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0690000000000008e-06 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000512+0000", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98095072000000005 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000598+0000", 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6235000000000004e-05 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226349 8.6 8:6373a4df:::benchmark_data_smithi043_104001_object102225:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 375.058382623, 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.139697025, 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226349 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896930+0000", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4199999999999996e-07 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896933+0000", 2024-04-25T05:48:54.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.551e-06 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896939+0000", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1380000000000004e-06 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896945+0000", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1959999999999999e-06 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019583+0000", 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122638941 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019591+0000", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.002927+0000", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98333629700000003 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003056+0000", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129037 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:48:54.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003403+0000", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000346733 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.005083+0000", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016799779999999999 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.008504+0000", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034209710000000001 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036533+0000", 2024-04-25T05:48:54.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028029505999999999 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036586+0000", 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2278000000000003e-05 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036626+0000", 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0438000000000001e-05 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.992 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228017 8.16 8:6e8160c2:::benchmark_data_smithi043_104001_object103893:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 371.977411281, 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.134225407, 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228017 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:54.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977902+0000", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5659999999999999e-06 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977908+0000", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.592e-06 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977923+0000", 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5132e-05 2024-04-25T05:48:54.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977934+0000", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0869999999999999e-05 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978035+0000", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101284 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978042+0000", 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5229999999999998e-06 2024-04-25T05:48:54.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982071+0000", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004028472 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982081+0000", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0589e-05 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104437+0000", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.122355209 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:48:54.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104572+0000", 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013536399999999999 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112038+0000", 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074657969999999997 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112126+0000", 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8009000000000004e-05 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:54.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228023 8.16 8:6ad7e352:::benchmark_data_smithi043_104001_object103899:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 371.97735558099998, 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1340453349999999, 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228023 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:54.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977957+0000", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9500000000000003e-07 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977960+0000", 2024-04-25T05:48:55.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3230000000000002e-06 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977967+0000", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977974+0000", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9659999999999999e-06 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978175+0000", 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020133299999999999 2024-04-25T05:48:55.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting_for_map not empty", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978182+0000", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1530000000000002e-06 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982727+0000", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045456259999999997 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982737+0000", 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3440000000000007e-06 2024-04-25T05:48:55.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104931+0000", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.12219398 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105055+0000", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012434499999999999 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5,6,7", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105352+0000", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297082 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:55.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.108987+0000", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00363503 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111435+0000", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024477959999999999 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111738+0000", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030323699999999998 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111789+0000", 2024-04-25T05:48:55.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0565999999999997e-05 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111869+0000", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0865999999999995e-05 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111908+0000", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8052000000000001e-05 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112001+0000", 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3929000000000001e-05 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:55.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:55.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:55.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:48:55.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:48:55.006 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:48:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:55.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:55.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:55.140 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:55.242 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:55.292 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:55.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:55.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:55.560 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:55.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:55.599 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:55 smithi043 ceph-mon[27010]: pgmap v784: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:55.599 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:55.599 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:55 smithi043 ceph-mon[37492]: pgmap v784: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:55.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:48:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:55 smithi060 ceph-mon[30898]: pgmap v784: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:55.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:55.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:55.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:55.972 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:55.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:56.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:56.105 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:56.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.243 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:56.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:56.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:56.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:56.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:48:56.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:56.483 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:48:56.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:56.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2212197656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2212197656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.654 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2212197656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:48:56.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:56.787 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:56.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:56.915 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:56.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:57.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:57.057 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:57.158 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:57.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:57.320 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:57.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:57.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:57.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:48:57.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:48:57.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:48:57.690 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[27010]: pgmap v785: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:57.690 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:57.690 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:57.690 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[27010]: osdmap e429: 8 total, 8 up, 7 in 2024-04-25T05:48:57.691 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[37492]: pgmap v785: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:57.691 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:57.691 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:57.691 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:57 smithi043 ceph-mon[37492]: osdmap e429: 8 total, 8 up, 7 in 2024-04-25T05:48:57.719 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:48:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:57 smithi060 ceph-mon[30898]: pgmap v785: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-04-25T05:48:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:57 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:48:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:48:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:57 smithi060 ceph-mon[30898]: osdmap e429: 8 total, 8 up, 7 in 2024-04-25T05:48:57.821 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:57.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:57.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:57.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:57.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:58.037 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:48:58.074 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:48:58.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:48:58.115 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:48:58.123 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:48:58.164 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:48:58.211 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:48:58.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:58.250 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:48:58.256 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:58.301 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:48:58.343 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:48:58.345 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:48:58.345 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:48:58.345 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:48:58.345 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:48:58.346 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:48:58.346 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:48:58.346 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:48:58.346 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:48:58.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:58.366 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:48:58.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:58.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:58.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:58.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:58.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/747468748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:48:58.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:58 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:48:58.777 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:58.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:48:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:48:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:48:58.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/747468748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/747468748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:48:58.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:58 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:48:58.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:48:58.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:58.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:48:59.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:59.072 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:59.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:59.204 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:59.306 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:59.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:59.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:48:59.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:48:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:48:59.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:48:59.605 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:48:59.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: pgmap v787: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 778 op/s 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: osdmap e430: 8 total, 8 up, 7 in 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: 3.1e scrub starts 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: 3.1e scrub ok 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:48:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:48:59 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:48:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:48:59.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:48:59.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:48:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:48:59.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:48:59.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:48:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:48:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:48:59.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: pgmap v787: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 778 op/s 2024-04-25T05:48:59.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: osdmap e430: 8 total, 8 up, 7 in 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: 3.1e scrub starts 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: 3.1e scrub ok 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: pgmap v787: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 778 op/s 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: osdmap e430: 8 total, 8 up, 7 in 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: 3.1e scrub starts 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: 3.1e scrub ok 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:48:59.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:48:59 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:48:59.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:00.024 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:00.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:00.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:00.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:00.297 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:00.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:00.432 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:00.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 4.d scrub starts 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 4.d scrub ok 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 10.a scrub starts 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:49:00.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 4.d scrub starts 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 4.d scrub ok 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 10.a scrub starts 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:49:00.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:00 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:49:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:00.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:00.600 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:00.600 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:00.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.703 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:00.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 4.d scrub starts 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 4.d scrub ok 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 10.a scrub starts 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:49:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:00 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:49:00.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:00.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:00.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:00.972 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:00.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:01.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:01.107 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:01.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:01.238 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:01.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:01.368 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:01.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:01.475 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:01.506 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:01.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.643 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:01.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:01.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:01 smithi060 ceph-mon[30898]: pgmap v789: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-04-25T05:49:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:01 smithi060 ceph-mon[30898]: 10.7 scrub starts 2024-04-25T05:49:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:01 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:49:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:01 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:49:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:01 smithi060 ceph-mon[30898]: 10.a scrub ok 2024-04-25T05:49:01.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:01.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[27010]: pgmap v789: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-04-25T05:49:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[27010]: 10.7 scrub starts 2024-04-25T05:49:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:49:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:49:01.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[27010]: 10.a scrub ok 2024-04-25T05:49:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[37492]: pgmap v789: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.69k op/s 2024-04-25T05:49:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[37492]: 10.7 scrub starts 2024-04-25T05:49:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:49:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:49:01.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:01 smithi043 ceph-mon[37492]: 10.a scrub ok 2024-04-25T05:49:01.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:01.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:02.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:02.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:49:02.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:02.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:02.186 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:02.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:02.323 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:02.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:02.458 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:02.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:02.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:49:02.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:49:02.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:02 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:49:02.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:02.711 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:02.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:49:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:02 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:49:02.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:02.882 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:02.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:02.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:03.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:03.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:03.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:03.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:03.286 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:03.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:03.404 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:03.404 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:03.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:03.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:49:03.514 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:49:03.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:03.549 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:03.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:03.682 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:03.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: pgmap v790: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 910 op/s 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: 10.7 scrub ok 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: 4.2 scrub starts 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: 4.2 scrub ok 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3105119400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:03.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:03.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: pgmap v790: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 910 op/s 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: 10.7 scrub ok 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: 4.2 scrub starts 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: 4.2 scrub ok 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3105119400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: pgmap v790: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 910 op/s 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: 10.7 scrub ok 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: 4.2 scrub starts 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: 4.2 scrub ok 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3105119400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:03.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:03.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:03.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:03.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:04.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:04.087 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:04.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:04.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:04.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:04.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:04.457 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:04.498 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:04.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:49:04.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:04.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:49:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: osdmap e431: 8 total, 8 up, 7 in 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:04 smithi060 ceph-mon[30898]: pgmap v792: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:04.771 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:04.873 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: osdmap e431: 8 total, 8 up, 7 in 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:04.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[27010]: pgmap v792: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: osdmap e431: 8 total, 8 up, 7 in 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:04.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:04 smithi043 ceph-mon[37492]: pgmap v792: 158 pgs: 158 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:04.911 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:04.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:05.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:05.023 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:49:05.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:49:05.023 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-25T05:49:05.023 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.995639 2024-04-25T05:49:05.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 0.9956392339655629 2024-04-25T05:49:05.047 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:05.116 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:05.149 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:05.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:05.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:05.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:05.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:05.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:05.594 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:05.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:49:05.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[27010]: 10.d scrub starts 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[37492]: 10.d scrub starts 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:49:05.605 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:05 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:49:05.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:05.734 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:05.816 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:05.816 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:05.827 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:05.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:05.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:05.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:05.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:05 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:49:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:05 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:49:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:05 smithi060 ceph-mon[30898]: 10.d scrub starts 2024-04-25T05:49:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:05 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:49:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:05 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:49:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:06.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:06.135 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:06.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:06.266 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:06.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:06.401 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:06.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:06.543 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:06.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:06.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:06.808 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: 10.2 scrub starts 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: 10.d scrub ok 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: pgmap v793: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[27010]: 4.1a deep-scrub starts 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: 10.2 scrub starts 2024-04-25T05:49:06.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:49:06.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:49:06.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: 10.d scrub ok 2024-04-25T05:49:06.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: pgmap v793: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:06.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:06 smithi043 ceph-mon[37492]: 4.1a deep-scrub starts 2024-04-25T05:49:06.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:06.939 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:06.944 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:06.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:06.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: 10.2 scrub starts 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: 10.d scrub ok 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: pgmap v793: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-04-25T05:49:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:06 smithi060 ceph-mon[30898]: 4.1a deep-scrub starts 2024-04-25T05:49:07.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:07.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:49:07.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:07.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:07.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:07.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:07.371 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:07.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:07.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:07.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:07.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:07.783 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:07.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[27010]: 4.1a deep-scrub ok 2024-04-25T05:49:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[27010]: 10.2 scrub ok 2024-04-25T05:49:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[27010]: 5.3 deep-scrub starts 2024-04-25T05:49:07.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[27010]: 5.3 deep-scrub ok 2024-04-25T05:49:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[37492]: 4.1a deep-scrub ok 2024-04-25T05:49:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[37492]: 10.2 scrub ok 2024-04-25T05:49:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[37492]: 5.3 deep-scrub starts 2024-04-25T05:49:07.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:07 smithi043 ceph-mon[37492]: 5.3 deep-scrub ok 2024-04-25T05:49:07.917 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:07.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:07 smithi060 ceph-mon[30898]: 4.1a deep-scrub ok 2024-04-25T05:49:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:07 smithi060 ceph-mon[30898]: 10.2 scrub ok 2024-04-25T05:49:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:07 smithi060 ceph-mon[30898]: 5.3 deep-scrub starts 2024-04-25T05:49:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:07 smithi060 ceph-mon[30898]: 5.3 deep-scrub ok 2024-04-25T05:49:08.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:08.050 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:08.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:08.155 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:08.181 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:08.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:08.198 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:08.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:08.276 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:08.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:08.311 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.321 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:08.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:08.348 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:08.383 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:08.420 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:08.422 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:08.458 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 0.995639 (8652502) 2024-04-25T05:49:08.552 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:08.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:08.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:08.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:08.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:08.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[37492]: pgmap v794: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 538 B/s rd, 0 B/s wr, 737 op/s 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]: dispatch 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3881296686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:49:08.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[27010]: pgmap v794: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 538 B/s rd, 0 B/s wr, 737 op/s 2024-04-25T05:49:08.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]: dispatch 2024-04-25T05:49:08.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3881296686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:08.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:08.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:08.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:08 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:49:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:08 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:49:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:08 smithi060 ceph-mon[30898]: pgmap v794: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 538 B/s rd, 0 B/s wr, 737 op/s 2024-04-25T05:49:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]: dispatch 2024-04-25T05:49:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3881296686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:09.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:09.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:09.180 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:09.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:09.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:09.448 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:09.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:09.550 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:49:09.572 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: 1.3 deep-scrub starts 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: 1.3 deep-scrub ok 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]': finished 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: osdmap e432: 8 total, 8 up, 7 in 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/823110601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:09.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: 1.3 deep-scrub starts 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: 1.3 deep-scrub ok 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]': finished 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: osdmap e432: 8 total, 8 up, 7 in 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/823110601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:09.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:09.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:09.707 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:09.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:09.841 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:09.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:09.978 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:09.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: 1.3 deep-scrub starts 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: 1.3 deep-scrub ok 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4115820335' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9956392339655629}]': finished 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: osdmap e432: 8 total, 8 up, 7 in 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/823110601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:09.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:10.060 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:10.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:10.114 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:10.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:10.257 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:10.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:10.265 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:49:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:49:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:196431 10.e e433/432)", 2024-04-25T05:49:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:10.263548+0000", 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 5.6971000000000001e-05, 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3770999999999995e-05, 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "queued for pg", 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263548+0000", 2024-04-25T05:49:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263548+0000", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263549+0000", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5499999999999997e-07 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263559+0000", 2024-04-25T05:49:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0654e-05 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263573+0000", 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3835000000000001e-05 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.263581+0000", 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.763e-06 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:196424 10.3 10:cd70cc1d:::benchmark_data_smithi043_152558_object68074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:10.256685+0000", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0069194590000000002, 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070215859999999998, 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196424 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256685+0000", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256685+0000", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256686+0000", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5099999999999999e-07 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256690+0000", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8689999999999997e-06 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256699+0000", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5060000000000001e-06 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256707+0000", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8280000000000003e-06 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262247+0000", 2024-04-25T05:49:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005539792 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262358+0000", 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011135499999999999 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5", 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262472+0000", 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113952 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:196425 10.5 10:adc95a84:::benchmark_data_smithi043_152558_object68075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-04-25T05:49:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:10.256719+0000", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0068862940000000003, 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070723899999999996, 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196425 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256719+0000", 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256719+0000", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256719+0000", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3000000000000001e-07 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256722+0000", 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5280000000000002e-06 2024-04-25T05:49:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256728+0000", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.489e-06 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256734+0000", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6169999999999998e-06 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256769+0000", 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5796000000000001e-05 2024-04-25T05:49:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256876+0000", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106438 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256990+0000", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011427899999999999 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262308+0000", 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053180129999999999 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:196426 10.5 10:a6526655:::benchmark_data_smithi043_152558_object68076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:10.256747+0000", 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.006857658, 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071312479999999998, 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196426 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256747+0000", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256747+0000", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256748+0000", 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3399999999999999e-07 2024-04-25T05:49:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256750+0000", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7980000000000001e-06 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256756+0000", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9970000000000004e-06 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.256762+0000", 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6790000000000002e-06 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.259501+0000", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0027391049999999999 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.259624+0000", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012282100000000001 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.259737+0000", 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112552 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:196430 10.5 10:aafa5d56:::benchmark_data_smithi043_152558_object68080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:10.261617+0000", 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0019877319999999999, 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023388380000000002, 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 196430 2024-04-25T05:49:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261617+0000", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261617+0000", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261618+0000", 2024-04-25T05:49:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3510000000000001e-06 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261625+0000", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4629999999999998e-06 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261639+0000", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4037e-05 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.261647+0000", 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1480000000000006e-06 2024-04-25T05:49:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262344+0000", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00069655400000000005 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262465+0000", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000121804 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:10.262566+0000", 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000100202 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:10.285 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:49:10.285 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 5 2024-04-25T05:49:10.285 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:49:10.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:10.392 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:10.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:10.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:10.627 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:10.658 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:10.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:10.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:10.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[27010]: osdmap e433: 8 total, 8 up, 7 in 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[27010]: pgmap v797: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 791 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[37492]: osdmap e433: 8 total, 8 up, 7 in 2024-04-25T05:49:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:10 smithi043 ceph-mon[37492]: pgmap v797: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 791 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-25T05:49:10.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:10.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:10 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:10 smithi060 ceph-mon[30898]: osdmap e433: 8 total, 8 up, 7 in 2024-04-25T05:49:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:10 smithi060 ceph-mon[30898]: pgmap v797: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 791 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-25T05:49:11.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:11.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:11.191 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:11.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:11.331 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:11.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:49:11.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:11.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:11.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:11.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:11.603 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:11.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:11.736 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:11.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:11.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:11.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:11.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:12.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:12.026 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:49:12.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:12.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:12.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:12.288 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:12.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:12.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:12.490 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:12.511 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:12.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:12.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:49:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:49:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[27010]: osdmap e434: 8 total, 8 up, 7 in 2024-04-25T05:49:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[27010]: pgmap v799: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-04-25T05:49:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:49:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:49:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[37492]: osdmap e434: 8 total, 8 up, 7 in 2024-04-25T05:49:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:12 smithi043 ceph-mon[37492]: pgmap v799: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-04-25T05:49:12.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.725 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:12.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:12.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:12.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:12.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:12.994 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:12.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:12 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:49:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:12 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:49:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:12 smithi060 ceph-mon[30898]: osdmap e434: 8 total, 8 up, 7 in 2024-04-25T05:49:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:12 smithi060 ceph-mon[30898]: pgmap v799: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-04-25T05:49:13.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:13.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:13.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:49:13.227 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:13.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.263 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:13.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:13.275 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:13.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:13.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:13.397 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:13.405 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:13.444 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:13.485 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:13.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:13.524 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:13.528 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:13.529 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:13.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:13.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:13.660 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:13.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.803 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:13.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:13.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/862827160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:13.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/862827160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:13.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:13.937 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:13.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:13.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/862827160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:13.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:14.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:14.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:14.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:14.207 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:14.309 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:14.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:14.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:14.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:14.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:14.601 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:14.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:14.736 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:14.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:14.869 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:14.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:14.876 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:14 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:14.877 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:14 smithi043 ceph-mon[27010]: pgmap v800: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.86k op/s; 294 B/s, 0 objects/s recovering 2024-04-25T05:49:14.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:14 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:14.877 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:14 smithi043 ceph-mon[37492]: pgmap v800: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.86k op/s; 294 B/s, 0 objects/s recovering 2024-04-25T05:49:14.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:14 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:14 smithi060 ceph-mon[30898]: pgmap v800: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 1.86k op/s; 294 B/s, 0 objects/s recovering 2024-04-25T05:49:15.005 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:15.107 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:15.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:15.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.279 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:15.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:15.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:15.417 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:15.429 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:15.430 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:15.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:15.551 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:15.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.691 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:15.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:15.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:15.825 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:15.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:15.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:15.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:16.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:16.092 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:16.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.241 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:16.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:16.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:16.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:16.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:16.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:16.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:16.648 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:16.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:16.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:49:16.944 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:16.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:16.946 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:17.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.082 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:17.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:17.094 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:17 smithi043 ceph-mon[27010]: pgmap v801: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 963 B/s rd, 0 B/s wr, 799 op/s; 237 B/s, 0 objects/s recovering 2024-04-25T05:49:17.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:17 smithi043 ceph-mon[37492]: pgmap v801: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 963 B/s rd, 0 B/s wr, 799 op/s; 237 B/s, 0 objects/s recovering 2024-04-25T05:49:17.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:17.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:17.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.360 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:17.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:17.420 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:49:17.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:49:17.420 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-04-25T05:49:17.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:49:17.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:17.497 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:17 smithi060 ceph-mon[30898]: pgmap v801: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 963 B/s rd, 0 B/s wr, 799 op/s; 237 B/s, 0 objects/s recovering 2024-04-25T05:49:17.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:17.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:17.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:17.780 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:17.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:17.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:49:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:17.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:18.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:18.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:18.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:18.191 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:18.218 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:18.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:18.324 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:18.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:18.461 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:18.544 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:18.545 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:18.563 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:18.604 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:18.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:18.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:18.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:18.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:19.009 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[27010]: pgmap v802: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 0 B/s wr, 661 op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3863453922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[37492]: pgmap v802: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 0 B/s wr, 661 op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3863453922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.057 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:49:19.111 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:19.151 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:19.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:19.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:19.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:19.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:19 smithi060 ceph-mon[30898]: pgmap v802: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 849 B/s rd, 0 B/s wr, 661 op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:19 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3863453922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:19 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:19.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:19.561 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:19.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:19.698 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:19.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:19.821 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:19.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:19.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:19.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:19.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:20.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:20.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:20.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:20.223 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:20.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:20.357 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[27010]: osdmap e435: 8 total, 8 up, 7 in 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:20.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:20 smithi043 ceph-mon[37492]: osdmap e435: 8 total, 8 up, 7 in 2024-04-25T05:49:20.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:20.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:20 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:20 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:20 smithi060 ceph-mon[30898]: osdmap e435: 8 total, 8 up, 7 in 2024-04-25T05:49:20.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:20.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:20.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:20.758 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:20.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:20.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:20.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:20.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:21.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:21.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.150 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:21.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:21.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:21.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:21.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[27010]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[27010]: pgmap v804: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[27010]: osdmap e436: 8 total, 8 up, 7 in 2024-04-25T05:49:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[37492]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:49:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:49:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:49:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[37492]: pgmap v804: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:21.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:21 smithi043 ceph-mon[37492]: osdmap e436: 8 total, 8 up, 7 in 2024-04-25T05:49:21.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:21 smithi060 ceph-mon[30898]: from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:49:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:21 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:49:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:21 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:49:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:21 smithi060 ceph-mon[30898]: pgmap v804: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 209 B/s, 0 objects/s recovering 2024-04-25T05:49:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:21 smithi060 ceph-mon[30898]: osdmap e436: 8 total, 8 up, 7 in 2024-04-25T05:49:21.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:21.542 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:21.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:21.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:21.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:21.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:21.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:21.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:22.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:22.072 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:22.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:22.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:22.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:22.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:22.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:22.598 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:22.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:22.736 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:22.838 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:22.870 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:22.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:22.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:22.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:23.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:23.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:23.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:23.285 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:49:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:49:23.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:23 smithi043 ceph-mon[27010]: pgmap v806: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s 2024-04-25T05:49:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:23 smithi043 ceph-mon[37492]: pgmap v806: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s 2024-04-25T05:49:23.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:23 smithi060 ceph-mon[30898]: pgmap v806: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 845 op/s 2024-04-25T05:49:23.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:23.536 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:23.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:23.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:23.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:23.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:49:23.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:23.959 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:23.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:23.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:24.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:24.096 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:24.100 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:24.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:24.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:24.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:24.373 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:24.475 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:24.507 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:24.608 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:24.620 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-04-25T05:49:19.048171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3265927731,"num_objects":50056,"num_object_clones":0,"num_object_copies":150168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50056,"num_whiteouts":0,"num_read":2315,"num_read_kb":2035,"num_write":207784,"num_write_kb":8214254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":626,"num_bytes_recovered":1198290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77482,"ondisk_log_size":77482,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14809012,"kb_used_data":9868988,"kb_used_omap":10,"kb_used_meta":4939765,"kb_avail":641407052,"statfs":{"total":671965249536,"available":656800821248,"internally_reserved":0,"allocated":10105843712,"data_stored":10103095790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5058319467},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-786169856,"num_objects":-11996,"num_object_clones":0,"num_object_copies":-35988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11996,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":11996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-24,"num_bytes_recovered":1766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.071818"},"pg_stats":[{"pgid":"10.2","version":"434'12886","reported_seq":13038,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.985882+0000","last_change":"2024-04-25T05:49:06.364061+0000","last_active":"2024-04-25T05:49:18.985882+0000","last_peered":"2024-04-25T05:49:18.985882+0000","last_clean":"2024-04-25T05:49:18.985882+0000","last_became_active":"2024-04-25T05:46:37.979097+0000","last_became_peered":"2024-04-25T05:46:37.979097+0000","last_unstale":"2024-04-25T05:49:18.985882+0000","last_undegraded":"2024-04-25T05:49:18.985882+0000","last_fullsized":"2024-04-25T05:49:18.985882+0000","mapping_epoch":387,"log_start":"421'8700","ondisk_log_start":"421'8700","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'11956","last_scrub_stamp":"2024-04-25T05:49:06.363691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.363691+0000","objects_scrubbed":3943,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:29.397252+0000","scrub_duration":1074,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195559424,"num_objects":2984,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12886,"num_write_kb":507840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":275,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554658+0000","last_change":"2024-04-25T05:49:09.581196+0000","last_active":"2024-04-25T05:49:11.554658+0000","last_peered":"2024-04-25T05:49:11.554658+0000","last_clean":"2024-04-25T05:49:11.554658+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:49:11.554658+0000","last_undegraded":"2024-04-25T05:49:11.554658+0000","last_fullsized":"2024-04-25T05:49:11.554658+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:46:09.856439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.856439+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":800,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555874+0000","last_change":"2024-04-25T05:49:05.117179+0000","last_active":"2024-04-25T05:49:11.555874+0000","last_peered":"2024-04-25T05:49:11.555874+0000","last_clean":"2024-04-25T05:49:11.555874+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:49:11.555874+0000","last_undegraded":"2024-04-25T05:49:11.555874+0000","last_fullsized":"2024-04-25T05:49:11.555874+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:15.248193+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":182,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.567373+0000","last_change":"2024-04-25T05:48:22.861651+0000","last_active":"2024-04-25T05:49:11.567373+0000","last_peered":"2024-04-25T05:49:11.567373+0000","last_clean":"2024-04-25T05:49:11.567373+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:49:11.567373+0000","last_undegraded":"2024-04-25T05:49:11.567373+0000","last_fullsized":"2024-04-25T05:49:11.567373+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:48:22.861531+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:48:22.861531+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:48.381292+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":352,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552935+0000","last_change":"2024-04-25T05:49:09.561860+0000","last_active":"2024-04-25T05:49:11.552935+0000","last_peered":"2024-04-25T05:49:11.552935+0000","last_clean":"2024-04-25T05:49:11.552935+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:49:11.552935+0000","last_undegraded":"2024-04-25T05:49:11.552935+0000","last_fullsized":"2024-04-25T05:49:11.552935+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.424618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.424618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1107,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558106+0000","last_change":"2024-04-25T05:46:23.149014+0000","last_active":"2024-04-25T05:49:11.558106+0000","last_peered":"2024-04-25T05:49:11.558106+0000","last_clean":"2024-04-25T05:49:11.558106+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:49:11.558106+0000","last_undegraded":"2024-04-25T05:49:11.558106+0000","last_fullsized":"2024-04-25T05:49:11.558106+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:39.361826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:39.361826+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"434'12764","reported_seq":12942,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.270097+0000","last_change":"2024-04-25T05:48:29.753066+0000","last_active":"2024-04-25T05:49:18.270097+0000","last_peered":"2024-04-25T05:49:18.270097+0000","last_clean":"2024-04-25T05:49:18.270097+0000","last_became_active":"2024-04-25T05:46:37.983078+0000","last_became_peered":"2024-04-25T05:46:37.983078+0000","last_unstale":"2024-04-25T05:49:18.270097+0000","last_undegraded":"2024-04-25T05:49:18.270097+0000","last_fullsized":"2024-04-25T05:49:18.270097+0000","mapping_epoch":387,"log_start":"418'7900","ondisk_log_start":"418'7900","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'9235","last_scrub_stamp":"2024-04-25T05:48:29.752799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:29.752799+0000","objects_scrubbed":6603,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:33.726398+0000","scrub_duration":1897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":3008,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":284,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551687+0000","last_change":"2024-04-25T05:49:09.556929+0000","last_active":"2024-04-25T05:49:11.551687+0000","last_peered":"2024-04-25T05:49:11.551687+0000","last_clean":"2024-04-25T05:49:11.551687+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:49:11.551687+0000","last_undegraded":"2024-04-25T05:49:11.551687+0000","last_fullsized":"2024-04-25T05:49:11.551687+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:48:59.073429+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.073429+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:08.458135+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":506,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563178+0000","last_change":"2024-04-25T05:46:42.728489+0000","last_active":"2024-04-25T05:49:11.563178+0000","last_peered":"2024-04-25T05:49:11.563178+0000","last_clean":"2024-04-25T05:49:11.563178+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:49:11.563178+0000","last_undegraded":"2024-04-25T05:49:11.563178+0000","last_fullsized":"2024-04-25T05:49:11.563178+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:42.728367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:46:42.728367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560394+0000","last_change":"2024-04-25T05:48:30.448422+0000","last_active":"2024-04-25T05:49:11.560394+0000","last_peered":"2024-04-25T05:49:11.560394+0000","last_clean":"2024-04-25T05:49:11.560394+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:49:11.560394+0000","last_undegraded":"2024-04-25T05:49:11.560394+0000","last_fullsized":"2024-04-25T05:49:11.560394+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:30.448296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.448296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:40.914219+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":133,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.569740+0000","last_change":"2024-04-25T05:48:03.971319+0000","last_active":"2024-04-25T05:49:11.569740+0000","last_peered":"2024-04-25T05:49:11.569740+0000","last_clean":"2024-04-25T05:49:11.569740+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:49:11.569740+0000","last_undegraded":"2024-04-25T05:49:11.569740+0000","last_fullsized":"2024-04-25T05:49:11.569740+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.971192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.971192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.239829+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":706,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560889+0000","last_change":"2024-04-25T05:47:25.988415+0000","last_active":"2024-04-25T05:49:11.560889+0000","last_peered":"2024-04-25T05:49:11.560889+0000","last_clean":"2024-04-25T05:49:11.560889+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:49:11.560889+0000","last_undegraded":"2024-04-25T05:49:11.560889+0000","last_fullsized":"2024-04-25T05:49:11.560889+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.988268+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923715+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"434'13051","reported_seq":13295,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.421025+0000","last_change":"2024-04-25T05:48:30.978065+0000","last_active":"2024-04-25T05:49:18.421025+0000","last_peered":"2024-04-25T05:49:18.421025+0000","last_clean":"2024-04-25T05:49:18.421025+0000","last_became_active":"2024-04-25T05:46:37.984393+0000","last_became_peered":"2024-04-25T05:46:37.984393+0000","last_unstale":"2024-04-25T05:49:18.421025+0000","last_undegraded":"2024-04-25T05:49:18.421025+0000","last_fullsized":"2024-04-25T05:49:18.421025+0000","mapping_epoch":387,"log_start":"418'8100","ondisk_log_start":"418'8100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'9484","last_scrub_stamp":"2024-04-25T05:48:30.977924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.977924+0000","objects_scrubbed":6775,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:49.990981+0000","scrub_duration":2766,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202964992,"num_objects":3097,"num_object_clones":0,"num_object_copies":9291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":516736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":584,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552521+0000","last_change":"2024-04-25T05:49:09.610330+0000","last_active":"2024-04-25T05:49:11.552521+0000","last_peered":"2024-04-25T05:49:11.552521+0000","last_clean":"2024-04-25T05:49:11.552521+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:49:11.552521+0000","last_undegraded":"2024-04-25T05:49:11.552521+0000","last_fullsized":"2024-04-25T05:49:11.552521+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:47:24.775084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.775084+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":819,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552291+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:49:11.552291+0000","last_peered":"2024-04-25T05:49:11.552291+0000","last_clean":"2024-04-25T05:49:11.552291+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:49:11.552291+0000","last_undegraded":"2024-04-25T05:49:11.552291+0000","last_fullsized":"2024-04-25T05:49:11.552291+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.578308+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":109,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562584+0000","last_change":"2024-04-25T05:46:34.982029+0000","last_active":"2024-04-25T05:49:11.562584+0000","last_peered":"2024-04-25T05:49:11.562584+0000","last_clean":"2024-04-25T05:49:11.562584+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:49:11.562584+0000","last_undegraded":"2024-04-25T05:49:11.562584+0000","last_fullsized":"2024-04-25T05:49:11.562584+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:18.666459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:46:18.666459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.922914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":786,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556696+0000","last_change":"2024-04-25T05:46:51.568352+0000","last_active":"2024-04-25T05:49:11.556696+0000","last_peered":"2024-04-25T05:49:11.556696+0000","last_clean":"2024-04-25T05:49:11.556696+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:49:11.556696+0000","last_undegraded":"2024-04-25T05:49:11.556696+0000","last_fullsized":"2024-04-25T05:49:11.556696+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:51.568255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:51.568255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"434'13056","reported_seq":13166,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.017004+0000","last_change":"2024-04-25T05:49:09.598859+0000","last_active":"2024-04-25T05:49:18.017004+0000","last_peered":"2024-04-25T05:49:18.017004+0000","last_clean":"2024-04-25T05:49:18.017004+0000","last_became_active":"2024-04-25T05:49:09.552357+0000","last_became_peered":"2024-04-25T05:49:09.552357+0000","last_unstale":"2024-04-25T05:49:18.017004+0000","last_undegraded":"2024-04-25T05:49:18.017004+0000","last_fullsized":"2024-04-25T05:49:18.017004+0000","mapping_epoch":432,"log_start":"421'8924","ondisk_log_start":"421'8924","created":387,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4132,"log_dups_size":0,"ondisk_log_size":4132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.595856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13056,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":828,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559366+0000","last_change":"2024-04-25T05:47:18.459728+0000","last_active":"2024-04-25T05:49:11.559366+0000","last_peered":"2024-04-25T05:49:11.559366+0000","last_clean":"2024-04-25T05:49:11.559366+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:49:11.559366+0000","last_undegraded":"2024-04-25T05:49:11.559366+0000","last_fullsized":"2024-04-25T05:49:11.559366+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:47:18.459587+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.459587+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":783,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554494+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:49:11.554494+0000","last_peered":"2024-04-25T05:49:11.554494+0000","last_clean":"2024-04-25T05:49:11.554494+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:49:11.554494+0000","last_undegraded":"2024-04-25T05:49:11.554494+0000","last_fullsized":"2024-04-25T05:49:11.554494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"433'16","reported_seq":428,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:13.054207+0000","last_change":"2024-04-25T05:49:09.610647+0000","last_active":"2024-04-25T05:49:13.054207+0000","last_peered":"2024-04-25T05:49:13.054207+0000","last_clean":"2024-04-25T05:49:13.054207+0000","last_became_active":"2024-04-25T05:49:09.561406+0000","last_became_peered":"2024-04-25T05:49:09.561406+0000","last_unstale":"2024-04-25T05:49:13.054207+0000","last_undegraded":"2024-04-25T05:49:13.054207+0000","last_fullsized":"2024-04-25T05:49:13.054207+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-25T05:48:21.090099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.090099+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:41.017417+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561768+0000","last_change":"2024-04-25T05:48:30.199408+0000","last_active":"2024-04-25T05:49:11.561768+0000","last_peered":"2024-04-25T05:49:11.561768+0000","last_clean":"2024-04-25T05:49:11.561768+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:49:11.561768+0000","last_undegraded":"2024-04-25T05:49:11.561768+0000","last_fullsized":"2024-04-25T05:49:11.561768+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:30.199282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.199282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:35.616954+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":670,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562726+0000","last_change":"2024-04-25T05:47:51.226427+0000","last_active":"2024-04-25T05:49:11.562726+0000","last_peered":"2024-04-25T05:49:11.562726+0000","last_clean":"2024-04-25T05:49:11.562726+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:49:11.562726+0000","last_undegraded":"2024-04-25T05:49:11.562726+0000","last_fullsized":"2024-04-25T05:49:11.562726+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:51.226126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.226126+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"434'12963","reported_seq":13152,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.048156+0000","last_change":"2024-04-25T05:49:09.610983+0000","last_active":"2024-04-25T05:49:18.048156+0000","last_peered":"2024-04-25T05:49:18.048156+0000","last_clean":"2024-04-25T05:49:18.048156+0000","last_became_active":"2024-04-25T05:49:09.558908+0000","last_became_peered":"2024-04-25T05:49:09.558908+0000","last_unstale":"2024-04-25T05:49:18.048156+0000","last_undegraded":"2024-04-25T05:49:18.048156+0000","last_fullsized":"2024-04-25T05:49:18.048156+0000","mapping_epoch":432,"log_start":"422'8853","ondisk_log_start":"422'8853","created":387,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7199","last_scrub_stamp":"2024-04-25T05:47:59.467288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.467288+0000","objects_scrubbed":7112,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.599806+0000","scrub_duration":2478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12963,"num_write_kb":517312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":799,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552212+0000","last_change":"2024-04-25T05:49:03.437727+0000","last_active":"2024-04-25T05:49:11.552212+0000","last_peered":"2024-04-25T05:49:11.552212+0000","last_clean":"2024-04-25T05:49:11.552212+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:49:11.552212+0000","last_undegraded":"2024-04-25T05:49:11.552212+0000","last_fullsized":"2024-04-25T05:49:11.552212+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:03.437604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:03.437604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:24.575740+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":937,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554544+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:49:11.554544+0000","last_peered":"2024-04-25T05:49:11.554544+0000","last_clean":"2024-04-25T05:49:11.554544+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:49:11.554544+0000","last_undegraded":"2024-04-25T05:49:11.554544+0000","last_fullsized":"2024-04-25T05:49:11.554544+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":815,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561118+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:49:11.561118+0000","last_peered":"2024-04-25T05:49:11.561118+0000","last_clean":"2024-04-25T05:49:11.561118+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:49:11.561118+0000","last_undegraded":"2024-04-25T05:49:11.561118+0000","last_fullsized":"2024-04-25T05:49:11.561118+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":346,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551295+0000","last_change":"2024-04-25T05:49:09.554521+0000","last_active":"2024-04-25T05:49:11.551295+0000","last_peered":"2024-04-25T05:49:11.551295+0000","last_clean":"2024-04-25T05:49:11.551295+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:49:11.551295+0000","last_undegraded":"2024-04-25T05:49:11.551295+0000","last_fullsized":"2024-04-25T05:49:11.551295+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:17.513668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:17.513668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1028,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551070+0000","last_change":"2024-04-25T05:49:01.441992+0000","last_active":"2024-04-25T05:49:11.551070+0000","last_peered":"2024-04-25T05:49:11.551070+0000","last_clean":"2024-04-25T05:49:11.551070+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:49:11.551070+0000","last_undegraded":"2024-04-25T05:49:11.551070+0000","last_fullsized":"2024-04-25T05:49:11.551070+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:01.440957+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.440957+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:16.418607+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"434'12896","reported_seq":13159,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.404277+0000","last_change":"2024-04-25T05:48:08.175544+0000","last_active":"2024-04-25T05:49:18.404277+0000","last_peered":"2024-04-25T05:49:18.404277+0000","last_clean":"2024-04-25T05:49:18.404277+0000","last_became_active":"2024-04-25T05:46:37.982701+0000","last_became_peered":"2024-04-25T05:46:37.982701+0000","last_unstale":"2024-04-25T05:49:18.404277+0000","last_undegraded":"2024-04-25T05:49:18.404277+0000","last_fullsized":"2024-04-25T05:49:18.404277+0000","mapping_epoch":387,"log_start":"418'7900","ondisk_log_start":"418'7900","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7494","last_scrub_stamp":"2024-04-25T05:48:05.024302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.024302+0000","objects_scrubbed":7249,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2970,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202113024,"num_objects":3084,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12896,"num_write_kb":511360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":977,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554673+0000","last_change":"2024-04-25T05:48:25.100906+0000","last_active":"2024-04-25T05:49:11.554673+0000","last_peered":"2024-04-25T05:49:11.554673+0000","last_clean":"2024-04-25T05:49:11.554673+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:49:11.554673+0000","last_undegraded":"2024-04-25T05:49:11.554673+0000","last_fullsized":"2024-04-25T05:49:11.554673+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:48:25.100752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.100752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:40.039257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":476,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552552+0000","last_change":"2024-04-25T05:49:09.598448+0000","last_active":"2024-04-25T05:49:11.552552+0000","last_peered":"2024-04-25T05:49:11.552552+0000","last_clean":"2024-04-25T05:49:11.552552+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:49:11.552552+0000","last_undegraded":"2024-04-25T05:49:11.552552+0000","last_fullsized":"2024-04-25T05:49:11.552552+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:48:05.148321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.148321+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":487,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563981+0000","last_change":"2024-04-25T05:48:04.608424+0000","last_active":"2024-04-25T05:49:11.563981+0000","last_peered":"2024-04-25T05:49:11.563981+0000","last_clean":"2024-04-25T05:49:11.563981+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:49:11.563981+0000","last_undegraded":"2024-04-25T05:49:11.563981+0000","last_fullsized":"2024-04-25T05:49:11.563981+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.608284+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":165,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.570731+0000","last_change":"2024-04-25T05:47:14.571538+0000","last_active":"2024-04-25T05:49:11.570731+0000","last_peered":"2024-04-25T05:49:11.570731+0000","last_clean":"2024-04-25T05:49:11.570731+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:49:11.570731+0000","last_undegraded":"2024-04-25T05:49:11.570731+0000","last_fullsized":"2024-04-25T05:49:11.570731+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:14.571417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:14.571417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":984,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556555+0000","last_change":"2024-04-25T05:48:27.160787+0000","last_active":"2024-04-25T05:49:11.556555+0000","last_peered":"2024-04-25T05:49:11.556555+0000","last_clean":"2024-04-25T05:49:11.556555+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:49:11.556555+0000","last_undegraded":"2024-04-25T05:49:11.556555+0000","last_fullsized":"2024-04-25T05:49:11.556555+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:48:27.160639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.160639+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:41.098992+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"434'13012","reported_seq":13116,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.130245+0000","last_change":"2024-04-25T05:46:37.986037+0000","last_active":"2024-04-25T05:49:18.130245+0000","last_peered":"2024-04-25T05:49:18.130245+0000","last_clean":"2024-04-25T05:49:18.130245+0000","last_became_active":"2024-04-25T05:46:37.985670+0000","last_became_peered":"2024-04-25T05:46:37.985670+0000","last_unstale":"2024-04-25T05:49:18.130245+0000","last_undegraded":"2024-04-25T05:49:18.130245+0000","last_fullsized":"2024-04-25T05:49:18.130245+0000","mapping_epoch":387,"log_start":"420'8400","ondisk_log_start":"420'8400","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13012,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":568,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562162+0000","last_change":"2024-04-25T05:49:06.550851+0000","last_active":"2024-04-25T05:49:11.562162+0000","last_peered":"2024-04-25T05:49:11.562162+0000","last_clean":"2024-04-25T05:49:11.562162+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:49:11.562162+0000","last_undegraded":"2024-04-25T05:49:11.562162+0000","last_fullsized":"2024-04-25T05:49:11.562162+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.550555+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:13.257284+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":533,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562274+0000","last_change":"2024-04-25T05:49:02.359488+0000","last_active":"2024-04-25T05:49:11.562274+0000","last_peered":"2024-04-25T05:49:11.562274+0000","last_clean":"2024-04-25T05:49:11.562274+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:49:11.562274+0000","last_undegraded":"2024-04-25T05:49:11.562274+0000","last_fullsized":"2024-04-25T05:49:11.562274+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:49:02.359173+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.359173+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:29.047149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561863+0000","last_change":"2024-04-25T05:47:15.018494+0000","last_active":"2024-04-25T05:49:11.561863+0000","last_peered":"2024-04-25T05:49:11.561863+0000","last_clean":"2024-04-25T05:49:11.561863+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:49:11.561863+0000","last_undegraded":"2024-04-25T05:49:11.561863+0000","last_fullsized":"2024-04-25T05:49:11.561863+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:15.018379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:15.018379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":421,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554119+0000","last_change":"2024-04-25T05:46:44.669771+0000","last_active":"2024-04-25T05:49:11.554119+0000","last_peered":"2024-04-25T05:49:11.554119+0000","last_clean":"2024-04-25T05:49:11.554119+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:49:11.554119+0000","last_undegraded":"2024-04-25T05:49:11.554119+0000","last_fullsized":"2024-04-25T05:49:11.554119+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.669615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.669615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":969,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563364+0000","last_change":"2024-04-25T05:47:19.766253+0000","last_active":"2024-04-25T05:49:11.563364+0000","last_peered":"2024-04-25T05:49:11.563364+0000","last_clean":"2024-04-25T05:49:11.563364+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:49:11.563364+0000","last_undegraded":"2024-04-25T05:49:11.563364+0000","last_fullsized":"2024-04-25T05:49:11.563364+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:47:19.766122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.766122+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"434'12718","reported_seq":12873,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.369495+0000","last_change":"2024-04-25T05:49:05.755086+0000","last_active":"2024-04-25T05:49:18.369495+0000","last_peered":"2024-04-25T05:49:18.369495+0000","last_clean":"2024-04-25T05:49:18.369495+0000","last_became_active":"2024-04-25T05:46:37.986694+0000","last_became_peered":"2024-04-25T05:46:37.986694+0000","last_unstale":"2024-04-25T05:49:18.369495+0000","last_undegraded":"2024-04-25T05:49:18.369495+0000","last_fullsized":"2024-04-25T05:49:18.369495+0000","mapping_epoch":387,"log_start":"418'7700","ondisk_log_start":"418'7700","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'11797","last_scrub_stamp":"2024-04-25T05:49:05.754660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.754660+0000","objects_scrubbed":4046,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:18.733089+0000","scrub_duration":1317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047512,"num_objects":3084,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12718,"num_write_kb":505601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":544,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563267+0000","last_change":"2024-04-25T05:47:51.904861+0000","last_active":"2024-04-25T05:49:11.563267+0000","last_peered":"2024-04-25T05:49:11.563267+0000","last_clean":"2024-04-25T05:49:11.563267+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:49:11.563267+0000","last_undegraded":"2024-04-25T05:49:11.563267+0000","last_fullsized":"2024-04-25T05:49:11.563267+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:51.904568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.904568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.642706+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":163,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561624+0000","last_change":"2024-04-25T05:48:59.056819+0000","last_active":"2024-04-25T05:49:11.561624+0000","last_peered":"2024-04-25T05:49:11.561624+0000","last_clean":"2024-04-25T05:49:11.561624+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:49:11.561624+0000","last_undegraded":"2024-04-25T05:49:11.561624+0000","last_fullsized":"2024-04-25T05:49:11.561624+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:48:59.056469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.056469+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:28.500757+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":821,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560174+0000","last_change":"2024-04-25T05:47:53.584048+0000","last_active":"2024-04-25T05:49:11.560174+0000","last_peered":"2024-04-25T05:49:11.560174+0000","last_clean":"2024-04-25T05:49:11.560174+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:49:11.560174+0000","last_undegraded":"2024-04-25T05:49:11.560174+0000","last_fullsized":"2024-04-25T05:49:11.560174+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:53.583930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:53.583930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:18.954029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":795,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563150+0000","last_change":"2024-04-25T05:46:47.388550+0000","last_active":"2024-04-25T05:49:11.563150+0000","last_peered":"2024-04-25T05:49:11.563150+0000","last_clean":"2024-04-25T05:49:11.563150+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:49:11.563150+0000","last_undegraded":"2024-04-25T05:49:11.563150+0000","last_fullsized":"2024-04-25T05:49:11.563150+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.388440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.388440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":987,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562034+0000","last_change":"2024-04-25T05:46:52.649991+0000","last_active":"2024-04-25T05:49:11.562034+0000","last_peered":"2024-04-25T05:49:11.562034+0000","last_clean":"2024-04-25T05:49:11.562034+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:49:11.562034+0000","last_undegraded":"2024-04-25T05:49:11.562034+0000","last_fullsized":"2024-04-25T05:49:11.562034+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:52.649850+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.649850+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"434'12902","reported_seq":13077,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.248531+0000","last_change":"2024-04-25T05:48:34.766599+0000","last_active":"2024-04-25T05:49:18.248531+0000","last_peered":"2024-04-25T05:49:18.248531+0000","last_clean":"2024-04-25T05:49:18.248531+0000","last_became_active":"2024-04-25T05:46:37.986157+0000","last_became_peered":"2024-04-25T05:46:37.986157+0000","last_unstale":"2024-04-25T05:49:18.248531+0000","last_undegraded":"2024-04-25T05:49:18.248531+0000","last_fullsized":"2024-04-25T05:49:18.248531+0000","mapping_epoch":387,"log_start":"418'8000","ondisk_log_start":"418'8000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'9679","last_scrub_stamp":"2024-04-25T05:48:34.766411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:34.766411+0000","objects_scrubbed":6382,"log_size":4902,"log_dups_size":0,"ondisk_log_size":4902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:38.665928+0000","scrub_duration":1894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12902,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562680+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:49:11.562680+0000","last_peered":"2024-04-25T05:49:11.562680+0000","last_clean":"2024-04-25T05:49:11.562680+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:49:11.562680+0000","last_undegraded":"2024-04-25T05:49:11.562680+0000","last_fullsized":"2024-04-25T05:49:11.562680+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:16.696528+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":131,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561910+0000","last_change":"2024-04-25T05:47:55.201717+0000","last_active":"2024-04-25T05:49:11.561910+0000","last_peered":"2024-04-25T05:49:11.561910+0000","last_clean":"2024-04-25T05:49:11.561910+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:49:11.561910+0000","last_undegraded":"2024-04-25T05:49:11.561910+0000","last_fullsized":"2024-04-25T05:49:11.561910+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:55.201430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.201430+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923827+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1743,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:16.777371+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:49:16.777371+0000","last_peered":"2024-04-25T05:49:16.777371+0000","last_clean":"2024-04-25T05:49:16.777371+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:49:16.777371+0000","last_undegraded":"2024-04-25T05:49:16.777371+0000","last_fullsized":"2024-04-25T05:49:16.777371+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":901,"num_read_kb":896,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":424,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552367+0000","last_change":"2024-04-25T05:49:05.488606+0000","last_active":"2024-04-25T05:49:11.552367+0000","last_peered":"2024-04-25T05:49:11.552367+0000","last_clean":"2024-04-25T05:49:11.552367+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:49:11.552367+0000","last_undegraded":"2024-04-25T05:49:11.552367+0000","last_fullsized":"2024-04-25T05:49:11.552367+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.488514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.488514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:21.686478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":791,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559925+0000","last_change":"2024-04-25T05:48:35.483025+0000","last_active":"2024-04-25T05:49:11.559925+0000","last_peered":"2024-04-25T05:49:11.559925+0000","last_clean":"2024-04-25T05:49:11.559925+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:49:11.559925+0000","last_undegraded":"2024-04-25T05:49:11.559925+0000","last_fullsized":"2024-04-25T05:49:11.559925+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.482886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.482886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:58.849222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":973,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.570626+0000","last_change":"2024-04-25T05:49:08.317583+0000","last_active":"2024-04-25T05:49:11.570626+0000","last_peered":"2024-04-25T05:49:11.570626+0000","last_clean":"2024-04-25T05:49:11.570626+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:49:11.570626+0000","last_undegraded":"2024-04-25T05:49:11.570626+0000","last_fullsized":"2024-04-25T05:49:11.570626+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_clean_scrub_stamp":"2024-04-25T05:49:08.317433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:25.174757+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"434'13029","reported_seq":13214,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.418676+0000","last_change":"2024-04-25T05:48:01.589216+0000","last_active":"2024-04-25T05:49:18.418676+0000","last_peered":"2024-04-25T05:49:18.418676+0000","last_clean":"2024-04-25T05:49:18.418676+0000","last_became_active":"2024-04-25T05:46:37.985141+0000","last_became_peered":"2024-04-25T05:46:37.985141+0000","last_unstale":"2024-04-25T05:49:18.418676+0000","last_undegraded":"2024-04-25T05:49:18.418676+0000","last_fullsized":"2024-04-25T05:49:18.418676+0000","mapping_epoch":387,"log_start":"418'8100","ondisk_log_start":"418'8100","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'7377","last_scrub_stamp":"2024-04-25T05:48:01.588877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.588877+0000","objects_scrubbed":7283,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203489280,"num_objects":3105,"num_object_clones":0,"num_object_copies":9315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":813,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557887+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:49:11.557887+0000","last_peered":"2024-04-25T05:49:11.557887+0000","last_clean":"2024-04-25T05:49:11.557887+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:49:11.557887+0000","last_undegraded":"2024-04-25T05:49:11.557887+0000","last_fullsized":"2024-04-25T05:49:11.557887+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"320'31","reported_seq":659,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562582+0000","last_change":"2024-04-25T05:47:29.876796+0000","last_active":"2024-04-25T05:49:11.562582+0000","last_peered":"2024-04-25T05:49:11.562582+0000","last_clean":"2024-04-25T05:49:11.562582+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:49:11.562582+0000","last_undegraded":"2024-04-25T05:49:11.562582+0000","last_fullsized":"2024-04-25T05:49:11.562582+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:47:29.876498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:29.876498+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:17.687938+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":807,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554119+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:49:11.554119+0000","last_peered":"2024-04-25T05:49:11.554119+0000","last_clean":"2024-04-25T05:49:11.554119+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:49:11.554119+0000","last_undegraded":"2024-04-25T05:49:11.554119+0000","last_fullsized":"2024-04-25T05:49:11.554119+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:28.893663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1100,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560209+0000","last_change":"2024-04-25T05:46:06.483237+0000","last_active":"2024-04-25T05:49:11.560209+0000","last_peered":"2024-04-25T05:49:11.560209+0000","last_clean":"2024-04-25T05:49:11.560209+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:49:11.560209+0000","last_undegraded":"2024-04-25T05:49:11.560209+0000","last_fullsized":"2024-04-25T05:49:11.560209+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:46:06.483108+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:46:06.483108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":90,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559491+0000","last_change":"2024-04-25T05:48:33.740566+0000","last_active":"2024-04-25T05:49:11.559491+0000","last_peered":"2024-04-25T05:49:11.559491+0000","last_clean":"2024-04-25T05:49:11.559491+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:49:11.559491+0000","last_undegraded":"2024-04-25T05:49:11.559491+0000","last_fullsized":"2024-04-25T05:49:11.559491+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:33.740445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:48:33.740445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:39.576467+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"434'12889","reported_seq":12992,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.057373+0000","last_change":"2024-04-25T05:46:37.984020+0000","last_active":"2024-04-25T05:49:18.057373+0000","last_peered":"2024-04-25T05:49:18.057373+0000","last_clean":"2024-04-25T05:49:18.057373+0000","last_became_active":"2024-04-25T05:46:37.983253+0000","last_became_peered":"2024-04-25T05:46:37.983253+0000","last_unstale":"2024-04-25T05:49:18.057373+0000","last_undegraded":"2024-04-25T05:49:18.057373+0000","last_fullsized":"2024-04-25T05:49:18.057373+0000","mapping_epoch":387,"log_start":"420'8200","ondisk_log_start":"420'8200","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:21.908309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203227136,"num_objects":3101,"num_object_clones":0,"num_object_copies":9303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12889,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":147,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.567671+0000","last_change":"2024-04-25T05:49:04.051632+0000","last_active":"2024-04-25T05:49:11.567671+0000","last_peered":"2024-04-25T05:49:11.567671+0000","last_clean":"2024-04-25T05:49:11.567671+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:49:11.567671+0000","last_undegraded":"2024-04-25T05:49:11.567671+0000","last_fullsized":"2024-04-25T05:49:11.567671+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:04.051391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:04.051391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:30.903757+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":930,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559318+0000","last_change":"2024-04-25T05:48:02.601745+0000","last_active":"2024-04-25T05:49:11.559318+0000","last_peered":"2024-04-25T05:49:11.559318+0000","last_clean":"2024-04-25T05:49:11.559318+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:49:11.559318+0000","last_undegraded":"2024-04-25T05:49:11.559318+0000","last_fullsized":"2024-04-25T05:49:11.559318+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:48:02.601574+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.601574+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:27.802514+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":141,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.569038+0000","last_change":"2024-04-25T05:48:31.849230+0000","last_active":"2024-04-25T05:49:11.569038+0000","last_peered":"2024-04-25T05:49:11.569038+0000","last_clean":"2024-04-25T05:49:11.569038+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:49:11.569038+0000","last_undegraded":"2024-04-25T05:49:11.569038+0000","last_fullsized":"2024-04-25T05:49:11.569038+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:31.849085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:31.849085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:59.100350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":827,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563484+0000","last_change":"2024-04-25T05:48:21.556337+0000","last_active":"2024-04-25T05:49:11.563484+0000","last_peered":"2024-04-25T05:49:11.563484+0000","last_clean":"2024-04-25T05:49:11.563484+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:49:11.563484+0000","last_undegraded":"2024-04-25T05:49:11.563484+0000","last_fullsized":"2024-04-25T05:49:11.563484+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:21.556096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.556096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:30.754267+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":791,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559664+0000","last_change":"2024-04-25T05:47:23.888942+0000","last_active":"2024-04-25T05:49:11.559664+0000","last_peered":"2024-04-25T05:49:11.559664+0000","last_clean":"2024-04-25T05:49:11.559664+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:49:11.559664+0000","last_undegraded":"2024-04-25T05:49:11.559664+0000","last_fullsized":"2024-04-25T05:49:11.559664+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:23.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1108,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556431+0000","last_change":"2024-04-25T05:46:23.137608+0000","last_active":"2024-04-25T05:49:11.556431+0000","last_peered":"2024-04-25T05:49:11.556431+0000","last_clean":"2024-04-25T05:49:11.556431+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:49:11.556431+0000","last_undegraded":"2024-04-25T05:49:11.556431+0000","last_fullsized":"2024-04-25T05:49:11.556431+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:45:30.250119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:45:30.250119+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"434'15219","reported_seq":15480,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.383711+0000","last_change":"2024-04-25T05:49:01.091877+0000","last_active":"2024-04-25T05:49:18.383711+0000","last_peered":"2024-04-25T05:49:18.383711+0000","last_clean":"2024-04-25T05:49:18.383711+0000","last_became_active":"2024-04-25T05:46:37.985031+0000","last_became_peered":"2024-04-25T05:46:37.985031+0000","last_unstale":"2024-04-25T05:49:18.383711+0000","last_undegraded":"2024-04-25T05:49:18.383711+0000","last_fullsized":"2024-04-25T05:49:18.383711+0000","mapping_epoch":387,"log_start":"424'10300","ondisk_log_start":"424'10300","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'13913","last_scrub_stamp":"2024-04-25T05:49:01.091383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.091383+0000","objects_scrubbed":4639,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:16.749327+0000","scrub_duration":1561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214564864,"num_objects":3274,"num_object_clones":0,"num_object_copies":9822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12824,"num_write_kb":515136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":416,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559121+0000","last_change":"2024-04-25T05:48:29.428977+0000","last_active":"2024-04-25T05:49:11.559121+0000","last_peered":"2024-04-25T05:49:11.559121+0000","last_clean":"2024-04-25T05:49:11.559121+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:49:11.559121+0000","last_undegraded":"2024-04-25T05:49:11.559121+0000","last_fullsized":"2024-04-25T05:49:11.559121+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:29.428764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:29.428764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:52.627959+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1113,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:13.919478+0000","last_change":"2024-04-25T05:46:52.415241+0000","last_active":"2024-04-25T05:49:13.919478+0000","last_peered":"2024-04-25T05:49:13.919478+0000","last_clean":"2024-04-25T05:49:13.919478+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:49:13.919478+0000","last_undegraded":"2024-04-25T05:49:13.919478+0000","last_fullsized":"2024-04-25T05:49:13.919478+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:46:52.415149+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.415149+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":359,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.553971+0000","last_change":"2024-04-25T05:49:09.560837+0000","last_active":"2024-04-25T05:49:11.553971+0000","last_peered":"2024-04-25T05:49:11.553971+0000","last_clean":"2024-04-25T05:49:11.553971+0000","last_became_active":"2024-04-25T05:49:09.559684+0000","last_became_peered":"2024-04-25T05:49:09.559684+0000","last_unstale":"2024-04-25T05:49:11.553971+0000","last_undegraded":"2024-04-25T05:49:11.553971+0000","last_fullsized":"2024-04-25T05:49:11.553971+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:37.254738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:48:37.254738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:42.250432+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":121,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562612+0000","last_change":"2024-04-25T05:46:49.868732+0000","last_active":"2024-04-25T05:49:11.562612+0000","last_peered":"2024-04-25T05:49:11.562612+0000","last_clean":"2024-04-25T05:49:11.562612+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:49:11.562612+0000","last_undegraded":"2024-04-25T05:49:11.562612+0000","last_fullsized":"2024-04-25T05:49:11.562612+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:46:49.868648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.868648+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":763,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555956+0000","last_change":"2024-04-25T05:46:16.748067+0000","last_active":"2024-04-25T05:49:11.555956+0000","last_peered":"2024-04-25T05:49:11.555956+0000","last_clean":"2024-04-25T05:49:11.555956+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:49:11.555956+0000","last_undegraded":"2024-04-25T05:49:11.555956+0000","last_fullsized":"2024-04-25T05:49:11.555956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:16.747963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:16.747963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":817,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560925+0000","last_change":"2024-04-25T05:49:00.009223+0000","last_active":"2024-04-25T05:49:11.560925+0000","last_peered":"2024-04-25T05:49:11.560925+0000","last_clean":"2024-04-25T05:49:11.560925+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:49:11.560925+0000","last_undegraded":"2024-04-25T05:49:11.560925+0000","last_fullsized":"2024-04-25T05:49:11.560925+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:00.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:00.008984+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:13.211612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"434'12934","reported_seq":13108,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.285125+0000","last_change":"2024-04-25T05:48:35.602553+0000","last_active":"2024-04-25T05:49:18.285125+0000","last_peered":"2024-04-25T05:49:18.285125+0000","last_clean":"2024-04-25T05:49:18.285125+0000","last_became_active":"2024-04-25T05:46:37.986037+0000","last_became_peered":"2024-04-25T05:46:37.986037+0000","last_unstale":"2024-04-25T05:49:18.285125+0000","last_undegraded":"2024-04-25T05:49:18.285125+0000","last_fullsized":"2024-04-25T05:49:18.285125+0000","mapping_epoch":387,"log_start":"418'8000","ondisk_log_start":"418'8000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'9701","last_scrub_stamp":"2024-04-25T05:48:35.602412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.602412+0000","objects_scrubbed":6251,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:58.878338+0000","scrub_duration":1701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194117632,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12934,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":501,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.570426+0000","last_change":"2024-04-25T05:47:23.436651+0000","last_active":"2024-04-25T05:49:11.570426+0000","last_peered":"2024-04-25T05:49:11.570426+0000","last_clean":"2024-04-25T05:49:11.570426+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:49:11.570426+0000","last_undegraded":"2024-04-25T05:49:11.570426+0000","last_fullsized":"2024-04-25T05:49:11.570426+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:23.436515+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.436515+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560900+0000","last_change":"2024-04-25T05:48:20.054569+0000","last_active":"2024-04-25T05:49:11.560900+0000","last_peered":"2024-04-25T05:49:11.560900+0000","last_clean":"2024-04-25T05:49:11.560900+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:49:11.560900+0000","last_undegraded":"2024-04-25T05:49:11.560900+0000","last_fullsized":"2024-04-25T05:49:11.560900+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:20.054446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:48:20.054446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:24.472298+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":133,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.568900+0000","last_change":"2024-04-25T05:48:21.841752+0000","last_active":"2024-04-25T05:49:11.568900+0000","last_peered":"2024-04-25T05:49:11.568900+0000","last_clean":"2024-04-25T05:49:11.568900+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:49:11.568900+0000","last_undegraded":"2024-04-25T05:49:11.568900+0000","last_fullsized":"2024-04-25T05:49:11.568900+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:21.841634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.841634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:46.106468+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":408,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558004+0000","last_change":"2024-04-25T05:47:19.913147+0000","last_active":"2024-04-25T05:49:11.558004+0000","last_peered":"2024-04-25T05:49:11.558004+0000","last_clean":"2024-04-25T05:49:11.558004+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:49:11.558004+0000","last_undegraded":"2024-04-25T05:49:11.558004+0000","last_fullsized":"2024-04-25T05:49:11.558004+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.913041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.913041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"434'12969","reported_seq":13080,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.055824+0000","last_change":"2024-04-25T05:49:09.580853+0000","last_active":"2024-04-25T05:49:18.055824+0000","last_peered":"2024-04-25T05:49:18.055824+0000","last_clean":"2024-04-25T05:49:18.055824+0000","last_became_active":"2024-04-25T05:49:09.552219+0000","last_became_peered":"2024-04-25T05:49:09.552219+0000","last_unstale":"2024-04-25T05:49:18.055824+0000","last_undegraded":"2024-04-25T05:49:18.055824+0000","last_fullsized":"2024-04-25T05:49:18.055824+0000","mapping_epoch":432,"log_start":"422'8900","ondisk_log_start":"422'8900","created":387,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.595884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":9597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12969,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"320'41","reported_seq":560,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563080+0000","last_change":"2024-04-25T05:47:31.914632+0000","last_active":"2024-04-25T05:49:11.563080+0000","last_peered":"2024-04-25T05:49:11.563080+0000","last_clean":"2024-04-25T05:49:11.563080+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:49:11.563080+0000","last_undegraded":"2024-04-25T05:49:11.563080+0000","last_fullsized":"2024-04-25T05:49:11.563080+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'41","last_scrub_stamp":"2024-04-25T05:47:31.914155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.914155+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"67'1","reported_seq":836,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559374+0000","last_change":"2024-04-25T05:46:45.682570+0000","last_active":"2024-04-25T05:49:11.559374+0000","last_peered":"2024-04-25T05:49:11.559374+0000","last_clean":"2024-04-25T05:49:11.559374+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:49:11.559374+0000","last_undegraded":"2024-04-25T05:49:11.559374+0000","last_fullsized":"2024-04-25T05:49:11.559374+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:46:45.682441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:45.682441+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":139,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.569164+0000","last_change":"2024-04-25T05:47:59.817680+0000","last_active":"2024-04-25T05:49:11.569164+0000","last_peered":"2024-04-25T05:49:11.569164+0000","last_clean":"2024-04-25T05:49:11.569164+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:49:11.569164+0000","last_undegraded":"2024-04-25T05:49:11.569164+0000","last_fullsized":"2024-04-25T05:49:11.569164+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.817559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.817559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:17.472338+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":791,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557435+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:49:11.557435+0000","last_peered":"2024-04-25T05:49:11.557435+0000","last_clean":"2024-04-25T05:49:11.557435+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:49:11.557435+0000","last_undegraded":"2024-04-25T05:49:11.557435+0000","last_fullsized":"2024-04-25T05:49:11.557435+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.889858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"434'12962","reported_seq":13120,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.421119+0000","last_change":"2024-04-25T05:49:02.199009+0000","last_active":"2024-04-25T05:49:18.421119+0000","last_peered":"2024-04-25T05:49:18.421119+0000","last_clean":"2024-04-25T05:49:18.421119+0000","last_became_active":"2024-04-25T05:46:37.983489+0000","last_became_peered":"2024-04-25T05:46:37.983489+0000","last_unstale":"2024-04-25T05:49:18.421119+0000","last_undegraded":"2024-04-25T05:49:18.421119+0000","last_fullsized":"2024-04-25T05:49:18.421119+0000","mapping_epoch":387,"log_start":"418'8000","ondisk_log_start":"418'8000","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'11749","last_scrub_stamp":"2024-04-25T05:49:02.198849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.198849+0000","objects_scrubbed":4359,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:27.549889+0000","scrub_duration":1683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12962,"num_write_kb":513536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":147,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.567999+0000","last_change":"2024-04-25T05:49:07.005458+0000","last_active":"2024-04-25T05:49:11.567999+0000","last_peered":"2024-04-25T05:49:11.567999+0000","last_clean":"2024-04-25T05:49:11.567999+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:49:11.567999+0000","last_undegraded":"2024-04-25T05:49:11.567999+0000","last_fullsized":"2024-04-25T05:49:11.567999+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:07.004816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:07.004816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:28.721551+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":206,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552683+0000","last_change":"2024-04-25T05:49:09.598004+0000","last_active":"2024-04-25T05:49:11.552683+0000","last_peered":"2024-04-25T05:49:11.552683+0000","last_clean":"2024-04-25T05:49:11.552683+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:49:11.552683+0000","last_undegraded":"2024-04-25T05:49:11.552683+0000","last_fullsized":"2024-04-25T05:49:11.552683+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:48:24.506969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.506969+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:54.359049+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":790,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558027+0000","last_change":"2024-04-25T05:46:46.685480+0000","last_active":"2024-04-25T05:49:11.558027+0000","last_peered":"2024-04-25T05:49:11.558027+0000","last_clean":"2024-04-25T05:49:11.558027+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:49:11.558027+0000","last_undegraded":"2024-04-25T05:49:11.558027+0000","last_fullsized":"2024-04-25T05:49:11.558027+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.685365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.685365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":774,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556385+0000","last_change":"2024-04-25T05:46:52.779088+0000","last_active":"2024-04-25T05:49:11.556385+0000","last_peered":"2024-04-25T05:49:11.556385+0000","last_clean":"2024-04-25T05:49:11.556385+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:49:11.556385+0000","last_undegraded":"2024-04-25T05:49:11.556385+0000","last_fullsized":"2024-04-25T05:49:11.556385+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:52.779013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:52.779013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"434'13015","reported_seq":13118,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:18.050175+0000","last_change":"2024-04-25T05:46:37.981240+0000","last_active":"2024-04-25T05:49:18.050175+0000","last_peered":"2024-04-25T05:49:18.050175+0000","last_clean":"2024-04-25T05:49:18.050175+0000","last_became_active":"2024-04-25T05:46:37.980913+0000","last_became_peered":"2024-04-25T05:46:37.980913+0000","last_unstale":"2024-04-25T05:49:18.050175+0000","last_undegraded":"2024-04-25T05:49:18.050175+0000","last_fullsized":"2024-04-25T05:49:18.050175+0000","mapping_epoch":387,"log_start":"420'8300","ondisk_log_start":"420'8300","created":387,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:46:36.966533+0000","last_clean_scrub_stamp":"2024-04-25T05:46:36.966533+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:21.907458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206110720,"num_objects":3145,"num_object_clones":0,"num_object_copies":9435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13015,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":999,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:13.141196+0000","last_change":"2024-04-25T05:46:47.674917+0000","last_active":"2024-04-25T05:49:13.141196+0000","last_peered":"2024-04-25T05:49:13.141196+0000","last_clean":"2024-04-25T05:49:13.141196+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:49:13.141196+0000","last_undegraded":"2024-04-25T05:49:13.141196+0000","last_fullsized":"2024-04-25T05:49:13.141196+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:46:47.674789+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.674789+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:16.812823+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":137,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560995+0000","last_change":"2024-04-25T05:47:23.979227+0000","last_active":"2024-04-25T05:49:11.560995+0000","last_peered":"2024-04-25T05:49:11.560995+0000","last_clean":"2024-04-25T05:49:11.560995+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:49:11.560995+0000","last_undegraded":"2024-04-25T05:49:11.560995+0000","last_fullsized":"2024-04-25T05:49:11.560995+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:47:23.979075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.979075+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923797+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":838,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558640+0000","last_change":"2024-04-25T05:48:59.765915+0000","last_active":"2024-04-25T05:49:11.558640+0000","last_peered":"2024-04-25T05:49:11.558640+0000","last_clean":"2024-04-25T05:49:11.558640+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:49:11.558640+0000","last_undegraded":"2024-04-25T05:49:11.558640+0000","last_fullsized":"2024-04-25T05:49:11.558640+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:48:59.765781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.765781+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:07.601502+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562636+0000","last_change":"2024-04-25T05:49:01.017521+0000","last_active":"2024-04-25T05:49:11.562636+0000","last_peered":"2024-04-25T05:49:11.562636+0000","last_clean":"2024-04-25T05:49:11.562636+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:49:11.562636+0000","last_undegraded":"2024-04-25T05:49:11.562636+0000","last_fullsized":"2024-04-25T05:49:11.562636+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:01.017406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.017406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:21.739720+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":808,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561406+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:49:11.561406+0000","last_peered":"2024-04-25T05:49:11.561406+0000","last_clean":"2024-04-25T05:49:11.561406+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:49:11.561406+0000","last_undegraded":"2024-04-25T05:49:11.561406+0000","last_fullsized":"2024-04-25T05:49:11.561406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":544,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561632+0000","last_change":"2024-04-25T05:48:35.181448+0000","last_active":"2024-04-25T05:49:11.561632+0000","last_peered":"2024-04-25T05:49:11.561632+0000","last_clean":"2024-04-25T05:49:11.561632+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:49:11.561632+0000","last_undegraded":"2024-04-25T05:49:11.561632+0000","last_fullsized":"2024-04-25T05:49:11.561632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.181334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.181334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:44.349931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561816+0000","last_change":"2024-04-25T05:47:56.163151+0000","last_active":"2024-04-25T05:49:11.561816+0000","last_peered":"2024-04-25T05:49:11.561816+0000","last_clean":"2024-04-25T05:49:11.561816+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:49:11.561816+0000","last_undegraded":"2024-04-25T05:49:11.561816+0000","last_fullsized":"2024-04-25T05:49:11.561816+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:56.163019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.163019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":798,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559096+0000","last_change":"2024-04-25T05:48:57.925411+0000","last_active":"2024-04-25T05:49:11.559096+0000","last_peered":"2024-04-25T05:49:11.559096+0000","last_clean":"2024-04-25T05:49:11.559096+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:49:11.559096+0000","last_undegraded":"2024-04-25T05:49:11.559096+0000","last_fullsized":"2024-04-25T05:49:11.559096+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:57.925301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.925301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:15.791534+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":488,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555876+0000","last_change":"2024-04-25T05:46:20.806968+0000","last_active":"2024-04-25T05:49:11.555876+0000","last_peered":"2024-04-25T05:49:11.555876+0000","last_clean":"2024-04-25T05:49:11.555876+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:49:11.555876+0000","last_undegraded":"2024-04-25T05:49:11.555876+0000","last_fullsized":"2024-04-25T05:49:11.555876+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:46:20.806869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:20.806869+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":808,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557734+0000","last_change":"2024-04-25T05:47:31.715369+0000","last_active":"2024-04-25T05:49:11.557734+0000","last_peered":"2024-04-25T05:49:11.557734+0000","last_clean":"2024-04-25T05:49:11.557734+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:49:11.557734+0000","last_undegraded":"2024-04-25T05:49:11.557734+0000","last_fullsized":"2024-04-25T05:49:11.557734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.715261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":766,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561251+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:49:11.561251+0000","last_peered":"2024-04-25T05:49:11.561251+0000","last_clean":"2024-04-25T05:49:11.561251+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:49:11.561251+0000","last_undegraded":"2024-04-25T05:49:11.561251+0000","last_fullsized":"2024-04-25T05:49:11.561251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":133,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.566633+0000","last_change":"2024-04-25T05:48:25.842658+0000","last_active":"2024-04-25T05:49:11.566633+0000","last_peered":"2024-04-25T05:49:11.566633+0000","last_clean":"2024-04-25T05:49:11.566633+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:49:11.566633+0000","last_undegraded":"2024-04-25T05:49:11.566633+0000","last_fullsized":"2024-04-25T05:49:11.566633+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:25.842542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.842542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:50.362682+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":129,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561459+0000","last_change":"2024-04-25T05:48:26.172633+0000","last_active":"2024-04-25T05:49:11.561459+0000","last_peered":"2024-04-25T05:49:11.561459+0000","last_clean":"2024-04-25T05:49:11.561459+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:49:11.561459+0000","last_undegraded":"2024-04-25T05:49:11.561459+0000","last_fullsized":"2024-04-25T05:49:11.561459+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:48:26.172484+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:48:26.172484+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:55.206877+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":402,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557259+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:49:11.557259+0000","last_peered":"2024-04-25T05:49:11.557259+0000","last_clean":"2024-04-25T05:49:11.557259+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:49:11.557259+0000","last_undegraded":"2024-04-25T05:49:11.557259+0000","last_fullsized":"2024-04-25T05:49:11.557259+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:24.171124+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":978,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557225+0000","last_change":"2024-04-25T05:47:18.482166+0000","last_active":"2024-04-25T05:49:11.557225+0000","last_peered":"2024-04-25T05:49:11.557225+0000","last_clean":"2024-04-25T05:49:11.557225+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:49:11.557225+0000","last_undegraded":"2024-04-25T05:49:11.557225+0000","last_fullsized":"2024-04-25T05:49:11.557225+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:47:18.482041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.482041+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":329,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551709+0000","last_change":"2024-04-25T05:49:09.557095+0000","last_active":"2024-04-25T05:49:11.551709+0000","last_peered":"2024-04-25T05:49:11.551709+0000","last_clean":"2024-04-25T05:49:11.551709+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:49:11.551709+0000","last_undegraded":"2024-04-25T05:49:11.551709+0000","last_fullsized":"2024-04-25T05:49:11.551709+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.987808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.987808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":832,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556780+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:49:11.556780+0000","last_peered":"2024-04-25T05:49:11.556780+0000","last_clean":"2024-04-25T05:49:11.556780+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:49:11.556780+0000","last_undegraded":"2024-04-25T05:49:11.556780+0000","last_fullsized":"2024-04-25T05:49:11.556780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":802,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556763+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:49:11.556763+0000","last_peered":"2024-04-25T05:49:11.556763+0000","last_clean":"2024-04-25T05:49:11.556763+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:49:11.556763+0000","last_undegraded":"2024-04-25T05:49:11.556763+0000","last_fullsized":"2024-04-25T05:49:11.556763+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:23.690045+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":153,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561714+0000","last_change":"2024-04-25T05:49:01.967445+0000","last_active":"2024-04-25T05:49:11.561714+0000","last_peered":"2024-04-25T05:49:11.561714+0000","last_clean":"2024-04-25T05:49:11.561714+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:49:11.561714+0000","last_undegraded":"2024-04-25T05:49:11.561714+0000","last_fullsized":"2024-04-25T05:49:11.561714+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'13","last_scrub_stamp":"2024-04-25T05:49:01.967348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.967348+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:17.338531+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":402,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557866+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:49:11.557866+0000","last_peered":"2024-04-25T05:49:11.557866+0000","last_clean":"2024-04-25T05:49:11.557866+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:49:11.557866+0000","last_undegraded":"2024-04-25T05:49:11.557866+0000","last_fullsized":"2024-04-25T05:49:11.557866+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":928,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554476+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:49:11.554476+0000","last_peered":"2024-04-25T05:49:11.554476+0000","last_clean":"2024-04-25T05:49:11.554476+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:49:11.554476+0000","last_undegraded":"2024-04-25T05:49:11.554476+0000","last_fullsized":"2024-04-25T05:49:11.554476+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.466214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":375,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554495+0000","last_change":"2024-04-25T05:49:09.581474+0000","last_active":"2024-04-25T05:49:11.554495+0000","last_peered":"2024-04-25T05:49:11.554495+0000","last_clean":"2024-04-25T05:49:11.554495+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:49:11.554495+0000","last_undegraded":"2024-04-25T05:49:11.554495+0000","last_fullsized":"2024-04-25T05:49:11.554495+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:45:33.294829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:45:33.294829+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":789,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562127+0000","last_change":"2024-04-25T05:47:30.502986+0000","last_active":"2024-04-25T05:49:11.562127+0000","last_peered":"2024-04-25T05:49:11.562127+0000","last_clean":"2024-04-25T05:49:11.562127+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:49:11.562127+0000","last_undegraded":"2024-04-25T05:49:11.562127+0000","last_fullsized":"2024-04-25T05:49:11.562127+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:30.502873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.502873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554836+0000","last_change":"2024-04-25T05:46:53.519364+0000","last_active":"2024-04-25T05:49:11.554836+0000","last_peered":"2024-04-25T05:49:11.554836+0000","last_clean":"2024-04-25T05:49:11.554836+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:49:11.554836+0000","last_undegraded":"2024-04-25T05:49:11.554836+0000","last_fullsized":"2024-04-25T05:49:11.554836+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.519223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.519223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":344,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562761+0000","last_change":"2024-04-25T05:48:57.339166+0000","last_active":"2024-04-25T05:49:11.562761+0000","last_peered":"2024-04-25T05:49:11.562761+0000","last_clean":"2024-04-25T05:49:11.562761+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:49:11.562761+0000","last_undegraded":"2024-04-25T05:49:11.562761+0000","last_fullsized":"2024-04-25T05:49:11.562761+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:48:57.338784+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.338784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:00.659125+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":790,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559185+0000","last_change":"2024-04-25T05:46:44.682768+0000","last_active":"2024-04-25T05:49:11.559185+0000","last_peered":"2024-04-25T05:49:11.559185+0000","last_clean":"2024-04-25T05:49:11.559185+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:49:11.559185+0000","last_undegraded":"2024-04-25T05:49:11.559185+0000","last_fullsized":"2024-04-25T05:49:11.559185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.682645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.682645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":790,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555976+0000","last_change":"2024-04-25T05:47:25.459410+0000","last_active":"2024-04-25T05:49:11.555976+0000","last_peered":"2024-04-25T05:49:11.555976+0000","last_clean":"2024-04-25T05:49:11.555976+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:49:11.555976+0000","last_undegraded":"2024-04-25T05:49:11.555976+0000","last_fullsized":"2024-04-25T05:49:11.555976+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:25.459222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:25.459222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560784+0000","last_change":"2024-04-25T05:46:53.839614+0000","last_active":"2024-04-25T05:49:11.560784+0000","last_peered":"2024-04-25T05:49:11.560784+0000","last_clean":"2024-04-25T05:49:11.560784+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:49:11.560784+0000","last_undegraded":"2024-04-25T05:49:11.560784+0000","last_fullsized":"2024-04-25T05:49:11.560784+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:53.839498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:53.839498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":867,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555119+0000","last_change":"2024-04-25T05:47:30.478629+0000","last_active":"2024-04-25T05:49:11.555119+0000","last_peered":"2024-04-25T05:49:11.555119+0000","last_clean":"2024-04-25T05:49:11.555119+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:49:11.555119+0000","last_undegraded":"2024-04-25T05:49:11.555119+0000","last_fullsized":"2024-04-25T05:49:11.555119+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:47:30.478462+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561650+0000","last_change":"2024-04-25T05:47:26.965285+0000","last_active":"2024-04-25T05:49:11.561650+0000","last_peered":"2024-04-25T05:49:11.561650+0000","last_clean":"2024-04-25T05:49:11.561650+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:49:11.561650+0000","last_undegraded":"2024-04-25T05:49:11.561650+0000","last_fullsized":"2024-04-25T05:49:11.561650+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:26.965135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:26.965135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923684+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":800,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560467+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:49:11.560467+0000","last_peered":"2024-04-25T05:49:11.560467+0000","last_clean":"2024-04-25T05:49:11.560467+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:49:11.560467+0000","last_undegraded":"2024-04-25T05:49:11.560467+0000","last_fullsized":"2024-04-25T05:49:11.560467+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":798,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556492+0000","last_change":"2024-04-25T05:46:19.986880+0000","last_active":"2024-04-25T05:49:11.556492+0000","last_peered":"2024-04-25T05:49:11.556492+0000","last_clean":"2024-04-25T05:49:11.556492+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:49:11.556492+0000","last_undegraded":"2024-04-25T05:49:11.556492+0000","last_fullsized":"2024-04-25T05:49:11.556492+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:17.841621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:46:17.841621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":814,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554348+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:49:11.554348+0000","last_peered":"2024-04-25T05:49:11.554348+0000","last_clean":"2024-04-25T05:49:11.554348+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:49:11.554348+0000","last_undegraded":"2024-04-25T05:49:11.554348+0000","last_fullsized":"2024-04-25T05:49:11.554348+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:43.306072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562434+0000","last_change":"2024-04-25T05:47:32.942934+0000","last_active":"2024-04-25T05:49:11.562434+0000","last_peered":"2024-04-25T05:49:11.562434+0000","last_clean":"2024-04-25T05:49:11.562434+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:49:11.562434+0000","last_undegraded":"2024-04-25T05:49:11.562434+0000","last_fullsized":"2024-04-25T05:49:11.562434+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:32.942795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.942795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923741+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":506,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561363+0000","last_change":"2024-04-25T05:47:28.867384+0000","last_active":"2024-04-25T05:49:11.561363+0000","last_peered":"2024-04-25T05:49:11.561363+0000","last_clean":"2024-04-25T05:49:11.561363+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:49:11.561363+0000","last_undegraded":"2024-04-25T05:49:11.561363+0000","last_fullsized":"2024-04-25T05:49:11.561363+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:28.867273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:47:28.867273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":791,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556522+0000","last_change":"2024-04-25T05:46:20.027611+0000","last_active":"2024-04-25T05:49:11.556522+0000","last_peered":"2024-04-25T05:49:11.556522+0000","last_clean":"2024-04-25T05:49:11.556522+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:49:11.556522+0000","last_undegraded":"2024-04-25T05:49:11.556522+0000","last_fullsized":"2024-04-25T05:49:11.556522+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:44:19.960488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:44:19.960488+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":618,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563472+0000","last_change":"2024-04-25T05:46:44.699964+0000","last_active":"2024-04-25T05:49:11.563472+0000","last_peered":"2024-04-25T05:49:11.563472+0000","last_clean":"2024-04-25T05:49:11.563472+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:49:11.563472+0000","last_undegraded":"2024-04-25T05:49:11.563472+0000","last_fullsized":"2024-04-25T05:49:11.563472+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:46:44.699808+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.699808+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554247+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:49:11.554247+0000","last_peered":"2024-04-25T05:49:11.554247+0000","last_clean":"2024-04-25T05:49:11.554247+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:49:11.554247+0000","last_undegraded":"2024-04-25T05:49:11.554247+0000","last_fullsized":"2024-04-25T05:49:11.554247+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":400,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558145+0000","last_change":"2024-04-25T05:46:19.988527+0000","last_active":"2024-04-25T05:49:11.558145+0000","last_peered":"2024-04-25T05:49:11.558145+0000","last_clean":"2024-04-25T05:49:11.558145+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:49:11.558145+0000","last_undegraded":"2024-04-25T05:49:11.558145+0000","last_fullsized":"2024-04-25T05:49:11.558145+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:44:14.724323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:44:14.724323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":788,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558826+0000","last_change":"2024-04-25T05:47:59.671965+0000","last_active":"2024-04-25T05:49:11.558826+0000","last_peered":"2024-04-25T05:49:11.558826+0000","last_clean":"2024-04-25T05:49:11.558826+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:49:11.558826+0000","last_undegraded":"2024-04-25T05:49:11.558826+0000","last_fullsized":"2024-04-25T05:49:11.558826+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:59.671833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:59.671833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":475,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.558294+0000","last_change":"2024-04-25T05:47:21.817524+0000","last_active":"2024-04-25T05:49:11.558294+0000","last_peered":"2024-04-25T05:49:11.558294+0000","last_clean":"2024-04-25T05:49:11.558294+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:49:11.558294+0000","last_undegraded":"2024-04-25T05:49:11.558294+0000","last_fullsized":"2024-04-25T05:49:11.558294+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:47:21.817370+0000","last_deep_scrub":"191'22","last_deep_scrub_stamp":"2024-04-25T05:39:02.184563+0000","last_clean_scrub_stamp":"2024-04-25T05:47:21.817370+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":791,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562834+0000","last_change":"2024-04-25T05:48:27.483676+0000","last_active":"2024-04-25T05:49:11.562834+0000","last_peered":"2024-04-25T05:49:11.562834+0000","last_clean":"2024-04-25T05:49:11.562834+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:49:11.562834+0000","last_undegraded":"2024-04-25T05:49:11.562834+0000","last_fullsized":"2024-04-25T05:49:11.562834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:48:27.483463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.483463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:36.657646+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":635,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563374+0000","last_change":"2024-04-25T05:47:56.016055+0000","last_active":"2024-04-25T05:49:11.563374+0000","last_peered":"2024-04-25T05:49:11.563374+0000","last_clean":"2024-04-25T05:49:11.563374+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:49:11.563374+0000","last_undegraded":"2024-04-25T05:49:11.563374+0000","last_fullsized":"2024-04-25T05:49:11.563374+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:47:56.015913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:56.015913+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:17.690591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":819,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.557776+0000","last_change":"2024-04-25T05:46:34.991626+0000","last_active":"2024-04-25T05:49:11.557776+0000","last_peered":"2024-04-25T05:49:11.557776+0000","last_clean":"2024-04-25T05:49:11.557776+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:49:11.557776+0000","last_undegraded":"2024-04-25T05:49:11.557776+0000","last_fullsized":"2024-04-25T05:49:11.557776+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:46:09.493093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:09.493093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":814,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.560218+0000","last_change":"2024-04-25T05:47:16.525921+0000","last_active":"2024-04-25T05:49:11.560218+0000","last_peered":"2024-04-25T05:49:11.560218+0000","last_clean":"2024-04-25T05:49:11.560218+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:49:11.560218+0000","last_undegraded":"2024-04-25T05:49:11.560218+0000","last_fullsized":"2024-04-25T05:49:11.560218+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:16.525560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:16.525560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":824,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551557+0000","last_change":"2024-04-25T05:48:00.830523+0000","last_active":"2024-04-25T05:49:11.551557+0000","last_peered":"2024-04-25T05:49:11.551557+0000","last_clean":"2024-04-25T05:49:11.551557+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:49:11.551557+0000","last_undegraded":"2024-04-25T05:49:11.551557+0000","last_fullsized":"2024-04-25T05:49:11.551557+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.830408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:12.175161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":347,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554749+0000","last_change":"2024-04-25T05:49:09.551217+0000","last_active":"2024-04-25T05:49:11.554749+0000","last_peered":"2024-04-25T05:49:11.554749+0000","last_clean":"2024-04-25T05:49:11.554749+0000","last_became_active":"2024-04-25T05:49:09.550767+0000","last_became_peered":"2024-04-25T05:49:09.550767+0000","last_unstale":"2024-04-25T05:49:11.554749+0000","last_undegraded":"2024-04-25T05:49:11.554749+0000","last_fullsized":"2024-04-25T05:49:11.554749+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:47.610387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:46:47.610387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":787,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561326+0000","last_change":"2024-04-25T05:46:49.931810+0000","last_active":"2024-04-25T05:49:11.561326+0000","last_peered":"2024-04-25T05:49:11.561326+0000","last_clean":"2024-04-25T05:49:11.561326+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:49:11.561326+0000","last_undegraded":"2024-04-25T05:49:11.561326+0000","last_fullsized":"2024-04-25T05:49:11.561326+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:49.930039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:46:49.930039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":157,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.566468+0000","last_change":"2024-04-25T05:49:01.024022+0000","last_active":"2024-04-25T05:49:11.566468+0000","last_peered":"2024-04-25T05:49:11.566468+0000","last_clean":"2024-04-25T05:49:11.566468+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:49:11.566468+0000","last_undegraded":"2024-04-25T05:49:11.566468+0000","last_fullsized":"2024-04-25T05:49:11.566468+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:49:01.023885+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.023885+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:03.611416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":147,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554284+0000","last_change":"2024-04-25T05:49:09.557107+0000","last_active":"2024-04-25T05:49:11.554284+0000","last_peered":"2024-04-25T05:49:11.554284+0000","last_clean":"2024-04-25T05:49:11.554284+0000","last_became_active":"2024-04-25T05:49:09.556671+0000","last_became_peered":"2024-04-25T05:49:09.556671+0000","last_unstale":"2024-04-25T05:49:11.554284+0000","last_undegraded":"2024-04-25T05:49:11.554284+0000","last_fullsized":"2024-04-25T05:49:11.554284+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:26.128164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.559041+0000","last_change":"2024-04-25T05:48:22.532473+0000","last_active":"2024-04-25T05:49:11.559041+0000","last_peered":"2024-04-25T05:49:11.559041+0000","last_clean":"2024-04-25T05:49:11.559041+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:49:11.559041+0000","last_undegraded":"2024-04-25T05:49:11.559041+0000","last_fullsized":"2024-04-25T05:49:11.559041+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:22.532394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:22.532394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:50.672244+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":819,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.552636+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:49:11.552636+0000","last_peered":"2024-04-25T05:49:11.552636+0000","last_clean":"2024-04-25T05:49:11.552636+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:49:11.552636+0000","last_undegraded":"2024-04-25T05:49:11.552636+0000","last_fullsized":"2024-04-25T05:49:11.552636+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:20.067934+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":298,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.554375+0000","last_change":"2024-04-25T05:49:09.554222+0000","last_active":"2024-04-25T05:49:11.554375+0000","last_peered":"2024-04-25T05:49:11.554375+0000","last_clean":"2024-04-25T05:49:11.554375+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:49:11.554375+0000","last_undegraded":"2024-04-25T05:49:11.554375+0000","last_fullsized":"2024-04-25T05:49:11.554375+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:48:36.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:36.108290+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:04.148243+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561009+0000","last_change":"2024-04-25T05:48:04.097751+0000","last_active":"2024-04-25T05:49:11.561009+0000","last_peered":"2024-04-25T05:49:11.561009+0000","last_clean":"2024-04-25T05:49:11.561009+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:49:11.561009+0000","last_undegraded":"2024-04-25T05:49:11.561009+0000","last_fullsized":"2024-04-25T05:49:11.561009+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:04.097613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:06.788358+0000","last_clean_scrub_stamp":"2024-04-25T05:48:04.097613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:13.875974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":780,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556336+0000","last_change":"2024-04-25T05:45:36.201509+0000","last_active":"2024-04-25T05:49:11.556336+0000","last_peered":"2024-04-25T05:49:11.556336+0000","last_clean":"2024-04-25T05:49:11.556336+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:49:11.556336+0000","last_undegraded":"2024-04-25T05:49:11.556336+0000","last_fullsized":"2024-04-25T05:49:11.556336+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:45:31.300040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:45:31.300040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561031+0000","last_change":"2024-04-25T05:48:31.213742+0000","last_active":"2024-04-25T05:49:11.561031+0000","last_peered":"2024-04-25T05:49:11.561031+0000","last_clean":"2024-04-25T05:49:11.561031+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:49:11.561031+0000","last_undegraded":"2024-04-25T05:49:11.561031+0000","last_fullsized":"2024-04-25T05:49:11.561031+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:31.213611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:31.213611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:56.661593+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":592,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551446+0000","last_change":"2024-04-25T05:49:09.555965+0000","last_active":"2024-04-25T05:49:11.551446+0000","last_peered":"2024-04-25T05:49:11.551446+0000","last_clean":"2024-04-25T05:49:11.551446+0000","last_became_active":"2024-04-25T05:49:09.554840+0000","last_became_peered":"2024-04-25T05:49:09.554840+0000","last_unstale":"2024-04-25T05:49:11.551446+0000","last_undegraded":"2024-04-25T05:49:11.551446+0000","last_fullsized":"2024-04-25T05:49:11.551446+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.169828+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:34.768656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561566+0000","last_change":"2024-04-25T05:48:00.061365+0000","last_active":"2024-04-25T05:49:11.561566+0000","last_peered":"2024-04-25T05:49:11.561566+0000","last_clean":"2024-04-25T05:49:11.561566+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:49:11.561566+0000","last_undegraded":"2024-04-25T05:49:11.561566+0000","last_fullsized":"2024-04-25T05:49:11.561566+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.061240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.061240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923881+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.561510+0000","last_change":"2024-04-25T05:48:24.173416+0000","last_active":"2024-04-25T05:49:11.561510+0000","last_peered":"2024-04-25T05:49:11.561510+0000","last_clean":"2024-04-25T05:49:11.561510+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:49:11.561510+0000","last_undegraded":"2024-04-25T05:49:11.561510+0000","last_fullsized":"2024-04-25T05:49:11.561510+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:24.173135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.173135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:48.991352+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":408,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.564564+0000","last_change":"2024-04-25T05:47:13.375334+0000","last_active":"2024-04-25T05:49:11.564564+0000","last_peered":"2024-04-25T05:49:11.564564+0000","last_clean":"2024-04-25T05:49:11.564564+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:49:11.564564+0000","last_undegraded":"2024-04-25T05:49:11.564564+0000","last_fullsized":"2024-04-25T05:49:11.564564+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:13.375091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:13.375091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1071,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555207+0000","last_change":"2024-04-25T05:46:48.610335+0000","last_active":"2024-04-25T05:49:11.555207+0000","last_peered":"2024-04-25T05:49:11.555207+0000","last_clean":"2024-04-25T05:49:11.555207+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:49:11.555207+0000","last_undegraded":"2024-04-25T05:49:11.555207+0000","last_fullsized":"2024-04-25T05:49:11.555207+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:46:48.610157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:46:48.610157+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":815,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.555671+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:49:11.555671+0000","last_peered":"2024-04-25T05:49:11.555671+0000","last_clean":"2024-04-25T05:49:11.555671+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:49:11.555671+0000","last_undegraded":"2024-04-25T05:49:11.555671+0000","last_fullsized":"2024-04-25T05:49:11.555671+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:50:25.401154+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":345,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551780+0000","last_change":"2024-04-25T05:49:09.556592+0000","last_active":"2024-04-25T05:49:11.551780+0000","last_peered":"2024-04-25T05:49:11.551780+0000","last_clean":"2024-04-25T05:49:11.551780+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:49:11.551780+0000","last_undegraded":"2024-04-25T05:49:11.551780+0000","last_fullsized":"2024-04-25T05:49:11.551780+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:18.657935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:18.657935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":774,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.556073+0000","last_change":"2024-04-25T05:46:46.830806+0000","last_active":"2024-04-25T05:49:11.556073+0000","last_peered":"2024-04-25T05:49:11.556073+0000","last_clean":"2024-04-25T05:49:11.556073+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:49:11.556073+0000","last_undegraded":"2024-04-25T05:49:11.556073+0000","last_fullsized":"2024-04-25T05:49:11.556073+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:46.830676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:46:46.830676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":880,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.550815+0000","last_change":"2024-04-25T05:47:55.921938+0000","last_active":"2024-04-25T05:49:11.550815+0000","last_peered":"2024-04-25T05:49:11.550815+0000","last_clean":"2024-04-25T05:49:11.550815+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:49:11.550815+0000","last_undegraded":"2024-04-25T05:49:11.550815+0000","last_fullsized":"2024-04-25T05:49:11.550815+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:47:55.921801+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.921801+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":117,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.562243+0000","last_change":"2024-04-25T05:46:44.971898+0000","last_active":"2024-04-25T05:49:11.562243+0000","last_peered":"2024-04-25T05:49:11.562243+0000","last_clean":"2024-04-25T05:49:11.562243+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:49:11.562243+0000","last_undegraded":"2024-04-25T05:49:11.562243+0000","last_fullsized":"2024-04-25T05:49:11.562243+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:46:44.971755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:46:44.971755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:14.923600+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":800,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.563050+0000","last_change":"2024-04-25T05:48:23.471843+0000","last_active":"2024-04-25T05:49:11.563050+0000","last_peered":"2024-04-25T05:49:11.563050+0000","last_clean":"2024-04-25T05:49:11.563050+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:49:11.563050+0000","last_undegraded":"2024-04-25T05:49:11.563050+0000","last_fullsized":"2024-04-25T05:49:11.563050+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:23.471693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.471693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:48.772628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":329,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-04-25T05:49:11.551195+0000","last_change":"2024-04-25T05:49:09.559432+0000","last_active":"2024-04-25T05:49:11.551195+0000","last_peered":"2024-04-25T05:49:11.551195+0000","last_clean":"2024-04-25T05:49:11.551195+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:49:11.551195+0000","last_undegraded":"2024-04-25T05:49:11.551195+0000","last_fullsized":"2024-04-25T05:49:11.551195+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.227001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.227001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:49:46.294800+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3265331224,"num_objects":49826,"num_object_clones":0,"num_object_copies":149478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206870,"num_write_kb":8214209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9971445760,"data_stored":9971433544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74988,"ondisk_log_size":74988,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":901,"num_read_kb":896,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":10594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":970,"ondisk_log_size":970,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070653,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729136,"kb_used_data":1003816,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92016016,"statfs":{"total":95995035648,"available":94224400384,"internally_reserved":0,"allocated":1027907584,"data_stored":1027502757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364993,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2075592,"kb_used_data":1426624,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91669560,"statfs":{"total":95995035648,"available":93869629440,"internally_reserved":0,"allocated":1460862976,"data_stored":1460497549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086908,"kb_used_data":1203252,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91658244,"statfs":{"total":95995035648,"available":93858041856,"internally_reserved":0,"allocated":1232130048,"data_stored":1231711577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411541,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981852,"kb_used_data":1405012,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91763300,"statfs":{"total":95995035648,"available":93965619200,"internally_reserved":0,"allocated":1438732288,"data_stored":1438346403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149902,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2776944,"kb_used_data":1785064,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":90968208,"statfs":{"total":95995035648,"available":93151444992,"internally_reserved":0,"allocated":1827905536,"data_stored":1827499654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1015675335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986378,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2457252,"kb_used_data":1834652,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91287900,"statfs":{"total":95995035648,"available":93478809600,"internally_reserved":0,"allocated":1878683648,"data_stored":1878289731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692038,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701328,"kb_used_data":1210568,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92043824,"statfs":{"total":95995035648,"available":94252875776,"internally_reserved":0,"allocated":1239621632,"data_stored":1239248119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1207173120,"data_stored":1207173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1424490496,"data_stored":1424490496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1792806912,"data_stored":1792802840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1859653632,"data_stored":1859649560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1232273408,"data_stored":1232273408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1434587136,"data_stored":1434583064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1020461056,"data_stored":1020461056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:49:24.626 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-25T05:49:24.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-04-25T05:49:24.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:49:24.650 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:24.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:24.793 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:24.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:24.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:24.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:25.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:25.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:25.203 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:25 smithi043 ceph-mon[27010]: pgmap v807: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:49:25.203 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:25.204 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:25 smithi043 ceph-mon[37492]: pgmap v807: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:49:25.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:25.217 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:25.306 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:25.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:25.348 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:25.375 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:25.385 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:25.421 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:25.460 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:25.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:25.496 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:25 smithi060 ceph-mon[30898]: pgmap v807: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-04-25T05:49:25.515 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:25.584 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:25.594 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:25.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:25.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:25.766 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:25.803 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:25.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:25.937 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:25.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:26.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:216295 10.3 e436/387 10:cbd85814:::benchmark_data_smithi043_152558_object87945:head v 436'13357, mlcod=436'13355)", 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:25.924825+0000", 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0061630870000000003, 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0061962780000000004, 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:49:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924825+0000", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924825+0000", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924826+0000", 2024-04-25T05:49:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3300000000000001e-07 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924834+0000", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8820000000000001e-06 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924850+0000", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5999999999999999e-05 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.924869+0000", 2024-04-25T05:49:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8978e-05 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.927346+0000", 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024767610000000001 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.927368+0000", 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1973000000000001e-05 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.15975.0:216296 10.2 e436/387 10:41a0915a:::benchmark_data_smithi043_152558_object87946:head v 436'13398, mlcod=436'13395)", 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:25.929026+0000", 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0019623280000000002, 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0020726149999999999, 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929026+0000", 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929026+0000", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929027+0000", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.068e-06 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929034+0000", 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9809999999999997e-06 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929049+0000", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4908e-05 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929057+0000", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0630000000000006e-06 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929143+0000", 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5556999999999994e-05 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:49:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:25.929162+0000", 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9219999999999999e-05 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 2 2024-04-25T05:49:26.076 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:49:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:26.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:26.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:26.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:26.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:26.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3956803755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3956803755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:26.393 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:26.393 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:26.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:26.472 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3956803755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:26.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:26.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:26.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:26.746 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:26.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:26.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:26.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:26.979 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:27.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:27.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:27.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:27.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:27.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:27.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:27 smithi043 ceph-mon[27010]: pgmap v808: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-04-25T05:49:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:27 smithi043 ceph-mon[37492]: pgmap v808: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-04-25T05:49:27.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:27 smithi060 ceph-mon[30898]: pgmap v808: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-04-25T05:49:27.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.547 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:27.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:27.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:27.656 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:49:27.680 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:27.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:27.818 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:27.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:27.957 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:27.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:28.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:28.088 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:28.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:28.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:28.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:28.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:28.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:28.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:28.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:28.636 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:28.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:28.866 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:28.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:28.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:29.005 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:29.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:29.050 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:49:29.140 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:29.148 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:49:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.172 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:29.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:29 smithi043 ceph-mon[27010]: pgmap v809: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 673 op/s 2024-04-25T05:49:29.207 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:49:29.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:29 smithi043 ceph-mon[37492]: pgmap v809: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 673 op/s 2024-04-25T05:49:29.207 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:49:29.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:29.311 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:29.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.455 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:29.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:29 smithi060 ceph-mon[30898]: pgmap v809: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 572 B/s rd, 0 B/s wr, 673 op/s 2024-04-25T05:49:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:49:29.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:29.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:29.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:29.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:29.836 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:29.875 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:29.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:29.876 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:29.976 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:30.014 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:30.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:30.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: osdmap e437: 8 total, 8 up, 7 in 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3944048139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: osdmap e437: 8 total, 8 up, 7 in 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3944048139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:30 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 10 pg_num to 26 2024-04-25T05:49:30.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:30.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:30.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:30.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:30.456 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3661859612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: osdmap e437: 8 total, 8 up, 7 in 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3944048139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:30 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-04-25T05:49:30.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:30.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:30.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:30.746 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:30.848 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:30.883 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:30.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:30.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:31.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:31.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:31.181 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:31.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: pgmap v811: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: osdmap e438: 8 total, 8 up, 7 in 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:49:31.308 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: pgmap v811: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-04-25T05:49:31.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:49:31.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: osdmap e438: 8 total, 8 up, 7 in 2024-04-25T05:49:31.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:49:31.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:49:31.309 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:31.397 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:31.436 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: pgmap v811: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: osdmap e438: 8 total, 8 up, 7 in 2024-04-25T05:49:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:49:31.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:49:31.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:31.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:31.559 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:31.660 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:31.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:31.830 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:31.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:31.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:32.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:32.099 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:32.127 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:32.174 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:32.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:32.226 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4287333519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:32.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4287333519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:32.244 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.268 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:32.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:32.302 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:32.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:32.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:32.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:49:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:32.436 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:32.456 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:32.456 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:32.459 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:32.480 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:32.482 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:32 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:49:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:32 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:49:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4287333519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:32.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:32.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:32.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:32.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:32.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:32.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:32.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:33.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:33.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:33.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.170 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:33.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.300 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: pgmap v813: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 4.16 scrub starts 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 4.16 scrub ok 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: osdmap e439: 8 total, 8 up, 7 in 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:49:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: pgmap v813: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 4.16 scrub starts 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 4.16 scrub ok 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: osdmap e439: 8 total, 8 up, 7 in 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:49:33.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:33 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:49:33.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:49:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:49:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:49:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: pgmap v813: 158 pgs: 158 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-04-25T05:49:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 4.16 scrub starts 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 4.16 scrub ok 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: osdmap e439: 8 total, 8 up, 7 in 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:49:33.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:49:33.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:33.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:49:33.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:33.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:33.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:33.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:33.846 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:33.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:33.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:33.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:33.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:34.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:34.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:34.184 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:34.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:34.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: osdmap e440: 8 total, 8 up, 7 in 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 10.1 scrub starts 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 10.7 scrub starts 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: 10.1 scrub ok 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:49:34.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: osdmap e440: 8 total, 8 up, 7 in 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 10.1 scrub starts 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 10.7 scrub starts 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: 10.1 scrub ok 2024-04-25T05:49:34.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:34.355 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:34.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:34.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: osdmap e440: 8 total, 8 up, 7 in 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 10.1 scrub starts 2024-04-25T05:49:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 10.7 scrub starts 2024-04-25T05:49:34.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: 10.1 scrub ok 2024-04-25T05:49:34.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-04-25T05:49:34.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:34.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:34.624 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:34.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:34.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:34.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:34.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:34.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:35.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:35.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:35.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:35.239 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:35.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.363 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:35.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:35.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:35.487 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.9 scrub starts 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: pgmap v816: 168 pgs: 7 peering, 8 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.2 scrub starts 2024-04-25T05:49:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.9 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: osdmap e441: 8 total, 8 up, 7 in 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.7 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.8 scrub starts 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.2 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.8 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.4 scrub starts 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.4 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.0 scrub starts 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 10.0 scrub ok 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: osdmap e442: 8 total, 8 up, 7 in 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:35.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 6.6 scrub starts 2024-04-25T05:49:35.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:35 smithi060 ceph-mon[30898]: 6.6 scrub ok 2024-04-25T05:49:35.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:35.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:49:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:49:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.9 scrub starts 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: pgmap v816: 168 pgs: 7 peering, 8 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.2 scrub starts 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.9 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: osdmap e441: 8 total, 8 up, 7 in 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.7 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.8 scrub starts 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.2 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.8 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.4 scrub starts 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.4 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.0 scrub starts 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 10.0 scrub ok 2024-04-25T05:49:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: osdmap e442: 8 total, 8 up, 7 in 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 6.6 scrub starts 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[27010]: 6.6 scrub ok 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.9 scrub starts 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: pgmap v816: 168 pgs: 7 peering, 8 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.2 scrub starts 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.9 scrub ok 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: osdmap e441: 8 total, 8 up, 7 in 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.7 scrub ok 2024-04-25T05:49:35.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.8 scrub starts 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.2 scrub ok 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.8 scrub ok 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.4 scrub starts 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.4 scrub ok 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.0 scrub starts 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 10.0 scrub ok 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: osdmap e442: 8 total, 8 up, 7 in 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 6.6 scrub starts 2024-04-25T05:49:35.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:35 smithi043 ceph-mon[37492]: 6.6 scrub ok 2024-04-25T05:49:35.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:35.839 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:35.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:35.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:35.984 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:35.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:36.021 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:49:36.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:36.099 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:49:36.099 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:49:36.099 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-25T05:49:36.099 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-04-25T05:49:36.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:36.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:49:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:36.140 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:36.141 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:36.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:36.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:36.367 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.6 scrub starts 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.6 scrub ok 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.5 scrub starts 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.5 scrub ok 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:49:36.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'14122 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: osdmap e443: 8 total, 8 up, 7 in 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14061 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[27010]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.6 scrub starts 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.6 scrub ok 2024-04-25T05:49:36.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.5 scrub starts 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.5 scrub ok 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'14122 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: osdmap e443: 8 total, 8 up, 7 in 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14061 2024-04-25T05:49:36.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:36 smithi043 ceph-mon[37492]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.433 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.6 scrub starts 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.6 scrub ok 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.5 scrub starts 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.5 scrub ok 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:49:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2601172476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 440'14122 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: osdmap e443: 8 total, 8 up, 7 in 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14061 2024-04-25T05:49:36.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:36 smithi060 ceph-mon[30898]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13975 2024-04-25T05:49:36.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:36.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:36.674 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:36.710 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:36.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:36.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:36.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:36.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:37.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:37.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:37.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:37.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:37.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:49:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 7 peering, 7 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 440'14163 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14154 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:37 smithi060 ceph-mon[30898]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.509 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:37.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 7 peering, 7 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 440'14163 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14154 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[27010]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 7 peering, 7 unknown, 153 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:49:37.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 440'14163 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 440'14059 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 440'14062 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14154 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 440'14256 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 440'13872 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:37 smithi043 ceph-mon[37492]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 440'13930 2024-04-25T05:49:37.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:37.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:37.782 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:37.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:37.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:37.956 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:37.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:38.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.093 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:38.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:38.151 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:49:38.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.239 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:38.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:38.276 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:38.340 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:38.373 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:38.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:38.515 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:38.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:38.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:38.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:38.809 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:38.911 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:38.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:39.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:39.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:39.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:39.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:39.384 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:39 smithi060 ceph-mon[30898]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 4 peering, 2 active+remapped+backfilling, 5 unknown, 156 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 2762/96006 objects misplaced (2.877%); 5.2 MiB/s, 83 objects/s recovering 2024-04-25T05:49:39.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:39 smithi043 ceph-mon[37492]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 4 peering, 2 active+remapped+backfilling, 5 unknown, 156 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 2762/96006 objects misplaced (2.877%); 5.2 MiB/s, 83 objects/s recovering 2024-04-25T05:49:39.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:39 smithi043 ceph-mon[27010]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 4 peering, 2 active+remapped+backfilling, 5 unknown, 156 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 2762/96006 objects misplaced (2.877%); 5.2 MiB/s, 83 objects/s recovering 2024-04-25T05:49:39.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:39.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:39.610 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:39.610 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:39.633 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:39.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:39.769 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:39.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:39.910 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:39.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:39.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:49:40.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:40.046 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:40.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-04-25T05:49:40.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:40.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:40.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:40.359 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:49:40.393 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:40 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:49:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.404 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:40.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:40.486 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:40.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:40.538 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:40.541 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:40 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:49:40.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:40 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:49:40.542 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:40.594 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:40.637 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:40.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:40.688 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:40.688 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:40.737 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:40.790 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:40.798 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:40.810 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:40.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:40.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:40.846 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:40.850 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:40.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:41.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:41.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:41.138 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:41.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.283 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:41.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:41.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:41.421 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[27010]: pgmap v822: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 18 MiB/s, 291 objects/s recovering 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1482600170' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[27010]: 6.13 scrub starts 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[37492]: pgmap v822: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 18 MiB/s, 291 objects/s recovering 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1482600170' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:41.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:41 smithi043 ceph-mon[37492]: 6.13 scrub starts 2024-04-25T05:49:41.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:41.541 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:41 smithi060 ceph-mon[30898]: pgmap v822: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 18 MiB/s, 291 objects/s recovering 2024-04-25T05:49:41.541 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1482600170' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:41.541 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:41 smithi060 ceph-mon[30898]: 6.13 scrub starts 2024-04-25T05:49:41.541 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:49:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:49:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:49:41.541 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:49:41.541 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:49:41.557 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:41.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:41.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:41.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:41.832 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:41.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:41.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:41.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:42.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:42.086 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:42.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:42.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:42.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:49:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:49:42.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:42.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[37492]: 6.13 scrub ok 2024-04-25T05:49:42.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[37492]: osdmap e444: 8 total, 8 up, 7 in 2024-04-25T05:49:42.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3021531554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.336 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[27010]: 6.13 scrub ok 2024-04-25T05:49:42.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[27010]: osdmap e444: 8 total, 8 up, 7 in 2024-04-25T05:49:42.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3021531554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.345 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:42.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:42.472 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:42 smithi060 ceph-mon[30898]: 6.13 scrub ok 2024-04-25T05:49:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:42 smithi060 ceph-mon[30898]: osdmap e444: 8 total, 8 up, 7 in 2024-04-25T05:49:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3021531554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:49:42.536 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:42.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:42.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:42.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:42.718 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:42.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:42.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:42.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:42.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:43.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:43.086 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226339 8.12 8:48d1585c:::benchmark_data_smithi043_104001_object102215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.20795004799999, 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1294363890000001, 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.103 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226339 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893517+0000", 2024-04-25T05:49:43.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893519+0000", 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1519999999999999e-06 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893531+0000", 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1849e-05 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893551+0000", 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9638000000000001e-05 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893564+0000", 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.323e-05 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000742+0000", 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10717832200000001 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000752+0000", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9930000000000007e-06 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006085+0000", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0053329209999999 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006212+0000", 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126873 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.107 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.006455+0000", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024327100000000001 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011490+0000", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050344350000000003 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.012956+0000", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001465872 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022845+0000", 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0098890200000000001 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022904+0000", 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9239e-05 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.022954+0000", 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9573999999999997e-05 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226343 8.12 8:4e9368a0:::benchmark_data_smithi043_104001_object102219:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 423.20787540700002, 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1317489059999999, 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226343 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893592+0000", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4000000000000002e-07 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893595+0000", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8600000000000001e-06 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893603+0000", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9000000000000006e-06 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893609+0000", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2500000000000003e-06 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000774+0000", 2024-04-25T05:49:43.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10716527300000001 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.000780+0000", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7119999999999997e-06 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009444+0000", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0086639690000001 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009577+0000", 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013284399999999999 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.009825+0000", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024752200000000002 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.013047+0000", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003222042 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020080+0000", 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0070331810000000003 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025198+0000", 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051180369999999998 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025295+0000", 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6797000000000002e-05 2024-04-25T05:49:43.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.025341+0000", 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6078999999999997e-05 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228033 8.14 8:2d5c3048:::benchmark_data_smithi043_104001_object103909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 421.11331362800001, 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0985097700000002, 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228033 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988154+0000", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988156+0000", 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9879999999999999e-06 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988162+0000", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7870000000000004e-06 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988180+0000", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7886000000000001e-05 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988190+0000", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9729999999999992e-06 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988272+0000", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1977999999999993e-05 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.988285+0000", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3176000000000001e-05 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980591+0000", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99230595399999999 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980607+0000", 2024-04-25T05:49:43.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5548000000000001e-05 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081723+0000", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1011164630000001 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.081910+0000", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018638699999999999 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.082205+0000", 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00029534400000000001 2024-04-25T05:49:43.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085239+0000", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030338330000000001 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085283+0000", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3695000000000001e-05 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085372+0000", 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.954e-05 2024-04-25T05:49:43.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086212+0000", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084000600000000002 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086435+0000", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022244300000000001 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086501+0000", 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6000000000000005e-05 2024-04-25T05:49:43.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086663+0000", 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000162769 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228057 8.14 8:29afc790:::benchmark_data_smithi043_104001_object103933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 421.091501424, 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0771824429999999, 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.124 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228057 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009966+0000", 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009967+0000", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.418e-06 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009982+0000", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5211000000000001e-05 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.009997+0000", 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4542000000000001e-05 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010006+0000", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3619999999999995e-06 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010067+0000", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0965000000000003e-05 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.010076+0000", 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1249999999999999e-06 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980702+0000", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97062569499999996 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980712+0000", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0081999999999999e-05 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083607+0000", 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1028946399999999 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.083747+0000", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013999600000000001 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,5,6,7", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.084002+0000", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025568299999999998 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085297+0000", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001294033 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.085711+0000", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041416199999999998 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086266+0000", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055577100000000004 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086728+0000", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000461335 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086920+0000", 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019192499999999999 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.086999+0000", 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9742999999999999e-05 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.087148+0000", 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014875500000000001 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195454 10.1 10:8ac0cd3c:::benchmark_data_smithi043_152558_object67104:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.548686773, 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044723351, 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195454 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552782+0000", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7039999999999999e-06 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552789+0000", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8340000000000001e-06 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552799+0000", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0064999999999999e-05 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552808+0000", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3190000000000001e-06 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558556+0000", 2024-04-25T05:49:43.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057473109999999997 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558565+0000", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3989999999999993e-06 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591223+0000", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0326579060000001 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.597504+0000", 2024-04-25T05:49:43.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062808129999999997 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195456 10.1 10:80db890b:::benchmark_data_smithi043_152558_object67106:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.546785444999998, 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0479446400000001, 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.136 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195456 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554683+0000", 2024-04-25T05:49:43.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554687+0000", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3379999999999998e-06 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554696+0000", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0720000000000006e-06 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554703+0000", 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3239999999999997e-06 2024-04-25T05:49:43.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558586+0000", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038829329999999999 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558699+0000", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011292 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602367+0000", 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.04366767 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602626+0000", 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025938399999999998 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195470 10.4 10:2594b5ae:::benchmark_data_smithi043_152558_object67120:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.546753676000002, 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0253313630000001, 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.140 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195470 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7500000000000001e-07 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554716+0000", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3029999999999998e-06 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554721+0000", 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.181e-06 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554726+0000", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9300000000000002e-06 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555243+0000", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051627300000000004 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555251+0000", 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.578884+0000", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0236325479999999 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.580045+0000", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001161048 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195477 10.e 10:7c0c42f7:::benchmark_data_smithi043_152558_object67127:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:49:43.144 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.546729311, 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0658362960000001, 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195477 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3299999999999998e-07 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554740+0000", 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2330000000000001e-06 2024-04-25T05:49:43.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554746+0000", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.13e-06 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554751+0000", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3750000000000002e-06 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554809+0000", 2024-04-25T05:49:43.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8230000000000003e-05 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554820+0000", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0581000000000001e-05 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596186+0000", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0413658189999999 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.598902+0000", 2024-04-25T05:49:43.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027164310000000001 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.599221+0000", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000319031 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.611133+0000", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011912219999999999 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.614119+0000", 2024-04-25T05:49:43.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029860350000000002 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620485+0000", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063658589999999998 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620537+0000", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1572e-05 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620574+0000", 2024-04-25T05:49:43.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7447000000000002e-05 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195487 10.4 10:295dbb8c:::benchmark_data_smithi043_152558_object67137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.546708142, 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0346735600000001, 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.151 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195487 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8599999999999999e-07 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554761+0000", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0310000000000001e-06 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554766+0000", 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9450000000000001e-06 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554771+0000", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0819999999999998e-06 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555283+0000", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051210299999999997 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555291+0000", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1690000000000003e-06 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.581742+0000", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0264516699999999 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582176+0000", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043403700000000002 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582318+0000", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000142003 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589262+0000", 2024-04-25T05:49:43.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069436080000000004 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589327+0000", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4770000000000005e-05 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589367+0000", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9824999999999999e-05 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589405+0000", 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8486999999999999e-05 2024-04-25T05:49:43.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589433+0000", 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7543999999999999e-05 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195490 10.4 10:28cf0ec2:::benchmark_data_smithi043_152558_object67140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.543179016000003, 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0313008349999999, 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195490 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558289+0000", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1960000000000001e-06 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558293+0000", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8619999999999996e-06 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558301+0000", 2024-04-25T05:49:43.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.396e-06 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558308+0000", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3629999999999996e-06 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558920+0000", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061224599999999999 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558926+0000", 2024-04-25T05:49:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.891e-06 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584147+0000", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0252211170000001 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584221+0000", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4258999999999995e-05 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584294+0000", 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2475e-05 2024-04-25T05:49:43.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589475+0000", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051810500000000004 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589486+0000", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1388e-05 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589536+0000", 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0037999999999998e-05 2024-04-25T05:49:43.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589566+0000", 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9144000000000001e-05 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589589+0000", 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3410000000000001e-05 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.163 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195491 10.4 10:22b59bea:::benchmark_data_smithi043_152558_object67141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.543145080000002, 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0361324119999999, 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195491 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:49:43.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4999999999999998e-07 2024-04-25T05:49:43.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558325+0000", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2699999999999999e-06 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558330+0000", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0189999999999997e-06 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558335+0000", 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0799999999999996e-06 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558939+0000", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060434400000000004 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558943+0000", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.036e-06 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.585996+0000", 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0270530360000001 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586084+0000", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7683999999999996e-05 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586156+0000", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1772000000000003e-05 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589499+0000", 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033435169999999998 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594275+0000", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047758619999999996 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594326+0000", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0866999999999999e-05 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594406+0000", 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9572000000000003e-05 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594455+0000", 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9002999999999998e-05 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195496 10.4 10:23a4e9e2:::benchmark_data_smithi043_152558_object67146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.538890690999999, 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.032050406, 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.171 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195496 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562578+0000", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0470000000000001e-06 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562581+0000", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.839e-06 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562590+0000", 2024-04-25T05:49:43.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2190000000000004e-06 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562595+0000", 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.468e-06 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562692+0000", 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7392999999999994e-05 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562699+0000", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7109999999999998e-06 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587618+0000", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.024919208 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587708+0000", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0075999999999996e-05 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587795+0000", 2024-04-25T05:49:43.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6837999999999998e-05 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594287+0000", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064916610000000001 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594498+0000", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021106299999999999 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594545+0000", 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.655e-05 2024-04-25T05:49:43.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594590+0000", 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5574000000000001e-05 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594627+0000", 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6758999999999998e-05 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195505 10.1 10:82918380:::benchmark_data_smithi043_152558_object67155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.177 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.525647696999997, 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.053126166, 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195505 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:49:43.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575821+0000", 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.08e-06 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575827+0000", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6819999999999997e-06 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575835+0000", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2160000000000006e-06 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575841+0000", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2499999999999997e-06 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575897+0000", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6360000000000002e-05 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575904+0000", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1749999999999999e-06 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602674+0000", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.02677006 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603270+0000", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00059573700000000005 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603459+0000", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018862499999999999 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615485+0000", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012026722 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621556+0000", 2024-04-25T05:49:43.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060705749999999999 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628800+0000", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072444809999999997 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628892+0000", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1471000000000006e-05 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628946+0000", 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3732e-05 2024-04-25T05:49:43.183 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195517 10.4 10:24331628:::benchmark_data_smithi043_152558_object67167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.514647678999999, 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007987988, 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.184 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195517 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586821+0000", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.252e-06 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586825+0000", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586836+0000", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0988999999999999e-05 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586845+0000", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586911+0000", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5667000000000007e-05 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586923+0000", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2104e-05 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589644+0000", 2024-04-25T05:49:43.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0027209029999999 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589733+0000", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9178000000000006e-05 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589854+0000", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120975 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594292+0000", 2024-04-25T05:49:43.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044382069999999996 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594675+0000", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038344300000000002 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594716+0000", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1041999999999998e-05 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594767+0000", 2024-04-25T05:49:43.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0926e-05 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594807+0000", 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.019e-05 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195519 10.4 10:29710d26:::benchmark_data_smithi043_152558_object67169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.510473476000001, 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.005228536, 2024-04-25T05:49:43.190 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195519 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590995+0000", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6339999999999999e-06 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591000+0000", 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.147e-06 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591014+0000", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3421000000000001e-05 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591022+0000", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1899999999999995e-06 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591079+0000", 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6852999999999999e-05 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591090+0000", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1545999999999999e-05 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591626+0000", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.000535591 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591729+0000", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102774 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591850+0000", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121255 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594508+0000", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026583100000000001 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596021+0000", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015122289999999999 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596061+0000", 2024-04-25T05:49:43.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0339999999999997e-05 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596174+0000", 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112823 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596222+0000", 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8423e-05 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195520 10.1 10:8be1fae5:::benchmark_data_smithi043_152558_object67170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.510427360999998, 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0383997030000001, 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195520 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1899999999999998e-07 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591045+0000", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7779999999999998e-06 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591051+0000", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.784e-06 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591056+0000", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591110+0000", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3823000000000003e-05 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591117+0000", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0520000000000004e-06 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612643+0000", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0215256699999999 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612745+0000", 2024-04-25T05:49:43.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102353 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612856+0000", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011074900000000001 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615525+0000", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002668784 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628786+0000", 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013261088000000001 2024-04-25T05:49:43.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629032+0000", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024633800000000003 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629115+0000", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2599e-05 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629440+0000", 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00032487500000000002 2024-04-25T05:49:43.202 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195526 10.1 10:8fa6926a:::benchmark_data_smithi043_152558_object67176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.506408403000002, 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.037524509, 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.203 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195526 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595060+0000", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.488e-06 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595066+0000", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6559999999999997e-06 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595080+0000", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3925e-05 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595089+0000", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6810000000000004e-06 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595171+0000", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2888999999999999e-05 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595186+0000", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4477e-05 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615572+0000", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0203861359999999 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615673+0000", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101087 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.616535+0000", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00086202200000000005 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621514+0000", 2024-04-25T05:49:43.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049784570000000004 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629450+0000", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0079360639999999996 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632427+0000", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029770019999999999 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632527+0000", 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100078 2024-04-25T05:49:43.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632583+0000", 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6546999999999999e-05 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195534 10.4 10:2bc4d148:::benchmark_data_smithi043_152558_object67184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.496023960000002, 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.004935758, 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.209 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195534 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.210 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605445+0000", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.59e-06 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605451+0000", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3500000000000002e-06 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605466+0000", 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4555e-05 2024-04-25T05:49:43.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605474+0000", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8639999999999996e-06 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605539+0000", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5467000000000002e-05 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605553+0000", 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4328000000000001e-05 2024-04-25T05:49:43.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596256+0000", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99070311899999997 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596366+0000", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109079 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596569+0000", 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020323699999999999 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.600295+0000", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00372573 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602218+0000", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001922972 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609480+0000", 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072625119999999996 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609538+0000", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7531999999999997e-05 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.610379+0000", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000841423 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195540 10.1 10:8978d76f:::benchmark_data_smithi043_152558_object67190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:49:43.215 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.486095470000002, 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.017365112, 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195540 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615373+0000", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.311e-06 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615378+0000", 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3249999999999998e-06 2024-04-25T05:49:43.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615391+0000", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2245e-05 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615397+0000", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7499999999999997e-06 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615462+0000", 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4750000000000002e-05 2024-04-25T05:49:43.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615470+0000", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6550000000000004e-06 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.618963+0000", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.003493653 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619188+0000", 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022493599999999999 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619317+0000", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012912399999999999 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621613+0000", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022951650000000001 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629456+0000", 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0078430449999999999 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632635+0000", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003178883 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632691+0000", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5965999999999999e-05 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632737+0000", 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6304000000000003e-05 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195547 10.1 10:8a6211c9:::benchmark_data_smithi043_152558_object67197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.635134+0000", 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 34.466333501000001, 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0005611219999999, 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195547 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.222 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635134+0000", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635134+0000", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635135+0000", 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3969999999999999e-06 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635140+0000", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8119999999999998e-06 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635151+0000", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1419e-05 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635157+0000", 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9889999999999997e-06 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.224 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635219+0000", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2093000000000003e-05 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.635228+0000", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8699999999999998e-06 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621650+0000", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98642219200000003 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:49:43.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621767+0000", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116136 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621896+0000", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012982599999999999 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.624930+0000", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030332900000000001 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:49:43.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629464+0000", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045341970000000002 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.635041+0000", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0055769060000000004 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.635094+0000", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3164999999999997e-05 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.635695+0000", 2024-04-25T05:49:43.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060083000000000001 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:49:43.228 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:49:43.228 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:43.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:43.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:43.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:43.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:43.610 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:43 smithi060 ceph-mon[30898]: pgmap v824: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 733 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:49:43.610 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:43 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:43.610 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:43.610 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:43 smithi060 ceph-mon[30898]: osdmap e445: 8 total, 8 up, 7 in 2024-04-25T05:49:43.616 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[27010]: pgmap v824: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 733 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[27010]: osdmap e445: 8 total, 8 up, 7 in 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[37492]: pgmap v824: 168 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 158 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 733 B/s rd, 0 op/s; 13032/85242 objects misplaced (15.288%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:49:43.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:49:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:43 smithi043 ceph-mon[37492]: osdmap e445: 8 total, 8 up, 7 in 2024-04-25T05:49:43.650 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:43.752 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:43.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:43.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:43.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:43.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:44.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:44.029 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:44.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:44.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:44.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:44.274 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:44.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:44.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:44.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:44.538 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:44.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:44 smithi043 ceph-mon[27010]: osdmap e446: 8 total, 8 up, 7 in 2024-04-25T05:49:44.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:44 smithi043 ceph-mon[37492]: osdmap e446: 8 total, 8 up, 7 in 2024-04-25T05:49:44.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:44.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:44 smithi060 ceph-mon[30898]: osdmap e446: 8 total, 8 up, 7 in 2024-04-25T05:49:44.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:44.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:44.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:44.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:44.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:45.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:45.071 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:45.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:45.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:45.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-25T05:49:45.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:45.335 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:45.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:45.468 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:45.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:49:45.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:45.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:45.633 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[27010]: pgmap v827: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 160 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 7779/73770 objects misplaced (10.545%); 42 MiB/s, 670 objects/s recovering 2024-04-25T05:49:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[27010]: osdmap e447: 8 total, 8 up, 7 in 2024-04-25T05:49:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[37492]: pgmap v827: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 160 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 7779/73770 objects misplaced (10.545%); 42 MiB/s, 670 objects/s recovering 2024-04-25T05:49:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:45 smithi043 ceph-mon[37492]: osdmap e447: 8 total, 8 up, 7 in 2024-04-25T05:49:45.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:45 smithi060 ceph-mon[30898]: pgmap v827: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 160 active+clean; 1.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 7779/73770 objects misplaced (10.545%); 42 MiB/s, 670 objects/s recovering 2024-04-25T05:49:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:45 smithi060 ceph-mon[30898]: osdmap e447: 8 total, 8 up, 7 in 2024-04-25T05:49:45.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:45.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:49:45.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:45.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:45.937 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:45.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.027 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:46.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:46.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:46.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:46.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:49:46.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:46.312 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:46.352 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:46.413 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:46.459 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:46.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:46.561 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:49:46.561 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:49:46.561 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:49:46.561 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:49:46.595 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:46.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:46.730 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:46.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:46.865 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:46.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:46.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:46.999 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:47.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:47.128 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:47.230 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:47.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:47.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:47.396 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:47.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-25T05:49:47.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:47.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:47 smithi060 ceph-mon[30898]: pgmap v829: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 5751/71850 objects misplaced (8.004%); 43 MiB/s, 694 objects/s recovering 2024-04-25T05:49:47.636 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:47 smithi043 ceph-mon[27010]: pgmap v829: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 5751/71850 objects misplaced (8.004%); 43 MiB/s, 694 objects/s recovering 2024-04-25T05:49:47.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:47 smithi043 ceph-mon[37492]: pgmap v829: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 946 op/s; 5751/71850 objects misplaced (8.004%); 43 MiB/s, 694 objects/s recovering 2024-04-25T05:49:47.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:49:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:47.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:47.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:47.858 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:47.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:47.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:48.012 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:48.086 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:48.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:48.116 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:48.157 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:48.262 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:48.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:48.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:48.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:48.525 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:49:48.525 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:49:48.525 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[27010]: osdmap e448: 8 total, 8 up, 7 in 2024-04-25T05:49:48.525 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:48.525 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:48.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:49:48.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:49:48.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[37492]: osdmap e448: 8 total, 8 up, 7 in 2024-04-25T05:49:48.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:48.526 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:48.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:48.567 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:48.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:48 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:49:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:48 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:49:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:48 smithi060 ceph-mon[30898]: osdmap e448: 8 total, 8 up, 7 in 2024-04-25T05:49:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:49:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:49:48.797 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:49:48.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:48.839 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:49:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:48.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:48.877 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:49:48.916 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:49:48.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:48.954 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:49:48.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:48.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:48.994 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:49:49.036 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:49:49.076 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:49:49.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:49.091 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:49:49.091 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:49:49.091 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:49:49.091 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:49:49.091 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:49:49.092 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:49:49.092 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:49:49.092 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:49:49.111 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:49.213 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:49.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:49.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:49.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: pgmap v831: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 906 op/s; 5751/69705 objects misplaced (8.250%); 36 MiB/s, 573 objects/s recovering 2024-04-25T05:49:49.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3026530460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:49:49.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1810293234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: pgmap v831: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 906 op/s; 5751/69705 objects misplaced (8.250%); 36 MiB/s, 573 objects/s recovering 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3026530460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:49:49.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1810293234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:49.359 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:49:49.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:49.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: pgmap v831: 168 pgs: 3 peering, 4 active+remapped+backfill_toofull, 161 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 906 op/s; 5751/69705 objects misplaced (8.250%); 36 MiB/s, 573 objects/s recovering 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3026530460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:49:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1810293234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:49:49.509 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:49.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.643 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:49.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:49.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:49.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:49.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:49.911 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:49.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:50.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:50.052 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:50.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:50.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:50.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:50.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.319 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:50.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:50.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:50.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:50 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:50 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:50 smithi060 ceph-mon[30898]: osdmap e449: 8 total, 8 up, 7 in 2024-04-25T05:49:50.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[27010]: osdmap e449: 8 total, 8 up, 7 in 2024-04-25T05:49:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:49:50.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:49:50.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:50 smithi043 ceph-mon[37492]: osdmap e449: 8 total, 8 up, 7 in 2024-04-25T05:49:50.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:50.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:50.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:50.851 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:50.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:50.985 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:50.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:51.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:51.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:51.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:51.249 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:51.256 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:51.256 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:51.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:51.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:51.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:51.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:51.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:51.629 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:51 smithi060 ceph-mon[30898]: pgmap v833: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 4205/53148 objects misplaced (7.912%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:49:51.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:51 smithi043 ceph-mon[27010]: pgmap v833: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 4205/53148 objects misplaced (7.912%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:49:51.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:51 smithi043 ceph-mon[37492]: pgmap v833: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 4205/53148 objects misplaced (7.912%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:49:51.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:51.761 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:51.794 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:51.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:51.931 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:51.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:52.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:52.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:49:52.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:52.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:52.231 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:52.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:52.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.504 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:52.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:52.546 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:52.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:52.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:52.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:52.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:52.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:52.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:52.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:53.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:53.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:53.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:53.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:53.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:53.291 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:53.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:53.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:53.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:53.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:53.616 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:53 smithi060 ceph-mon[30898]: pgmap v834: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 664 B/s rd, 0 B/s wr, 809 op/s; 4205/53148 objects misplaced (7.912%) 2024-04-25T05:49:53.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:53 smithi043 ceph-mon[27010]: pgmap v834: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 664 B/s rd, 0 B/s wr, 809 op/s; 4205/53148 objects misplaced (7.912%) 2024-04-25T05:49:53.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:53 smithi043 ceph-mon[37492]: pgmap v834: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 1.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 664 B/s rd, 0 B/s wr, 809 op/s; 4205/53148 objects misplaced (7.912%) 2024-04-25T05:49:53.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:53.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:53.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:53.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:53.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:53.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:49:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:49:53.957 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:53.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:54.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:54.098 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:54.200 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:54.241 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:54.317 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:54.317 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:54.343 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.376 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:54.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:54.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.501 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:54.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:54.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:54.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:54.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:54.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:54.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:54.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:55.008 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:55.046 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:55.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:55.179 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:55.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:55.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:55.403 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:55.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:55.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:55.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:55.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:55 smithi043 ceph-mon[27010]: pgmap v835: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 719 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 3554/35172 objects misplaced (10.105%) 2024-04-25T05:49:55.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:55.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:55 smithi043 ceph-mon[37492]: pgmap v835: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 719 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 3554/35172 objects misplaced (10.105%) 2024-04-25T05:49:55.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:49:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:55 smithi060 ceph-mon[30898]: pgmap v835: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 719 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 3554/35172 objects misplaced (10.105%) 2024-04-25T05:49:55.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:55.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:55.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:55.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:49:55.952 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:55.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:56.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:56.102 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:56.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:56.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:56.336 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:56.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:56.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:56.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:56.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.650 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:56.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:56.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:56.789 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:56.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:56.935 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:56.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:57.016 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:57.016 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:57.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:57.074 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:57.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:57.213 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:57.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:57.351 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:57.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:57.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:57.588 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:57.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:57.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:57 smithi043 ceph-mon[27010]: pgmap v836: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 657 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 2551/32190 objects misplaced (7.925%) 2024-04-25T05:49:57.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:57 smithi043 ceph-mon[37492]: pgmap v836: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 657 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 2551/32190 objects misplaced (7.925%) 2024-04-25T05:49:57.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-04-25T05:49:57.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:57.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:57 smithi060 ceph-mon[30898]: pgmap v836: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 657 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 2551/32190 objects misplaced (7.925%) 2024-04-25T05:49:57.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:57.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:49:57.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:49:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:57.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:57.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:49:58.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:58.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:58.062 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:58.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:58.175 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:58.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:49:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:49:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.338 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:58.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:49:58.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:49:58.475 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:49:58.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:58.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:49:58.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:49:58.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:58.873 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:58.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:58.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:49:58.943 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:49:58.944 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:49:58.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:59.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:49:59.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:49:59.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:49:59.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:59.279 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:59.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:49:59.416 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:49:59.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:49:59 smithi043 ceph-mon[27010]: pgmap v837: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 608 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 2551/29841 objects misplaced (8.549%) 2024-04-25T05:49:59.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:49:59 smithi043 ceph-mon[37492]: pgmap v837: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 608 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 2551/29841 objects misplaced (8.549%) 2024-04-25T05:49:59.517 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:59.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-04-25T05:49:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:59.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:59.645 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:49:59 smithi060 ceph-mon[30898]: pgmap v837: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 608 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s; 2551/29841 objects misplaced (8.549%) 2024-04-25T05:49:59.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:49:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:59.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:49:59.697 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:49:59.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:49:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:49:59.788 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:49:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:49:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:49:59.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:49:59.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:49:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:49:59.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:49:59.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:00.089 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:50:00.122 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:00.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:00.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:00.295 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:00.397 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:00.435 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: pg 10.11 is active+remapped+backfill_toofull, acting [1,7,4] 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: pg 10.14 is active+remapped+backfill_toofull, acting [1,4,3] 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: pg 10.15 is active+remapped+backfill_toofull, acting [3,2,7] 2024-04-25T05:50:00.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:00 smithi060 ceph-mon[30898]: pg 10.19 is active+remapped+backfill_toofull, acting [5,2,7] 2024-04-25T05:50:00.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: pg 10.11 is active+remapped+backfill_toofull, acting [1,7,4] 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: pg 10.14 is active+remapped+backfill_toofull, acting [1,4,3] 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: pg 10.15 is active+remapped+backfill_toofull, acting [3,2,7] 2024-04-25T05:50:00.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[27010]: pg 10.19 is active+remapped+backfill_toofull, acting [5,2,7] 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: pg 10.11 is active+remapped+backfill_toofull, acting [1,7,4] 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: pg 10.14 is active+remapped+backfill_toofull, acting [1,4,3] 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: pg 10.15 is active+remapped+backfill_toofull, acting [3,2,7] 2024-04-25T05:50:00.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:00 smithi043 ceph-mon[37492]: pg 10.19 is active+remapped+backfill_toofull, acting [5,2,7] 2024-04-25T05:50:00.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.694 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:00.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:00.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:00.830 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:00.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:00.962 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:00.968 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:00.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:01.035 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:01.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:01.073 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:01.079 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 128347 objects 2024-04-25T05:50:01.079 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :109.111 2024-04-25T05:50:01.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:01.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:01.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:01.115 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:01.136 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:01.157 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:01.177 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:01.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:01.198 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:01.199 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:01.200 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.219 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:01.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:01.341 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:01.380 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:50:01.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:01.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[27010]: pgmap v838: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.29k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1185065921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[37492]: pgmap v838: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.29k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:01.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1185065921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:01.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:01.588 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:01.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:01.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:01 smithi060 ceph-mon[30898]: pgmap v838: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.29k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1185065921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:01.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:01.835 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:01.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:01.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:01.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:02.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:02.086 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:02.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:02.209 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:02.311 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:02.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:02.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:02.462 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:02.564 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:02.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:02.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:02.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:02.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: osdmap e450: 8 total, 8 up, 7 in 2024-04-25T05:50:02.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:50:02.623 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:50:02.624 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: 10.13 scrub starts 2024-04-25T05:50:02.624 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:02 smithi060 ceph-mon[30898]: 10.13 scrub ok 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: osdmap e450: 8 total, 8 up, 7 in 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: 10.13 scrub starts 2024-04-25T05:50:02.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[27010]: 10.13 scrub ok 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1682974619' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: osdmap e450: 8 total, 8 up, 7 in 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: 10.13 scrub starts 2024-04-25T05:50:02.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:02 smithi043 ceph-mon[37492]: 10.13 scrub ok 2024-04-25T05:50:02.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:02.711 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:02.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:02.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:02.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:02.958 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:02.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:03.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:03.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:03.125 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-25T05:50:03.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-25T05:50:03.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:03.206 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:03.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.332 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:03.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:03.348 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-04-25T05:50:03.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226342 8.5 8:a7a71f38:::benchmark_data_smithi043_104001_object102218:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:50:03.444 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 442.678225688, 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1555693469999999, 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226342 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:50:03.445 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893567+0000", 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893569+0000", 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2000000000000001e-06 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.446 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893577+0000", 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5590000000000006e-06 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893597+0000", 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9826e-05 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893608+0000", 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1221e-05 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.447 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893676+0000", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7723000000000002e-05 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893681+0000", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6539999999999998e-06 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082813+0000", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.189132576 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.082823+0000", 2024-04-25T05:50:03.448 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6500000000000008e-06 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015505+0000", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.93268209199999996 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.015634+0000", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012852800000000001 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048918+0000", 2024-04-25T05:50:03.449 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033284728999999999 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049136+0000", 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021758899999999999 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226356 8.5 8:a631803d:::benchmark_data_smithi043_104001_object102232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 442.44209163699998, 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92008493099999999, 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226356 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129701+0000", 2024-04-25T05:50:03.451 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129703+0000", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6369999999999999e-06 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129718+0000", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4413000000000001e-05 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129742+0000", 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.472e-05 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.452 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.129757+0000", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4751e-05 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130011+0000", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025401500000000001 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.130048+0000", 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6837e-05 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.453 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016167+0000", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88611901299999996 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016334+0000", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016642 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,7", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.016642+0000", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00030824900000000002 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.454 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.048561+0000", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.031919212000000002 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049494+0000", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093273599999999998 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049612+0000", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000118149 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049689+0000", 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7195999999999997e-05 2024-04-25T05:50:03.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.049786+0000", 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6582999999999993e-05 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232758 10.16 10:6a536367:::benchmark_data_smithi043_152558_object104408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.220849893, 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99219000999999996, 2024-04-25T05:50:03.456 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232758 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.457 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350943+0000", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3699999999999999e-07 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350948+0000", 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2919999999999997e-06 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350957+0000", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7089999999999997e-06 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350965+0000", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5610000000000001e-06 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351038+0000", 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3561999999999998e-05 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.459 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351047+0000", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4549999999999995e-06 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323496+0000", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97244899900000004 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323612+0000", 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011592200000000001 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.460 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323850+0000", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023792799999999999 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326511+0000", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026609490000000001 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329075+0000", 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002564607 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.461 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343036+0000", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013961031 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343089+0000", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2522999999999998e-05 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343132+0000", 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3634999999999998e-05 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.462 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232859 10.16 10:6a2a3b0c:::benchmark_data_smithi043_152558_object104509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.133288659000002, 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90477883999999997, 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232859 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.463 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438505+0000", 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5820000000000001e-06 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438514+0000", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3529999999999995e-06 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438522+0000", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7870000000000007e-06 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438528+0000", 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6030000000000004e-06 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438592+0000", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4029000000000006e-05 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438602+0000", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8349999999999999e-06 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325262+0000", 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88666051999999995 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325370+0000", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010740599999999999 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325516+0000", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000146389 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329163+0000", 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036465989999999999 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.467 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329175+0000", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2473999999999999e-05 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343188+0000", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014012754000000001 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343242+0000", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3992000000000001e-05 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.468 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343283+0000", 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0516999999999998e-05 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232861 10.16 10:6d117552:::benchmark_data_smithi043_152558_object104511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.129559319999998, 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90115485500000003, 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.469 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232861 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442234+0000", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2589999999999999e-06 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442239+0000", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442248+0000", 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4299999999999995e-06 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442257+0000", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2199999999999998e-06 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442330+0000", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2401e-05 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442341+0000", 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1063999999999999e-05 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.472 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326560+0000", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88421923400000002 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326658+0000", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7768000000000003e-05 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326835+0000", 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000177527 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.473 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329180+0000", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023450599999999999 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330157+0000", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00097623399999999998 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343317+0000", 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013160420000000001 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.474 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343352+0000", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5067000000000002e-05 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343388+0000", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5781000000000002e-05 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232866 10.16 10:6de63d91:::benchmark_data_smithi043_152558_object104516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:50:03.475 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.115149434999999, 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88762149599999995, 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232866 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456644+0000", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5700000000000001e-07 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456648+0000", 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.69e-06 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456658+0000", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0780000000000002e-06 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456663+0000", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5609999999999998e-06 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456723+0000", 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9577999999999998e-05 2024-04-25T05:50:03.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456731+0000", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1929999999999993e-06 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327815+0000", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.871084317 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327953+0000", 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000137389 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.328167+0000", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000214324 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329185+0000", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001018166 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330231+0000", 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010455309999999999 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.480 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344181+0000", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013950071 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344227+0000", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6589999999999999e-05 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344264+0000", 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7150999999999997e-05 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232888 10.12 10:4f53b376:::benchmark_data_smithi043_152558_object104538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.224063835999999, 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96954986499999996, 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232888 2024-04-25T05:50:03.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347731+0000", 2024-04-25T05:50:03.483 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.295e-06 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347736+0000", 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347752+0000", 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6121e-05 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.484 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347760+0000", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1659999999999994e-06 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347880+0000", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932199999999999 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347889+0000", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.482e-06 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.485 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314176+0000", 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96628726499999995 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314286+0000", 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010990899999999999 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314365+0000", 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9061999999999996e-05 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.486 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317147+0000", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027818740000000002 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317183+0000", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5639000000000002e-05 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317219+0000", 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6324e-05 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.487 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317250+0000", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0596000000000003e-05 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317278+0000", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8479999999999998e-05 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232966 10.12 10:49a888b9:::benchmark_data_smithi043_152558_object104616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.176606459999999, 2024-04-25T05:50:03.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92239090000000001, 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232966 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395187+0000", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2399999999999997e-07 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395190+0000", 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4920000000000002e-06 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395200+0000", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7589999999999992e-06 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395208+0000", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8110000000000003e-06 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395287+0000", 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9079000000000006e-05 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395295+0000", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8080000000000005e-06 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315576+0000", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92028153000000001 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315678+0000", 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102095 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315768+0000", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0304999999999997e-05 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317349+0000", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00158056 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317389+0000", 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9878e-05 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317472+0000", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2762000000000005e-05 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317523+0000", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1730000000000001e-05 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317577+0000", 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3366999999999999e-05 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.494 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233055 10.12 10:4ec1d645:::benchmark_data_smithi043_152558_object104705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.124685052, 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87067929700000002, 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233055 2024-04-25T05:50:03.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447109+0000", 2024-04-25T05:50:03.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4980000000000001e-06 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447114+0000", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1510000000000003e-06 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447128+0000", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4012000000000001e-05 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447136+0000", 2024-04-25T05:50:03.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7579999999999992e-06 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447275+0000", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013975899999999999 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447283+0000", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6590000000000007e-06 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316413+0000", 2024-04-25T05:50:03.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.86913016700000001 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316518+0000", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010422800000000001 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316597+0000", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9097000000000005e-05 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317594+0000", 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099745499999999996 2024-04-25T05:50:03.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317631+0000", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7342999999999998e-05 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317700+0000", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8861000000000004e-05 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317752+0000", 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1872000000000001e-05 2024-04-25T05:50:03.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317787+0000", 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4437000000000002e-05 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233093 10.12 10:4d4ed51c:::benchmark_data_smithi043_152558_object104743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.467745+0000", 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.104047755, 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85331620200000002, 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.501 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233093 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467745+0000", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467745+0000", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467745+0000", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1099999999999995e-07 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467749+0000", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9430000000000002e-06 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467757+0000", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6539999999999999e-06 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467762+0000", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2089999999999996e-06 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467817+0000", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4963000000000002e-05 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.467824+0000", 2024-04-25T05:50:03.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4270000000000002e-06 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317834+0000", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85000929800000002 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317907+0000", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3467000000000007e-05 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.318008+0000", 2024-04-25T05:50:03.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100827 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.320855+0000", 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0028470940000000001 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.320867+0000", 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1836e-05 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.320950+0000", 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3369999999999996e-05 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.321015+0000", 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4908999999999998e-05 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.321061+0000", 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5593999999999997e-05 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233150 10.12 10:4a243804:::benchmark_data_smithi043_152558_object104800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.504048+0000", 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.067744665999999, 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.82261190100000003, 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233150 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504048+0000", 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504048+0000", 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504048+0000", 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1899999999999998e-07 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504051+0000", 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5299999999999999e-06 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504056+0000", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.801e-06 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504063+0000", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1840000000000004e-06 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504117+0000", 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3882999999999999e-05 2024-04-25T05:50:03.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.504126+0000", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3659999999999998e-06 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.318717+0000", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.81459157900000001 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.318840+0000", 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000122299 2024-04-25T05:50:03.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.318959+0000", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011919200000000001 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.320872+0000", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019129430000000001 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.324118+0000", 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032464780000000001 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326532+0000", 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024132289999999998 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326596+0000", 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3987999999999998e-05 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326660+0000", 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4010000000000005e-05 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233177 10.12 10:4eb8fbc4:::benchmark_data_smithi043_152558_object104827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.521908+0000", 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 19.049884835, 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.80486948599999997, 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.515 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233177 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521908+0000", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521908+0000", 2024-04-25T05:50:03.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521908+0000", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5300000000000002e-07 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521913+0000", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1019999999999999e-06 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521922+0000", 2024-04-25T05:50:03.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3379999999999994e-06 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.521929+0000", 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1829999999999997e-06 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.522004+0000", 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4946999999999999e-05 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.522012+0000", 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0539999999999995e-06 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.319796+0000", 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.79778424699999995 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.319899+0000", 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103142 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.320029+0000", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012904100000000001 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326347+0000", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00631841 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326705+0000", 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00035791000000000002 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326715+0000", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0199e-05 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326751+0000", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6350999999999998e-05 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326777+0000", 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.5609000000000002e-05 2024-04-25T05:50:03.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233210 10.12 10:49033063:::benchmark_data_smithi043_152558_object104860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.594661+0000", 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.977130934000002, 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.73862501800000002, 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233210 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594661+0000", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594661+0000", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594662+0000", 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.06e-07 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594665+0000", 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.129e-06 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594671+0000", 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1210000000000003e-06 2024-04-25T05:50:03.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594677+0000", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5319999999999997e-06 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594819+0000", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014270799999999999 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.594828+0000", 2024-04-25T05:50:03.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1710000000000002e-06 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.321110+0000", 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.72628275499999995 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.321221+0000", 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111117 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.321333+0000", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011158399999999999 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326393+0000", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050596479999999999 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326849+0000", 2024-04-25T05:50:03.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00045628500000000001 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333195+0000", 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063461460000000004 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333243+0000", 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7848e-05 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333286+0000", 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3467999999999998e-05 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233214 10.12 10:4c5628c7:::benchmark_data_smithi043_152558_object104864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.595021+0000", 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.976770974000001, 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.73835113100000005, 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233214 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595021+0000", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595021+0000", 2024-04-25T05:50:03.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595022+0000", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7300000000000005e-07 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595027+0000", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1410000000000004e-06 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595041+0000", 2024-04-25T05:50:03.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4011e-05 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595049+0000", 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9759999999999995e-06 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595118+0000", 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8629999999999999e-05 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.595126+0000", 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4039999999999995e-06 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.322195+0000", 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.72706886599999998 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.322310+0000", 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114583 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.322401+0000", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1570999999999995e-05 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326432+0000", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040302819999999996 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326910+0000", 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00047873099999999998 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333294+0000", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063840449999999997 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333336+0000", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1561999999999999e-05 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333373+0000", 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6556e-05 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233237 10.12 10:4e9d35f7:::benchmark_data_smithi043_152558_object104887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.609888+0000", 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.961904776000001, 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.72356988600000005, 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233237 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609888+0000", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609888+0000", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609888+0000", 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0999999999999997e-07 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609893+0000", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5940000000000004e-06 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609903+0000", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5959999999999993e-06 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.609911+0000", 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0630000000000006e-06 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.610001+0000", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0824999999999999e-05 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.610007+0000", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9030000000000001e-06 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.323049+0000", 2024-04-25T05:50:03.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.71304146899999998 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.323174+0000", 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000124697 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.323293+0000", 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119531 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326490+0000", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031964350000000001 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.326963+0000", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00047361199999999999 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333379+0000", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064162050000000003 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333422+0000", 2024-04-25T05:50:03.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2280999999999997e-05 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333457+0000", 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5865000000000003e-05 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233288 10.12 10:4acd60be:::benchmark_data_smithi043_152558_object104938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.646613+0000", 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.925179643, 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.68692024500000004, 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233288 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646613+0000", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646613+0000", 2024-04-25T05:50:03.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646613+0000", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4199999999999995e-07 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646618+0000", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4410000000000003e-06 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646635+0000", 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6844e-05 2024-04-25T05:50:03.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646643+0000", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9340000000000006e-06 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646809+0000", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016615900000000001 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.646817+0000", 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9009999999999994e-06 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.324150+0000", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.67733296499999995 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.324247+0000", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7590000000000006e-05 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.324347+0000", 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9789000000000003e-05 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327048+0000", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027012709999999999 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327132+0000", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3725000000000002e-05 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333464+0000", 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063316900000000001 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333500+0000", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6226000000000002e-05 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333533+0000", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2968e-05 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233355 10.12 10:4d95d370:::benchmark_data_smithi043_152558_object105005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.695961+0000", 2024-04-25T05:50:03.548 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.875830885999999, 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.63764412500000001, 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233355 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695961+0000", 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695961+0000", 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695963+0000", 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1710000000000001e-06 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695968+0000", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2190000000000003e-06 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695988+0000", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0472e-05 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.695997+0000", 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8100000000000004e-06 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.696132+0000", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013478399999999999 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.696140+0000", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3529999999999995e-06 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.325232+0000", 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.62909138200000003 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.325332+0000", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010062 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.325436+0000", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000103902 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327088+0000", 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001652244 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.329532+0000", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024430879999999999 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333539+0000", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040071869999999997 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333574+0000", 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5703e-05 2024-04-25T05:50:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333606+0000", 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1189999999999998e-05 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233386 10.12 10:4f3f9716:::benchmark_data_smithi043_152558_object105036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.719893+0000", 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.851899589999999, 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.61928121199999997, 2024-04-25T05:50:03.555 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233386 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719893+0000", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719893+0000", 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719894+0000", 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3300000000000001e-07 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719898+0000", 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.997e-06 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719907+0000", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4029999999999996e-06 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719913+0000", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5310000000000001e-06 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719959+0000", 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6616999999999998e-05 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.719965+0000", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7440000000000004e-06 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327163+0000", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.60719815700000002 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327265+0000", 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102006 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.327362+0000", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7281999999999995e-05 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333041+0000", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0056789010000000001 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333614+0000", 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00057253499999999999 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.339058+0000", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0054437569999999996 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.339131+0000", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3491999999999994e-05 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.339174+0000", 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2957000000000002e-05 2024-04-25T05:50:03.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233391 10.12 10:4d18bb9a:::benchmark_data_smithi043_152558_object105041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.723307+0000", 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.848485653000001, 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.618958593, 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233391 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723307+0000", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723307+0000", 2024-04-25T05:50:03.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723308+0000", 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0610000000000001e-06 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723313+0000", 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4859999999999997e-06 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723324+0000", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0304e-05 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723331+0000", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7670000000000003e-06 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723390+0000", 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9048000000000001e-05 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.723397+0000", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3579999999999999e-06 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.328333+0000", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.60493587500000001 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.328435+0000", 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102351 2024-04-25T05:50:03.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.328541+0000", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000106179 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333083+0000", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045415170000000001 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333619+0000", 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00053590300000000001 2024-04-25T05:50:03.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342162+0000", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0085432289999999994 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342221+0000", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8975999999999999e-05 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342265+0000", 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4539000000000002e-05 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.568 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233494 10.12 10:4e8549ae:::benchmark_data_smithi043_152558_object105144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.861830+0000", 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.709962738000002, 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.48057040899999998, 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:50:03.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233494 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861830+0000", 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861830+0000", 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861831+0000", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1119999999999999e-06 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861836+0000", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1499999999999998e-06 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861847+0000", 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1270000000000001e-05 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861853+0000", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4e-06 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861908+0000", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4997000000000002e-05 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.861916+0000", 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7870000000000007e-06 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.329564+0000", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.46764786200000003 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.329665+0000", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010067 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.329767+0000", 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101856 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333123+0000", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033560719999999999 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.333623+0000", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00050021500000000003 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342315+0000", 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086922419999999993 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342362+0000", 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6368000000000001e-05 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.342400+0000", 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8408000000000002e-05 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:50:03.575 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:50:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:03.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:03.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[27010]: pgmap v840: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:03.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:03.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:03.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:03.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[37492]: pgmap v840: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:03.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:03.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:03.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:03 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:03.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:03.701 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:03.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:03 smithi060 ceph-mon[30898]: pgmap v840: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 236 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 880/12012 objects misplaced (7.326%) 2024-04-25T05:50:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:03 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:03 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:03 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:03.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:03.823 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:03.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:03.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:03.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:03.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:04.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:04.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:04.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:04.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.334 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: osdmap e451: 8 total, 8 up, 7 in 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: 10.3 scrub starts 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[27010]: 10.3 scrub ok 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:50:04.427 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:50:04.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:04.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:04.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: osdmap e451: 8 total, 8 up, 7 in 2024-04-25T05:50:04.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: 10.3 scrub starts 2024-04-25T05:50:04.428 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:04 smithi043 ceph-mon[37492]: 10.3 scrub ok 2024-04-25T05:50:04.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:04.457 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:04.492 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:04.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:04.581 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:04.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:04.704 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: osdmap e451: 8 total, 8 up, 7 in 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: 10.3 scrub starts 2024-04-25T05:50:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:04 smithi060 ceph-mon[30898]: 10.3 scrub ok 2024-04-25T05:50:04.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:50:04.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:04.829 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:04.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:04.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:50:04.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:04.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:05.045 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_3' removed 2024-04-25T05:50:05.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:05.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:05.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:05.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:05.245 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:50:05.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:05.325 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:05.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:05.450 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:05.552 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.574 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:05.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: pgmap v842: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 37 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 553/2442 objects misplaced (22.645%) 2024-04-25T05:50:05.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 10.18 scrub starts 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 10.18 scrub ok 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 10.10 scrub starts 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 10.10 scrub ok 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: osdmap e452: 8 total, 8 up, 7 in 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[27010]: 6.1f scrub starts 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: pgmap v842: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 37 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 553/2442 objects misplaced (22.645%) 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 10.18 scrub starts 2024-04-25T05:50:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 10.18 scrub ok 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 10.10 scrub starts 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 10.10 scrub ok 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: osdmap e452: 8 total, 8 up, 7 in 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:05 smithi043 ceph-mon[37492]: 6.1f scrub starts 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:05.643 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: pgmap v842: 168 pgs: 4 active+remapped+backfill_toofull, 164 active+clean; 37 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 553/2442 objects misplaced (22.645%) 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 10.18 scrub starts 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 10.18 scrub ok 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 10.10 scrub starts 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 10.10 scrub ok 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2184366199' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: osdmap e452: 8 total, 8 up, 7 in 2024-04-25T05:50:05.644 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:05 smithi060 ceph-mon[30898]: 6.1f scrub starts 2024-04-25T05:50:05.675 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:05.698 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:50:05.790 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:50:05.790 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:50:05.790 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:50:05.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:05.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:50:05.822 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:05.924 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:05.945 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:05.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:05.972 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-25T05:50:05.973 INFO:tasks.radosbench:Beginning radosbench... 2024-04-25T05:50:05.973 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-25T05:50:05.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-04-25T05:50:06.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:06.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:06.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:06.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:06.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:06.303 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:06.318 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:06.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:06.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:06.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:06.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 6.1f scrub ok 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:50:06.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 6.1f scrub ok 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:50:06.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:06 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 6.1f scrub ok 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:50:06.648 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:06 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:50:06.649 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:50:06.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:06.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:06.752 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:06.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:06.810 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:06.897 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:50:06.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:06.933 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:06.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:07.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:07.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:07.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:07.178 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:07.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:07.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:07.400 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:50:07.400 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:50:07.401 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:50:07.401 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:50:07.401 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:50:07.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:07.426 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:07.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.551 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:07.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:07.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:07.673 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:07.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[27010]: pgmap v844: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[37492]: pgmap v844: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:07.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:07 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:50:07.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:07 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:50:07.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:07 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:50:07.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:07 smithi060 ceph-mon[30898]: pgmap v844: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:07.725 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:07 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-04-25T05:50:07.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.798 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:07.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:07.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:07.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:07.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:08.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:08.046 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:08.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:08.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:50:08.169 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:08.229 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:08.234 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:08.268 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:08.271 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:08.292 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:08.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:08.312 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:08.334 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:08.354 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:08.375 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:08.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:08.396 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:08.398 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:08.409 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:50:08.418 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:50:08.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: osdmap e453: 8 total, 8 up, 7 in 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:50:08.501 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 4.4 scrub starts 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: 4.4 scrub ok 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1391061230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: osdmap e453: 8 total, 8 up, 7 in 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:08.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:50:08.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:50:08.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 4.4 scrub starts 2024-04-25T05:50:08.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: 4.4 scrub ok 2024-04-25T05:50:08.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1391061230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:08.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:08.643 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:08.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:50:08.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:50:08.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:08.656 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: osdmap e453: 8 total, 8 up, 7 in 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 4.4 scrub starts 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: 4.4 scrub ok 2024-04-25T05:50:08.657 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1391061230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:08.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:08.787 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:08.889 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:08.899 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:50:08.899 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:50:08.910 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:08.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:09.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:09.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:09.155 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:09.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:09.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:09.400 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:09.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:09.523 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:09.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:50:09.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:09.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:50:09.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:50:09.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: pgmap v846: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:09.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:09.663 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: osdmap e454: 8 total, 8 up, 7 in 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 6.18 deep-scrub starts 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 6.18 deep-scrub ok 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: pgmap v846: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:09.664 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: osdmap e454: 8 total, 8 up, 7 in 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 6.18 deep-scrub starts 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 6.18 deep-scrub ok 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:50:09.665 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:50:09.748 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: pgmap v846: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1189596604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: osdmap e454: 8 total, 8 up, 7 in 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 6.18 deep-scrub starts 2024-04-25T05:50:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 6.18 deep-scrub ok 2024-04-25T05:50:09.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:50:09.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:09 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:50:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:09.871 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:09.892 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:09.938 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:50:09.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:10.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:10.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:10.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:10.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:10.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:10.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:10.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:10.518 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:10.519 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:10.621 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:10.642 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: osdmap e455: 8 total, 8 up, 7 in 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3865824199' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: osdmap e455: 8 total, 8 up, 7 in 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3865824199' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.745 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: osdmap e455: 8 total, 8 up, 7 in 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3865824199' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-25T05:50:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:10.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:10.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:10.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:10.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:10.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:11.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:11.037 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_4' created 2024-04-25T05:50:11.080 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:50:11.080 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:50:11.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.146 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:11.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:11.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:11.268 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:11.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:11.395 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:11.413 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=sqlstore.transactions t=2024-04-25T05:50:11.120611077Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-25T05:50:11.413 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=sqlstore.transactions t=2024-04-25T05:50:11.131274425Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-25T05:50:11.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:11.522 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:11.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:50:11.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[27010]: pgmap v848: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[27010]: 4.f scrub starts 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[27010]: 4.f scrub ok 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[27010]: osdmap e456: 8 total, 8 up, 7 in 2024-04-25T05:50:11.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[37492]: pgmap v848: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:11.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[37492]: 4.f scrub starts 2024-04-25T05:50:11.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[37492]: 4.f scrub ok 2024-04-25T05:50:11.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-25T05:50:11.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:11 smithi043 ceph-mon[37492]: osdmap e456: 8 total, 8 up, 7 in 2024-04-25T05:50:11.648 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:11.742 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:50:11.743 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:50:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-mon[30898]: pgmap v848: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:50:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-mon[30898]: 4.f scrub starts 2024-04-25T05:50:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-mon[30898]: 4.f scrub ok 2024-04-25T05:50:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-25T05:50:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:11 smithi060 ceph-mon[30898]: osdmap e456: 8 total, 8 up, 7 in 2024-04-25T05:50:11.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:11.774 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:11.876 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:11.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:11.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:11.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:12.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:12.147 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:12.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:12.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:12.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:50:12.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:12.405 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:12.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:12.531 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: osdmap e457: 8 total, 8 up, 7 in 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:50:12.539 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:50:12.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:50:12.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:50:12.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: osdmap e457: 8 total, 8 up, 7 in 2024-04-25T05:50:12.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:50:12.540 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:12 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:50:12.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-25T05:50:12.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:50:12.546 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:12.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:12.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: osdmap e457: 8 total, 8 up, 7 in 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:50:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:12 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:50:12.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:12.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:12.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:12.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:12.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:12.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:13.026 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:13.110 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:13.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:13.129 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:13.149 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:13.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:13.174 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:13.174 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:13.211 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:13.221 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:13.224 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:13.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:13.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:13.262 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:13.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:13.280 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:13.281 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:13.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:13.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:13.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:13.521 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:13.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:13.645 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:13.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: pgmap v851: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 4.5 deep-scrub starts 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 4.5 deep-scrub ok 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: osdmap e458: 8 total, 8 up, 7 in 2024-04-25T05:50:13.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 6.1b scrub starts 2024-04-25T05:50:13.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:50:13.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:50:13.751 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3364493143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:13.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:13.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:50:13.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:50:13.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: pgmap v851: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:50:13.752 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 4.5 deep-scrub starts 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 4.5 deep-scrub ok 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: osdmap e458: 8 total, 8 up, 7 in 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 6.1b scrub starts 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:50:13.753 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3364493143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: pgmap v851: 158 pgs: 16 unknown, 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 4.5 deep-scrub starts 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 4.5 deep-scrub ok 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: osdmap e458: 8 total, 8 up, 7 in 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 6.1b scrub starts 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:50:13.754 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3364493143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:13.768 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:13.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:13.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:13.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:13.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:13.998 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:50:13.998 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:50:13.998 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:50:13.998 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:50:13.998 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:50:14.015 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:14.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:14.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:14.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:14.263 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:14.365 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:14.385 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:14.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:14.488 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:14.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:14.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:14.632 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:14.733 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.1b scrub ok 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:50:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:50:14.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.c scrub starts 2024-04-25T05:50:14.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:14 smithi060 ceph-mon[30898]: 6.c scrub ok 2024-04-25T05:50:14.759 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:14.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:14.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:14.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.1b scrub ok 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:50:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.c scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[27010]: 6.c scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.1b scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.c scrub starts 2024-04-25T05:50:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:14 smithi043 ceph-mon[37492]: 6.c scrub ok 2024-04-25T05:50:14.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:15.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:15.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:15.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:15.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:15.260 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:15.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:15.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:15.436 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:50:15.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:15.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:15.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: pgmap v854: 158 pgs: 6 unknown, 152 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3837571270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: 3.6 scrub starts 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[27010]: 3.6 scrub ok 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: pgmap v854: 158 pgs: 6 unknown, 152 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3837571270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:50:15.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:50:15.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: 3.6 scrub starts 2024-04-25T05:50:15.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:15 smithi043 ceph-mon[37492]: 3.6 scrub ok 2024-04-25T05:50:15.633 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:15.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: pgmap v854: 158 pgs: 6 unknown, 152 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3837571270' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: 3.6 scrub starts 2024-04-25T05:50:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:15 smithi060 ceph-mon[30898]: 3.6 scrub ok 2024-04-25T05:50:15.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:15.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:15.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:15.882 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:15.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:15.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:16.005 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:16.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:16.132 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:16.234 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:16.259 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:16.361 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:16.441 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:50:16.488 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:16.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:16.640 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:16.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:50:16.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:50:16.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: osdmap e459: 8 total, 8 up, 7 in 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: 4.a deep-scrub starts 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[27010]: 4.a deep-scrub ok 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:16.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: osdmap e459: 8 total, 8 up, 7 in 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: 4.a deep-scrub starts 2024-04-25T05:50:16.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:16 smithi043 ceph-mon[37492]: 4.a deep-scrub ok 2024-04-25T05:50:16.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: osdmap e459: 8 total, 8 up, 7 in 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: 4.a deep-scrub starts 2024-04-25T05:50:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:16 smithi060 ceph-mon[30898]: 4.a deep-scrub ok 2024-04-25T05:50:16.766 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:16.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:16.889 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:16.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:16.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:17.017 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:17.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.140 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:17.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:17.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:50:17.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.266 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:17.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:17.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:17.390 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:17.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:17.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:50:17.449 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-25T05:50:17.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:17.517 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:17.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.640 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:17.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:17.741 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: pgmap v856: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: osdmap e460: 8 total, 8 up, 7 in 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4279625292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:50:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: 6.1d deep-scrub starts 2024-04-25T05:50:17.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:17 smithi060 ceph-mon[30898]: 6.1d deep-scrub ok 2024-04-25T05:50:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:17.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:17.861 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: pgmap v856: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:17.861 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:50:17.861 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: osdmap e460: 8 total, 8 up, 7 in 2024-04-25T05:50:17.861 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4279625292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.861 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: 6.1d deep-scrub starts 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[37492]: 6.1d deep-scrub ok 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: pgmap v856: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2441570137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: osdmap e460: 8 total, 8 up, 7 in 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4279625292' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: 6.1d deep-scrub starts 2024-04-25T05:50:17.862 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:17 smithi043 ceph-mon[27010]: 6.1d deep-scrub ok 2024-04-25T05:50:17.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:17.889 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:50:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:17.890 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:17.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:17.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:18.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:18.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:18.139 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:18.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:18.537 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:18.537 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:18.538 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:18.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:18.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:18.689 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:18 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:50:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:18 smithi060 ceph-mon[30898]: osdmap e461: 8 total, 8 up, 7 in 2024-04-25T05:50:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:18.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:18.815 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[27010]: osdmap e461: 8 total, 8 up, 7 in 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-25T05:50:18.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[37492]: osdmap e461: 8 total, 8 up, 7 in 2024-04-25T05:50:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:18.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:18.940 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:18.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:19.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:19.066 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:19.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:19.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:19.294 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:19.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:19.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:19.443 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:19.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:19.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:19.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:19.697 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:19.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:19 smithi043 ceph-mon[27010]: pgmap v859: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:19.719 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:19 smithi043 ceph-mon[27010]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:50:19.719 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:19 smithi043 ceph-mon[37492]: pgmap v859: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:19.719 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:19 smithi043 ceph-mon[37492]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:50:19.720 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:50:19.720 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:50:19.720 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:50:19.720 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:50:19.720 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:50:19.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184730 8.c 8:3895024e:::benchmark_data_smithi043_104001_object60606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.81629039000001, 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0106298600000001, 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184730 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.721 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902410+0000", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902411+0000", 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.483e-06 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902418+0000", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3269999999999997e-06 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902437+0000", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9673999999999998e-05 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902446+0000", 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4440000000000008e-06 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902550+0000", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000104485 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902561+0000", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0351e-05 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906055+0000", 2024-04-25T05:50:19.724 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.003494664 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906151+0000", 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5391999999999997e-05 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.906246+0000", 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5552999999999998e-05 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.725 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912838+0000", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.006591675 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912925+0000", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7159999999999999e-05 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912941+0000", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5968999999999999e-05 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.726 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.912992+0000", 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0841000000000003e-05 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913040+0000", 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7842e-05 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184731 8.c 8:3f348624:::benchmark_data_smithi043_104001_object60607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.81623630500002, 2024-04-25T05:50:19.727 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.010906069, 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184731 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902464+0000", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3099999999999998e-07 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902468+0000", 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.337e-06 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902473+0000", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5310000000000001e-06 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902479+0000", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4700000000000001e-06 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902585+0000", 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106054 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.902591+0000", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3790000000000003e-06 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908540+0000", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.00594837 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908663+0000", 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000123437 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.908776+0000", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112724 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913122+0000", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0043464560000000003 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913170+0000", 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7431e-05 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.732 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913290+0000", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000120698 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913330+0000", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0070000000000001e-05 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913370+0000", 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9580999999999999e-05 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.733 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184780 8.c 8:3fdf1b59:::benchmark_data_smithi043_104001_object60656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.76136553700002, 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95616667600000005, 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184780 2024-04-25T05:50:19.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957335+0000", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957336+0000", 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5799999999999999e-06 2024-04-25T05:50:19.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957344+0000", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.807e-06 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957356+0000", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2404e-05 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957365+0000", 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5040000000000002e-06 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957437+0000", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2350999999999999e-05 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.957448+0000", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0669999999999999e-05 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910672+0000", 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95322447200000004 2024-04-25T05:50:19.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910797+0000", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000124442 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.910926+0000", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128849 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913223+0000", 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022977219999999999 2024-04-25T05:50:19.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913420+0000", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019708099999999999 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913431+0000", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0312000000000001e-05 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913459+0000", 2024-04-25T05:50:19.739 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8577000000000001e-05 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913501+0000", 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1904999999999999e-05 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:184787 8.c 8:3320ee50:::benchmark_data_smithi043_104001_object60663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 493.75403431000001, 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95582188800000001, 2024-04-25T05:50:19.740 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 184787 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.741 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964666+0000", 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964667+0000", 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964673+0000", 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2299999999999999e-06 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.742 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964690+0000", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7629999999999999e-05 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964699+0000", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1390000000000001e-06 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964760+0000", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0217999999999998e-05 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.743 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:05.964769+0000", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7860000000000008e-06 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913564+0000", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94879425100000003 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913643+0000", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9556999999999997e-05 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.913717+0000", 2024-04-25T05:50:19.744 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4209999999999996e-05 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920254+0000", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065365060000000001 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920304+0000", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9512000000000003e-05 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920385+0000", 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1979999999999998e-05 2024-04-25T05:50:19.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.746 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.746 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920439+0000", 2024-04-25T05:50:19.746 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3928000000000003e-05 2024-04-25T05:50:19.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:06.920488+0000", 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8301999999999999e-05 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226338 8.6 8:6585b86d:::benchmark_data_smithi043_104001_object102214:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 459.82186380000002, 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.103761714, 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226338 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896836+0000", 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896840+0000", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2289999999999999e-06 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896861+0000", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1603999999999998e-05 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896888+0000", 2024-04-25T05:50:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7047999999999999e-05 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896903+0000", 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4888999999999999e-05 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019552+0000", 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.12264891999999999 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019561+0000", 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0690000000000008e-06 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000512+0000", 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98095072000000005 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.000598+0000", 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6235000000000004e-05 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:226349 8.6 8:6373a4df:::benchmark_data_smithi043_104001_object102225:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 459.82177105199997, 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.139697025, 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 226349 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896929+0000", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896930+0000", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4199999999999996e-07 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896933+0000", 2024-04-25T05:50:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.551e-06 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896939+0000", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1380000000000004e-06 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:39.896945+0000", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1959999999999999e-06 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019583+0000", 2024-04-25T05:50:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122638941 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:40.019591+0000", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4189999999999996e-06 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.002927+0000", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98333629700000003 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003056+0000", 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129037 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.003403+0000", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000346733 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.005083+0000", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0016799779999999999 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.008504+0000", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034209710000000001 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036533+0000", 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028029505999999999 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036586+0000", 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2278000000000003e-05 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:41.036626+0000", 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0438000000000001e-05 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228017 8.16 8:6e8160c2:::benchmark_data_smithi043_104001_object103893:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 456.74079970999998, 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.134225407, 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228017 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977900+0000", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977902+0000", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5659999999999999e-06 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977908+0000", 2024-04-25T05:50:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.592e-06 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977923+0000", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5132e-05 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977934+0000", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0869999999999999e-05 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978035+0000", 2024-04-25T05:50:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101284 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op must wait for map", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978042+0000", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5229999999999998e-06 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982071+0000", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004028472 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982081+0000", 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0589e-05 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104437+0000", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.122355209 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104572+0000", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013536399999999999 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112038+0000", 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0074657969999999997 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112126+0000", 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8009000000000004e-05 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25115.0:228023 8.16 8:6ad7e352:::benchmark_data_smithi043_104001_object103899:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 456.74074401000001, 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1340453349999999, 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25115", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 228023 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977956+0000", 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977957+0000", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9500000000000003e-07 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977960+0000", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3230000000000002e-06 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977967+0000", 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.977974+0000", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9659999999999999e-06 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978175+0000", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020133299999999999 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting_for_map not empty", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.978182+0000", 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1530000000000002e-06 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982727+0000", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045456259999999997 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:42.982737+0000", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3440000000000007e-06 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.104931+0000", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.12219398 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105055+0000", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012434499999999999 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,5,6,7", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.105352+0000", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297082 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.108987+0000", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00363503 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111435+0000", 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024477959999999999 2024-04-25T05:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111738+0000", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00030323699999999998 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111789+0000", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0565999999999997e-05 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111869+0000", 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0865999999999995e-05 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.111908+0000", 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8052000000000001e-05 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:42:44.112001+0000", 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3929000000000001e-05 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:50:19.771 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:19 smithi060 ceph-mon[30898]: pgmap v859: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:50:19.771 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:19 smithi060 ceph-mon[30898]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T05:50:19.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:19.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:19.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:19.943 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:19.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:20.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:20.067 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:20.139 DEBUG:teuthology.orchestra.run.smithi043:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-25T05:50:20.142 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-25T05:50:20.142 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-25T05:50:20.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:20.190 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:20.192 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:50:20.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:hints = 1 2024-04-25T05:50:20.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-25T05:50:20.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object prefix: benchmark_data_smithi043_179003 2024-04-25T05:50:20.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:50:20.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 0 0 0 0 0 0 - 0 2024-04-25T05:50:20.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:20.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:20.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:20.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:20.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:20.568 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:20.599 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:20.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:20.733 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:20.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:20.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:20.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:20.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:21.011 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:21.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:21.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:21.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 1 16 3256 3240 202.489 202.5 0.00649325 0.00492112 2024-04-25T05:50:21.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:21.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:50:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.281 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:21.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:21 smithi043 ceph-mon[27010]: pgmap v860: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:21.297 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:21 smithi043 ceph-mon[27010]: osdmap e462: 8 total, 8 up, 7 in 2024-04-25T05:50:21.298 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:21 smithi043 ceph-mon[37492]: pgmap v860: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:21.298 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:21 smithi043 ceph-mon[37492]: osdmap e462: 8 total, 8 up, 7 in 2024-04-25T05:50:21.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:21.404 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:21 smithi060 ceph-mon[30898]: pgmap v860: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:21 smithi060 ceph-mon[30898]: osdmap e462: 8 total, 8 up, 7 in 2024-04-25T05:50:21.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:21.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:21.681 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:21.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:21.813 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:21.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:21.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:21.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:21.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:22.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:22.055 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:50:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.117 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:22.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:22.170 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:22.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:50:22.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:50:22.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 2 16 6009 5993 187.239 172.062 0.00385283 0.00532841 2024-04-25T05:50:22.210 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:22.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:22.250 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:22.251 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:22.292 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:22.335 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:22.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:22.377 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:22.412 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:22.457 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:22.462 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:22.463 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:22.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:50:22.526 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:22.628 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:22.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:22.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:22.800 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:22.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:22.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:23.040 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:23.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:23.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:23.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 3 16 9110 9094 189.425 193.812 0.00367236 0.00527082 2024-04-25T05:50:23.213 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:23.315 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:23.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: pgmap v862: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: osdmap e463: 8 total, 8 up, 7 in 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2352468474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: pgmap v862: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:50:23.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: osdmap e463: 8 total, 8 up, 7 in 2024-04-25T05:50:23.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2352468474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:23.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:23.322 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:23 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:23.351 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:23.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:23.487 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: pgmap v862: 158 pgs: 158 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4069543503' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: osdmap e463: 8 total, 8 up, 7 in 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2352468474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:23.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:23.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:23.767 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:23.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:23.905 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:23.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:23.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:24.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:24.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:24.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:24.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 4 16 12021 12005 187.549 181.938 0.00852577 0.00532464 2024-04-25T05:50:24.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:50:24.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:24.302 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:24.344 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:24 smithi043 ceph-mon[27010]: osdmap e464: 8 total, 8 up, 7 in 2024-04-25T05:50:24.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:24 smithi043 ceph-mon[37492]: osdmap e464: 8 total, 8 up, 7 in 2024-04-25T05:50:24.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:24.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:24.483 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:24.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:24 smithi060 ceph-mon[30898]: osdmap e464: 8 total, 8 up, 7 in 2024-04-25T05:50:24.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:24.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:24.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:24.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:24.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:24.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:24.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:24.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.038 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:25.083 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:50:25.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:25.170 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:25.193 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 5 16 14772 14756 184.424 171.938 0.00830847 0.00541473 2024-04-25T05:50:25.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:25.310 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:25.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[27010]: pgmap v865: 158 pgs: 158 active+clean; 347 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2111974470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:25.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[37492]: pgmap v865: 158 pgs: 158 active+clean; 347 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-04-25T05:50:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2111974470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:25 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:25 smithi060 ceph-mon[30898]: pgmap v865: 158 pgs: 158 active+clean; 347 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 924 op/s 2024-04-25T05:50:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2111974470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:25 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:25.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:25.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:25.744 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:25.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:25.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:25.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:26.006 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:26.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:26.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:26.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 6 16 17136 17120 177.993 147.75 0.0157375 0.00558762 2024-04-25T05:50:26.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:26.282 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:26.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[27010]: osdmap e465: 8 total, 8 up, 7 in 2024-04-25T05:50:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:26.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:26 smithi043 ceph-mon[37492]: osdmap e465: 8 total, 8 up, 7 in 2024-04-25T05:50:26.416 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:26 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:26 smithi060 ceph-mon[30898]: osdmap e465: 8 total, 8 up, 7 in 2024-04-25T05:50:26.519 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:26.552 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:26.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:26.714 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:50:26.714 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:50:26.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:26.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:26.848 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:26.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:26.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:26.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:27.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:27.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:27.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 7 16 19672 19656 175.211 158.5 0.00547213 0.00570137 2024-04-25T05:50:27.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:27.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:27.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:27 smithi043 ceph-mon[27010]: pgmap v867: 158 pgs: 158 active+clean; 618 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 856 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:50:27.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:27 smithi043 ceph-mon[37492]: pgmap v867: 158 pgs: 158 active+clean; 618 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 856 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:50:27.392 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:27.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:27 smithi060 ceph-mon[30898]: pgmap v867: 158 pgs: 158 active+clean; 618 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 856 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-04-25T05:50:27.526 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:27.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:27.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:27.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:27.797 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:27.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:27.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:27.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:28.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:28.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:28.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:28.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:28.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 8 16 21998 21982 171.485 145.375 0.00318766 0.0058257 2024-04-25T05:50:28.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:28.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:28.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:28.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:28.703 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:28.724 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:28.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:28.856 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:28.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:28.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:28.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:29.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:29.128 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:29.204 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 9 16 24092 24076 166.977 130.875 0.0085055 0.0059829 2024-04-25T05:50:29.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:29.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:29.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:29 smithi043 ceph-mon[27010]: pgmap v868: 158 pgs: 158 active+clean; 770 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-25T05:50:29.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:29 smithi043 ceph-mon[37492]: pgmap v868: 158 pgs: 158 active+clean; 770 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-25T05:50:29.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:29.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:29 smithi060 ceph-mon[30898]: pgmap v868: 158 pgs: 158 active+clean; 770 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-25T05:50:29.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:29.544 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:29.646 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:29.679 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:29.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:29.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:29.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:29.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:29.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:30.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:30.088 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:30.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:50:30.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:30.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 10 16 26436 26420 164.93 146.5 0.00768686 0.00605739 2024-04-25T05:50:30.223 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:30.281 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:30.287 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:50:30.294 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:30.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:30.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:30.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:30.539 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:30.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:30.679 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:30.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:30.812 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:30.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:30.949 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:30.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:31.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:31.086 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:31.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:31.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 11 15 28416 28401 161.195 123.812 0.00985605 0.00619776 2024-04-25T05:50:31.207 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:31.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:31.251 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:31.297 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[27010]: pgmap v869: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[27010]: osdmap e466: 8 total, 8 up, 7 in 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[37492]: pgmap v869: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[37492]: osdmap e466: 8 total, 8 up, 7 in 2024-04-25T05:50:31.299 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:31 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:50:31.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:31.362 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:31.362 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:31.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:50:31.394 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:31.439 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:31.464 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:31.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:31.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:31 smithi060 ceph-mon[30898]: pgmap v869: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-04-25T05:50:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:31 smithi060 ceph-mon[30898]: osdmap e466: 8 total, 8 up, 7 in 2024-04-25T05:50:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:31 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:50:31.532 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:31.550 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:31.551 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:31.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:31.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:31.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:31.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:31.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:31.885 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:31.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:31.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:32.013 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:32.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:32.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:32.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:32.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 12 16 29991 29975 155.964 98.375 0.00715766 0.00640604 2024-04-25T05:50:32.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:32.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4180907977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4180907977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:32.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:32 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4180907977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:32.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.570 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:32.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:32.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:32.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:32.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:32.857 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:32.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:32.992 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:32.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:33.049 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[27010]: pgmap v871: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:50:33.057 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:50:33.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:33.129 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:33.205 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 13 16 32037 32021 153.804 127.875 0.0085412 0.00649684 2024-04-25T05:50:33.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:33.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:33.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/798215716' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[37492]: pgmap v871: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:50:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/798215716' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:33.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:33 smithi060 ceph-mon[30898]: pgmap v871: 158 pgs: 158 active+clean; 1.4 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-04-25T05:50:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/798215716' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:50:33.502 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:33.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:33.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:33.676 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:33.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:33.812 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:33.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:33.915 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:33.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:50:33.976 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:33.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:33.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:34.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:34.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:50:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[27010]: osdmap e467: 8 total, 8 up, 7 in 2024-04-25T05:50:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:34.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:50:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[37492]: osdmap e467: 8 total, 8 up, 7 in 2024-04-25T05:50:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:34.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:34.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 14 16 34048 34032 151.766 125.688 0.0108171 0.00658293 2024-04-25T05:50:34.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:34.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:34.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:34.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:34 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:50:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:34 smithi060 ceph-mon[30898]: osdmap e467: 8 total, 8 up, 7 in 2024-04-25T05:50:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:34.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:34.549 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:34.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:34.681 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:34.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:34.816 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:34.918 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:34.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:34.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:35.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:35.073 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[27010]: pgmap v873: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[27010]: osdmap e468: 8 total, 8 up, 7 in 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[37492]: pgmap v873: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.125 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:35 smithi043 ceph-mon[37492]: osdmap e468: 8 total, 8 up, 7 in 2024-04-25T05:50:35.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:35.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:50:35.208 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 15 16 35631 35615 148.246 98.9375 0.00831666 0.00673885 2024-04-25T05:50:35.211 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:35.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:35.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:35.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:35.476 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:35 smithi060 ceph-mon[30898]: pgmap v873: 158 pgs: 158 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-04-25T05:50:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:35 smithi060 ceph-mon[30898]: osdmap e468: 8 total, 8 up, 7 in 2024-04-25T05:50:35.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:35.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:35.741 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:35.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:35.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:35.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:35.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:36.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:36.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:36.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:36.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 16 16 37249 37233 145.287 101.125 0.0216037 0.00687574 2024-04-25T05:50:36.275 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:36.291 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:50:36.291 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:50:36.291 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:50:36.291 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:50:36.291 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:50:36.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:36.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:36.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:36.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:36.592 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:36.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:36.729 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:36.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:36.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:36.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:36.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:37.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:37.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[37492]: pgmap v875: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:50:37.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:50:37.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:50:37.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:37.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:37.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 17 16 39009 38993 143.213 110 0.00479823 0.0069767 2024-04-25T05:50:37.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[27010]: pgmap v875: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:50:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:50:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:50:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:37.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:37.440 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:37 smithi060 ceph-mon[30898]: pgmap v875: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-04-25T05:50:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:50:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:50:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:37.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:37.578 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:37.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:37.715 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:37.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:37.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:37.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:37.871 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:37.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:50:37.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:37.992 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:37.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:38.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:38.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:38 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4166152962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.111 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:38 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:50:38.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:38.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 18 16 40817 40801 141.535 113 0.00894244 0.00705755 2024-04-25T05:50:38.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.304 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:38.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:38 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4166152962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:38 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:38.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:38.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:38 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4166152962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:38 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:50:38.544 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:38.577 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:38.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:38.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:38.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:38.846 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:38.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:38.976 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:38.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:39.077 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:39.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:39.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 19 16 42628 42612 140.044 113.188 0.00916216 0.00713588 2024-04-25T05:50:39.242 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:39.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:39.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[27010]: pgmap v876: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 641 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:39.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:39.354 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:39.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[27010]: osdmap e469: 8 total, 8 up, 7 in 2024-04-25T05:50:39.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[37492]: pgmap v876: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 641 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:39.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:39.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:39.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-mon[37492]: osdmap e469: 8 total, 8 up, 7 in 2024-04-25T05:50:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:39.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:39 smithi060 ceph-mon[30898]: pgmap v876: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 641 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:39 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:39 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:50:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:39 smithi060 ceph-mon[30898]: osdmap e469: 8 total, 8 up, 7 in 2024-04-25T05:50:39.513 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:39.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:39.651 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:39.753 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:39.790 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:39.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:39.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:39.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:39.954 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:50:40.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:40.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:40.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:40.095 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:40.096 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:40.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:40.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:50:40.211423+0000 min lat: 0.00134748 max lat: 0.111275 avg lat: 0.00722617 2024-04-25T05:50:40.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:50:40.210 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 20 16 44305 44289 138.283 104.812 0.00919373 0.00722617 2024-04-25T05:50:40.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/36073573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: osdmap e470: 8 total, 8 up, 7 in 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1991579135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:50:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/36073573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: osdmap e470: 8 total, 8 up, 7 in 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1991579135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:40.474 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:40.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/36073573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: osdmap e470: 8 total, 8 up, 7 in 2024-04-25T05:50:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1991579135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:50:40.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:40.722 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:40.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:40.858 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:40.893 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:40.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:40.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:41.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:41.052 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:50:41.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:41.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 21 16 46120 46104 137.097 113.438 0.00886995 0.00728848 2024-04-25T05:50:41.212 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:41.314 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: pgmap v878: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: 4.7 scrub starts 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: 4.7 scrub ok 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:50:41.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[27010]: osdmap e471: 8 total, 8 up, 7 in 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: pgmap v878: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: 4.7 scrub starts 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: 4.7 scrub ok 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:50:41.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:41 smithi043 ceph-mon[37492]: osdmap e471: 8 total, 8 up, 7 in 2024-04-25T05:50:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:41.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:41.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:41.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: pgmap v878: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.44k op/s 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: 4.7 scrub starts 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: 4.7 scrub ok 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:50:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:41 smithi060 ceph-mon[30898]: osdmap e471: 8 total, 8 up, 7 in 2024-04-25T05:50:41.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:41.624 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:41.726 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:41.758 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:41.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:41.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:41.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:41.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:42.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:50:42.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:50:42.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:42.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:42.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:42.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 22 16 47864 47848 135.819 109 0.00903264 0.00735681 2024-04-25T05:50:42.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 4.19 scrub starts 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 4.19 scrub ok 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:50:42.291 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 4.19 scrub starts 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 4.19 scrub ok 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:50:42.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:42 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:50:42.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:42.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:42.460 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 4.19 scrub starts 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 4.19 scrub ok 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:50:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:42 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:50:42.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:42.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:42.695 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:42.696 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:50:42.697 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:50:42.697 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:42.723 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:42.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:42.857 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:42.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:42.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:43.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:43.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:43.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:50:43.141 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:43.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 23 16 49433 49417 134.178 98.0625 0.00878083 0.00744758 2024-04-25T05:50:43.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:43.274 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:43.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[27010]: pgmap v881: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:50:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:50:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:50:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:50:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[37492]: pgmap v881: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:50:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:50:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:50:43.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:43 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:50:43.406 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-mon[30898]: pgmap v881: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-04-25T05:50:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:50:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:50:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:50:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:50:43.508 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:43.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:50:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:43.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:43.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:43.716 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:43.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:43.865 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:43.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:43.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:43.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:44.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:44.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:44.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 24 16 51016 51000 132.71 98.9375 0.00692344 0.00752983 2024-04-25T05:50:44.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:44.307 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:44.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:44.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:44 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:50:44.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:44 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:50:44.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:44 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:50:44.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:44 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:50:44.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:44.418 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:44.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:44 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:50:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:44 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:50:44.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:44.576 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:44.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:44.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:44.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:44.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:44.981 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:44.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:45.049 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:50:45.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:45.115 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:45.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 25 15 52514 52499 131.15 93.6875 0.0151861 0.00761965 2024-04-25T05:50:45.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:45.252 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:45.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:45.388 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 5.16 scrub starts 2024-04-25T05:50:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 5.16 scrub ok 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: pgmap v882: 158 pgs: 158 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: osdmap e472: 8 total, 8 up, 7 in 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 5.16 scrub starts 2024-04-25T05:50:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 5.16 scrub ok 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: pgmap v882: 158 pgs: 158 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: osdmap e472: 8 total, 8 up, 7 in 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:50:45.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:45 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:50:45.490 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 5.16 scrub starts 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 5.16 scrub ok 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: pgmap v882: 158 pgs: 158 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:50:45.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:50:45.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/265426460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:50:45.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: osdmap e472: 8 total, 8 up, 7 in 2024-04-25T05:50:45.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:50:45.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:45 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:50:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:45.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:45.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:45.661 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:45.763 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:45.797 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:45.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:45.929 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:45.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:46.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:46.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:46.172 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:46.205 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:46.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 26 16 54095 54079 129.905 98.75 0.0168087 0.00769251 2024-04-25T05:50:46.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:46.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:50:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:50:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:50:46.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:50:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:50:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:50:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:50:46.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:46 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:50:46.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:46.471 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:46.489 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:46 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:50:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:46 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:50:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:46 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:50:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:46 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:50:46.533 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:46.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:46.578 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:46.604 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:46.620 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:46.660 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:46.696 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:46.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:46.741 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:46.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:46.781 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:46.788 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:46.789 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:46.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:46.881 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:46.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:46.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:47.017 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:47.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:47.188 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:50:47.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 27 15 55550 55535 128.464 91 0.00769576 0.00777901 2024-04-25T05:50:47.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:47.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:47.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: pgmap v884: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/676837902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:50:47.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: pgmap v884: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/676837902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:50:47.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:47 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:50:47.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:47.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: pgmap v884: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/676837902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:50:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:47 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:50:47.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:47.580 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:47.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:47.715 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:47.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:47.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:47.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:47.991 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:47.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:48.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:48.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:48.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:48.211 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 28 16 57205 57189 127.568 103.375 0.00698934 0.00783367 2024-04-25T05:50:48.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:48.264 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:48.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:48.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[27010]: 3.d deep-scrub starts 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[27010]: 3.d deep-scrub ok 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[27010]: osdmap e473: 8 total, 8 up, 7 in 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[37492]: 3.d deep-scrub starts 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[37492]: 3.d deep-scrub ok 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:50:48.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:48 smithi043 ceph-mon[37492]: osdmap e473: 8 total, 8 up, 7 in 2024-04-25T05:50:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:48.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:48.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:48 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:50:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:48 smithi060 ceph-mon[30898]: 3.d deep-scrub starts 2024-04-25T05:50:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:48 smithi060 ceph-mon[30898]: 3.d deep-scrub ok 2024-04-25T05:50:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2453690827' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:50:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:48 smithi060 ceph-mon[30898]: osdmap e473: 8 total, 8 up, 7 in 2024-04-25T05:50:48.512 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:48.547 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:48.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:48.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:50:48.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:48.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:48.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:48.828 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:49.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:58210 11.f e473/456 11:fd55b9d5:::benchmark_data_smithi043_179003_object58209:head v 473'3685, mlcod=473'3681)", 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.937307+0000", 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.013222726000000001, 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013254683, 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.087 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937307+0000", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937307+0000", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937307+0000", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8699999999999995e-07 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937335+0000", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7974e-05 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937345+0000", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4390000000000006e-06 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937352+0000", 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3590000000000002e-06 2024-04-25T05:50:49.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937417+0000", 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5260999999999997e-05 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.937437+0000", 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.0335000000000001e-05 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58212 11.6 11:69c75fd9:::benchmark_data_smithi043_179003_object58211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.938004+0000", 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.012524802999999999, 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012655530999999999, 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58212 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938004+0000", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938004+0000", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938006+0000", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.096e-06 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938028+0000", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1991999999999999e-05 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938037+0000", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4919999999999999e-06 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938045+0000", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9750000000000007e-06 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938107+0000", 2024-04-25T05:50:49.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2219999999999997e-05 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938271+0000", 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016387300000000001 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938430+0000", 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015888799999999999 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944730+0000", 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0063003979999999996 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.948252+0000", 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0035214949999999999 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58214 11.6 11:66e6a94f:::benchmark_data_smithi043_179003_object58213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.938483+0000", 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.012046559, 2024-04-25T05:50:49.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012284223, 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58214 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938483+0000", 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938483+0000", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938483+0000", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6599999999999995e-07 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938505+0000", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2002e-05 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938515+0000", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1519999999999998e-06 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938523+0000", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1470000000000001e-06 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.941128+0000", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002605422 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.941267+0000", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000139116 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.941417+0000", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014999800000000001 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944780+0000", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033625759999999999 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.948258+0000", 2024-04-25T05:50:49.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0034785430000000002 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:58211 11.f e473/456 11:f603c43f:::benchmark_data_smithi043_179003_object58210:head v 473'3686, mlcod=473'3681)", 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.938844+0000", 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.011684958, 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012026113, 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "started", 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938844+0000", 2024-04-25T05:50:49.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938844+0000", 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938845+0000", 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7100000000000002e-07 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938952+0000", 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000107341 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938961+0000", 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9819999999999997e-06 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.938968+0000", 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0589999999999997e-06 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.943016+0000", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040480070000000002 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.949046+0000", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0060298419999999997 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58216 11.6 11:68f27eb8:::benchmark_data_smithi043_179003_object58215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.939284+0000", 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.011245359, 2024-04-25T05:50:49.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.011663153000000001, 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58216 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939284+0000", 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939284+0000", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939285+0000", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1000000000000001e-06 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939307+0000", 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1607000000000002e-05 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939321+0000", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4647999999999999e-05 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.939330+0000", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5320000000000006e-06 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944815+0000", 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0054855920000000001 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.945109+0000", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00029403699999999998 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.945311+0000", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020139999999999999 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.948646+0000", 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033352239999999999 2024-04-25T05:50:49.107 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58219 11.2 11:4d4472d4:::benchmark_data_smithi043_179003_object58218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.942408+0000", 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0081214430000000008, 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0086334370000000007, 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58219 2024-04-25T05:50:49.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942408+0000", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942408+0000", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942409+0000", 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.517e-06 2024-04-25T05:50:49.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942442+0000", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2348999999999998e-05 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942454+0000", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.207e-05 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942462+0000", 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2130000000000008e-06 2024-04-25T05:50:49.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942616+0000", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015354 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942754+0000", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013860700000000001 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942903+0000", 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014913799999999999 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.945993+0000", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030900350000000001 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.946628+0000", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00063477300000000004 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58221 11.d 11:bde11db1:::benchmark_data_smithi043_179003_object58220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.942487+0000", 2024-04-25T05:50:49.112 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.008042034, 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0086558880000000005, 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58221 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942487+0000", 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942487+0000", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942488+0000", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8100000000000003e-07 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942539+0000", 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1107999999999999e-05 2024-04-25T05:50:49.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942548+0000", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9439999999999994e-06 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.942555+0000", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1779999999999997e-06 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944924+0000", 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023693210000000002 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.945052+0000", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012795100000000001 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.945195+0000", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000142299 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947609+0000", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024140059999999998 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:50:49.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.948693+0000", 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001084453 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:58218 11.c e473/456 11:30abf886:::benchmark_data_smithi043_179003_object58217:head v 473'3799, mlcod=473'3797)", 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.944384+0000", 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0061454639999999998, 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0068556479999999998, 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.117 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944384+0000", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944384+0000", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944385+0000", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9599999999999998e-07 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944411+0000", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6821e-05 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944422+0000", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0281e-05 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944429+0000", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6699999999999994e-06 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944516+0000", 2024-04-25T05:50:49.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6378999999999994e-05 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.944536+0000", 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9890000000000001e-05 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.951234+0000", 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066987119999999999 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:58222 11.2 11:4d96ae5e:::benchmark_data_smithi043_179003_object58221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:50:48.947382+0000", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.003147356, 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003934524, 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 58222 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:50:49.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947382+0000", 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947382+0000", 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947383+0000", 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4920000000000001e-06 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947444+0000", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1037000000000005e-05 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947459+0000", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4567e-05 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947469+0000", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7340000000000003e-06 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:50:49.123 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947534+0000", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4978e-05 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947695+0000", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016167800000000001 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.947841+0000", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000145165 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:50:49.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:50:48.951229+0000", 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0033887190000000001 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 9 2024-04-25T05:50:49.125 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:50:49.125 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:50:49.187 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:49.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 29 16 58618 58602 126.21 88.3125 0.0208457 0.0079178 2024-04-25T05:50:49.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:49.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:49.379 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: pgmap v886: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 734 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 4.1e deep-scrub starts 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 4.1e deep-scrub ok 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:50:49.435 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 6.b scrub starts 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[37492]: 6.b scrub ok 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: pgmap v886: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 734 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 4.1e deep-scrub starts 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 4.1e deep-scrub ok 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 6.b scrub starts 2024-04-25T05:50:49.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:49.437 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:49.437 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-mon[27010]: 6.b scrub ok 2024-04-25T05:50:49.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: pgmap v886: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 734 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 4.1e deep-scrub starts 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 4.1e deep-scrub ok 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 6.b scrub starts 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:50:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:50:49.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:49 smithi060 ceph-mon[30898]: 6.b scrub ok 2024-04-25T05:50:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:49.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:49.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:49.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:49.759 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:49.794 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:49.861 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:50:49.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:49.920 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:50:49.928 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:49.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:50.027 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:50.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:50.059 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:50.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:50:50.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:50.196 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:50.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 30 16 60067 60051 125.021 90.5625 0.00358383 0.00799403 2024-04-25T05:50:50.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:50.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:50.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:50.466 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:50.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:50.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:50.707 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:50.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:50.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:50.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:50.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.021 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:51.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:51.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:51.161 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:51.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 31 16 61473 61457 123.823 87.875 0.00839677 0.00807145 2024-04-25T05:50:51.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:51.296 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:51.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[27010]: 5.13 scrub starts 2024-04-25T05:50:51.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[27010]: 5.13 scrub ok 2024-04-25T05:50:51.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[27010]: pgmap v887: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-04-25T05:50:51.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[27010]: osdmap e474: 8 total, 8 up, 7 in 2024-04-25T05:50:51.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[37492]: 5.13 scrub starts 2024-04-25T05:50:51.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[37492]: 5.13 scrub ok 2024-04-25T05:50:51.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[37492]: pgmap v887: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-04-25T05:50:51.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:51 smithi043 ceph-mon[37492]: osdmap e474: 8 total, 8 up, 7 in 2024-04-25T05:50:51.398 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:51.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:51 smithi060 ceph-mon[30898]: 5.13 scrub starts 2024-04-25T05:50:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:51 smithi060 ceph-mon[30898]: 5.13 scrub ok 2024-04-25T05:50:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:51 smithi060 ceph-mon[30898]: pgmap v887: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-04-25T05:50:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:51 smithi060 ceph-mon[30898]: osdmap e474: 8 total, 8 up, 7 in 2024-04-25T05:50:51.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:51.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:51.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:51.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:51.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:51.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:51.928 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:51.965 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:51.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:52.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:52.092 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:52.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:52.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 32 16 62734 62718 122.417 78.8125 0.0118986 0.00816322 2024-04-25T05:50:52.224 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:52.265 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:50:52.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:52.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[27010]: 3.1c deep-scrub starts 2024-04-25T05:50:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[27010]: 3.1c deep-scrub ok 2024-04-25T05:50:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:50:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:50:52.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:50:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[37492]: 3.1c deep-scrub starts 2024-04-25T05:50:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[37492]: 3.1c deep-scrub ok 2024-04-25T05:50:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:50:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:50:52.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:50:52.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:52.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:52 smithi060 ceph-mon[30898]: 3.1c deep-scrub starts 2024-04-25T05:50:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:52 smithi060 ceph-mon[30898]: 3.1c deep-scrub ok 2024-04-25T05:50:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:52 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:50:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:52 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:50:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:50:52.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.616 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:52.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:52.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:52.754 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:52.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:52.892 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:52.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:52.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:53.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:53.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:53.165 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:53.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 33 16 64109 64093 121.312 85.9375 0.00705938 0.00823806 2024-04-25T05:50:53.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:53.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: 5.14 scrub starts 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: 5.14 scrub ok 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: 6.17 deep-scrub starts 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: pgmap v889: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: osdmap e475: 8 total, 8 up, 7 in 2024-04-25T05:50:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[27010]: 6.7 scrub starts 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: 5.14 scrub starts 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: 5.14 scrub ok 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: 6.17 deep-scrub starts 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: pgmap v889: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: osdmap e475: 8 total, 8 up, 7 in 2024-04-25T05:50:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:53 smithi043 ceph-mon[37492]: 6.7 scrub starts 2024-04-25T05:50:53.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:53.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: 5.14 scrub starts 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: 5.14 scrub ok 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: 6.17 deep-scrub starts 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: pgmap v889: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 734 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1958369125' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: osdmap e475: 8 total, 8 up, 7 in 2024-04-25T05:50:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-mon[30898]: 6.7 scrub starts 2024-04-25T05:50:53.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:53.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:53.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:53.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:53.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:53.842 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:53.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:53.979 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:53.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:53.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:50:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:50:54.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.106 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:54.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:54.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:54.213 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 34 16 65282 65266 119.901 73.3125 0.0115483 0.00833431 2024-04-25T05:50:54.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:54.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:54.377 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 6.17 deep-scrub ok 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 4.11 scrub starts 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 4.11 scrub ok 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 6.7 scrub ok 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:50:54.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 6.17 deep-scrub ok 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 4.11 scrub starts 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 4.11 scrub ok 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 6.7 scrub ok 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:50:54.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:54 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:50:54.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 6.17 deep-scrub ok 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 4.11 scrub starts 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 4.11 scrub ok 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 6.7 scrub ok 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:50:54.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:54 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:50:54.509 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:54.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:54.654 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:54.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:54.785 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:54.886 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:54.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:55.024 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:55.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:55.197 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:50:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.208 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:55.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:55.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 35 16 66717 66701 119.038 89.6875 0.00897363 0.008396 2024-04-25T05:50:55.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:55.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:55.386 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:55.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[27010]: pgmap v891: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-04-25T05:50:55.387 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[27010]: osdmap e476: 8 total, 8 up, 7 in 2024-04-25T05:50:55.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:55.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[37492]: pgmap v891: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-04-25T05:50:55.387 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:55 smithi043 ceph-mon[37492]: osdmap e476: 8 total, 8 up, 7 in 2024-04-25T05:50:55.451 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:55.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:50:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:55 smithi060 ceph-mon[30898]: pgmap v891: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-04-25T05:50:55.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:55 smithi060 ceph-mon[30898]: osdmap e476: 8 total, 8 up, 7 in 2024-04-25T05:50:55.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.616 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:55.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:55.659 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:55.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:55.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:55.750 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:55.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:55.880 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:55.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:55.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.020 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:56.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:56.095 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:50:56.121 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:56.214 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 36 16 67951 67935 117.874 77.125 0.00450049 0.00847872 2024-04-25T05:50:56.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:56.294 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3240363088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:50:56.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3240363088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.347 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:50:56.438 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3240363088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:50:56.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.575 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:56.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:56.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:56.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:56.854 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:56.920 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:50:56.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:56.973 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:50:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:56.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:50:57.026 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:50:57.069 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:50:57.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:57.116 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:50:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:50:57.162 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:50:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:50:57.164 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:50:57.213 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:50:57.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: pgmap v893: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T05:50:57.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:50:57.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:50:57.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:57.215 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:57.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: osdmap e477: 8 total, 8 up, 7 in 2024-04-25T05:50:57.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/470564943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:57.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 37 16 69301 69285 116.965 84.375 0.00551571 0.00854489 2024-04-25T05:50:57.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: pgmap v893: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T05:50:57.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:50:57.216 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:50:57.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:57.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:57.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: osdmap e477: 8 total, 8 up, 7 in 2024-04-25T05:50:57.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:57 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/470564943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:57.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:57.254 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:50:57.258 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:50:57.259 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:50:57.267 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:57.368 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:57.406 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: pgmap v893: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.13k op/s 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: osdmap e477: 8 total, 8 up, 7 in 2024-04-25T05:50:57.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:57 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/470564943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:50:57.508 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:57.541 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:57.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:57.677 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:57.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:57.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:57.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:50:57.953 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:57.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:50:58.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:58.087 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:58.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:58.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 38 16 70705 70689 116.196 87.75 0.00833217 0.00860049 2024-04-25T05:50:58.220 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:58.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:58.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:58.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:50:58.497 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:50:58.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:58.627 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:58.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:58.759 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:58.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:58.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:58.893 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:58.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:58.994 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:59.028 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:59.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:59.161 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:59.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 39 16 72166 72150 115.559 91.3125 0.00745727 0.00864814 2024-04-25T05:50:59.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:50:59.300 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:50:59.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:50:59 smithi043 ceph-mon[27010]: pgmap v895: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:50:59.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:50:59 smithi043 ceph-mon[37492]: pgmap v895: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:50:59.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:50:59.426 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:50:59.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:50:59 smithi060 ceph-mon[30898]: pgmap v895: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-25T05:50:59.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:50:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:50:59.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:59.696 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:59.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:50:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.831 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:50:59.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:50:59.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:50:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:50:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:50:59.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:00.022 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:50:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:50:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:00.069 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:00.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:00.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:51:00.216771+0000 min lat: 0.00134748 max lat: 0.115213 avg lat: 0.00872584 2024-04-25T05:51:00.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:51:00.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 40 16 73364 73348 114.541 74.875 0.00512013 0.00872584 2024-04-25T05:51:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:00.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:00.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:00.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:51:00.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:00.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:00.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:00.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:00.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:00.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:00.774 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:00.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:00.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:00.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:01.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:01.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:01.165 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:01.215 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 41 16 74457 74441 113.414 68.3125 0.00815734 0.00881197 2024-04-25T05:51:01.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:01.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:01 smithi043 ceph-mon[27010]: pgmap v896: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-04-25T05:51:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:01 smithi043 ceph-mon[27010]: osdmap e478: 8 total, 8 up, 7 in 2024-04-25T05:51:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:01 smithi043 ceph-mon[37492]: pgmap v896: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-04-25T05:51:01.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:01 smithi043 ceph-mon[37492]: osdmap e478: 8 total, 8 up, 7 in 2024-04-25T05:51:01.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:01.442 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:01 smithi060 ceph-mon[30898]: pgmap v896: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-04-25T05:51:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:01 smithi060 ceph-mon[30898]: osdmap e478: 8 total, 8 up, 7 in 2024-04-25T05:51:01.543 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:01.568 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:01.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:01.706 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:01.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:01.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:01.980 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:01.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:02.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:02.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:02.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 42 16 75817 75801 112.738 85 0.00514869 0.00886532 2024-04-25T05:51:02.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:02.251 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:02.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:02.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:02.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:02.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:02.534 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:02.636 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:02.673 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:02.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:02.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:02.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:02.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:02.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:02.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:03.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:03.079 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:03.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:03.214 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:03.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 43 16 77204 77188 112.132 86.6875 0.00952932 0.00891322 2024-04-25T05:51:03.317 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:03.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:03 smithi043 ceph-mon[27010]: pgmap v898: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 733 B/s rd, 60 MiB/s wr, 965 op/s 2024-04-25T05:51:03.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:03 smithi043 ceph-mon[37492]: pgmap v898: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 733 B/s rd, 60 MiB/s wr, 965 op/s 2024-04-25T05:51:03.355 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:03.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:03.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:03 smithi060 ceph-mon[30898]: pgmap v898: 158 pgs: 158 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 733 B/s rd, 60 MiB/s wr, 965 op/s 2024-04-25T05:51:03.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:03.628 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:03.730 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:03.734 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:51:03.768 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:03.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:03.913 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:03.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:03.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:04.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:04.069 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:04.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:04.206 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:04.216 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 44 16 78525 78509 111.46 82.5625 0.0147946 0.00896669 2024-04-25T05:51:04.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:04.332 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:04.337 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:04.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:04.471 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:04.573 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:04.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:04.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:04.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:04.747 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:04.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:04.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:04.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:04.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:04.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:05.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:05.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:51:05.158 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:05.188 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:05.217 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 45 16 79713 79697 110.63 74.25 0.00775456 0.00903316 2024-04-25T05:51:05.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:05.319 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:05.421 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:05.463 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:51:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:05.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:05.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:05.630 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[27010]: pgmap v899: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/879406694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[37492]: pgmap v899: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/879406694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:05 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:05 smithi060 ceph-mon[30898]: pgmap v899: 158 pgs: 158 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-25T05:51:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/879406694' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:05 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:51:05.766 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:05.824 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:05.824 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:05.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:05.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:05.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:06.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:06.145 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:06.181 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:06.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 46 16 80962 80946 109.922 78.0625 0.00566836 0.00909274 2024-04-25T05:51:06.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:06.317 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:06.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:06.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:06.472 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:51:06.480 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:51:06.481 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[27010]: osdmap e479: 8 total, 8 up, 7 in 2024-04-25T05:51:06.481 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:06.481 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:51:06.481 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[37492]: osdmap e479: 8 total, 8 up, 7 in 2024-04-25T05:51:06.481 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:06.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:06.594 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:06.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:06.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:51:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:51:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:06 smithi060 ceph-mon[30898]: osdmap e479: 8 total, 8 up, 7 in 2024-04-25T05:51:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:06.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:06.865 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:06.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:06.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:07.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:07.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:07.136 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:07.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 47 16 82342 82326 109.416 86.25 0.0101894 0.00913455 2024-04-25T05:51:07.238 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:07.281 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:07.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:07.520 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:07.557 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:07.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[27010]: pgmap v901: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:51:07.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:07.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:07.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[27010]: osdmap e480: 8 total, 8 up, 7 in 2024-04-25T05:51:07.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[37492]: pgmap v901: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:51:07.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:07.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:07.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:07 smithi043 ceph-mon[37492]: osdmap e480: 8 total, 8 up, 7 in 2024-04-25T05:51:07.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:07.727 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:07 smithi060 ceph-mon[30898]: pgmap v901: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-25T05:51:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:07 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/175466124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:07 smithi060 ceph-mon[30898]: osdmap e480: 8 total, 8 up, 7 in 2024-04-25T05:51:07.829 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:07.865 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:07.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:07.966 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:08.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:08.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:08.153 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:08.196 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:51:08.196 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:51:08.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 48 16 83701 83685 108.907 84.9375 0.00820807 0.00917718 2024-04-25T05:51:08.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.312 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:08.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:08.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:08.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:08.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:08.698 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:08.729 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:08.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:08.851 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:08.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:08.906 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:08.906 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:08.928 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:08.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:08.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:08.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:09.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:09.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:09.218 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 49 16 84985 84969 108.322 80.25 0.00477678 0.00922702 2024-04-25T05:51:09.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:09.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:09.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:09.412 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:09.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:09.546 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:09.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:09 smithi043 ceph-mon[27010]: pgmap v903: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 642 B/s rd, 61 MiB/s wr, 975 op/s 2024-04-25T05:51:09.567 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:09 smithi043 ceph-mon[37492]: pgmap v903: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 642 B/s rd, 61 MiB/s wr, 975 op/s 2024-04-25T05:51:09.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:51:09.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:09.690 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:09.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:09 smithi060 ceph-mon[30898]: pgmap v903: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 642 B/s rd, 61 MiB/s wr, 975 op/s 2024-04-25T05:51:09.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:09.841 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:09.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:09.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:09.948 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:09.978 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:09.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:09.988 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:10.026 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:10.067 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:10.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:10.109 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:10.125 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:10.157 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:10.199 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:10.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 50 15 86200 86185 107.675 76 0.0199251 0.00928201 2024-04-25T05:51:10.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:10.240 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:10.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:10.363 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:10.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:10.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:10.547 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:10.649 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:10.684 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:51:10.695 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:10.697 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:51:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2581092735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:10 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:10 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:10.797 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:10.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2581092735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:10.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2581092735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:10.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:10 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:10.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:10.971 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:10.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:11.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:11.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:11.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:11.219 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 51 16 87503 87487 107.159 81.375 0.0214685 0.00932688 2024-04-25T05:51:11.244 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:11.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.380 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:11.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:11.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:11.515 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:11.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:11.652 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:11 smithi060 ceph-mon[30898]: pgmap v904: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-04-25T05:51:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:11 smithi060 ceph-mon[30898]: osdmap e481: 8 total, 8 up, 7 in 2024-04-25T05:51:11.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.787 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:11.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:11.853 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:11 smithi043 ceph-mon[27010]: pgmap v904: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-04-25T05:51:11.853 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:11 smithi043 ceph-mon[27010]: osdmap e481: 8 total, 8 up, 7 in 2024-04-25T05:51:11.853 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:11 smithi043 ceph-mon[37492]: pgmap v904: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-04-25T05:51:11.853 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:11 smithi043 ceph-mon[37492]: osdmap e481: 8 total, 8 up, 7 in 2024-04-25T05:51:11.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:11.922 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:51:11.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:11.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:12.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:12.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:12.189 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:12.220 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 52 16 88863 88847 106.732 85 0.00703296 0.00936499 2024-04-25T05:51:12.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:12.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:12.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:12.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:12.495 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:12.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:12.526 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":905,"stamp":"2024-04-25T05:51:10.472587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5508110875,"num_objects":84268,"num_object_clones":0,"num_object_copies":252804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84268,"num_whiteouts":0,"num_read":2424,"num_read_kb":2144,"num_write":84952,"num_write_kb":5378477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":626,"num_bytes_recovered":1198290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74151,"ondisk_log_size":74151,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21691224,"kb_used_data":16072476,"kb_used_omap":10,"kb_used_meta":5618677,"kb_avail":634524840,"statfs":{"total":671965249536,"available":649753436160,"internally_reserved":0,"allocated":16458215424,"data_stored":16454170207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5753525355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":586153984,"num_objects":8944,"num_object_clones":0,"num_object_copies":26832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8944,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8944,"num_write_kb":572416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.450331"},"pg_stats":[{"pgid":"11.3","version":"480'5236","reported_seq":5292,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.556436+0000","last_change":"2024-04-25T05:50:12.040793+0000","last_active":"2024-04-25T05:51:08.556436+0000","last_peered":"2024-04-25T05:51:08.556436+0000","last_clean":"2024-04-25T05:51:08.556436+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:51:08.556436+0000","last_undegraded":"2024-04-25T05:51:08.556436+0000","last_fullsized":"2024-04-25T05:51:08.556436+0000","mapping_epoch":456,"log_start":"462'300","ondisk_log_start":"462'300","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:11.025188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":335104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":375,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468305+0000","last_change":"2024-04-25T05:50:08.705811+0000","last_active":"2024-04-25T05:51:06.468305+0000","last_peered":"2024-04-25T05:51:06.468305+0000","last_clean":"2024-04-25T05:51:06.468305+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:51:06.468305+0000","last_undegraded":"2024-04-25T05:51:06.468305+0000","last_fullsized":"2024-04-25T05:51:06.468305+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:50:08.705677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.705677+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.222090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":893,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471722+0000","last_change":"2024-04-25T05:49:05.117179+0000","last_active":"2024-04-25T05:51:06.471722+0000","last_peered":"2024-04-25T05:51:06.471722+0000","last_clean":"2024-04-25T05:51:06.471722+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:51:06.471722+0000","last_undegraded":"2024-04-25T05:51:06.471722+0000","last_fullsized":"2024-04-25T05:51:06.471722+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":305,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469750+0000","last_change":"2024-04-25T05:50:12.546520+0000","last_active":"2024-04-25T05:51:06.469750+0000","last_peered":"2024-04-25T05:51:06.469750+0000","last_clean":"2024-04-25T05:51:06.469750+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:51:06.469750+0000","last_undegraded":"2024-04-25T05:51:06.469750+0000","last_fullsized":"2024-04-25T05:51:06.469750+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:50:12.546342+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.546342+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.973854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":452,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469891+0000","last_change":"2024-04-25T05:50:13.709776+0000","last_active":"2024-04-25T05:51:06.469891+0000","last_peered":"2024-04-25T05:51:06.469891+0000","last_clean":"2024-04-25T05:51:06.469891+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:51:06.469891+0000","last_undegraded":"2024-04-25T05:51:06.469891+0000","last_fullsized":"2024-04-25T05:51:06.469891+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.709675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.709675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.101183+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1207,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471985+0000","last_change":"2024-04-25T05:49:33.878907+0000","last_active":"2024-04-25T05:51:06.471985+0000","last_peered":"2024-04-25T05:51:06.471985+0000","last_clean":"2024-04-25T05:51:06.471985+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:51:06.471985+0000","last_undegraded":"2024-04-25T05:51:06.471985+0000","last_fullsized":"2024-04-25T05:51:06.471985+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:33.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:33.878785+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907776+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"480'5197","reported_seq":5255,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:07.696911+0000","last_change":"2024-04-25T05:50:12.040238+0000","last_active":"2024-04-25T05:51:07.696911+0000","last_peered":"2024-04-25T05:51:07.696911+0000","last_clean":"2024-04-25T05:51:07.696911+0000","last_became_active":"2024-04-25T05:50:12.039934+0000","last_became_peered":"2024-04-25T05:50:12.039934+0000","last_unstale":"2024-04-25T05:51:07.696911+0000","last_undegraded":"2024-04-25T05:51:07.696911+0000","last_fullsized":"2024-04-25T05:51:07.696911+0000","mapping_epoch":456,"log_start":"462'300","ondisk_log_start":"462'300","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4897,"log_dups_size":0,"ondisk_log_size":4897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.997447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340590592,"num_objects":5197,"num_object_clones":0,"num_object_copies":15591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5197,"num_write_kb":332608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":376,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469192+0000","last_change":"2024-04-25T05:49:09.556929+0000","last_active":"2024-04-25T05:51:06.469192+0000","last_peered":"2024-04-25T05:51:06.469192+0000","last_clean":"2024-04-25T05:51:06.469192+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:51:06.469192+0000","last_undegraded":"2024-04-25T05:51:06.469192+0000","last_fullsized":"2024-04-25T05:51:06.469192+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:48:59.073429+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.073429+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716444+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":606,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475020+0000","last_change":"2024-04-25T05:49:32.886598+0000","last_active":"2024-04-25T05:51:06.475020+0000","last_peered":"2024-04-25T05:51:06.475020+0000","last_clean":"2024-04-25T05:51:06.475020+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:51:06.475020+0000","last_undegraded":"2024-04-25T05:51:06.475020+0000","last_fullsized":"2024-04-25T05:51:06.475020+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.886504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.886504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":906,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.481435+0000","last_change":"2024-04-25T05:50:55.073084+0000","last_active":"2024-04-25T05:51:06.481435+0000","last_peered":"2024-04-25T05:51:06.481435+0000","last_clean":"2024-04-25T05:51:06.481435+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:51:06.481435+0000","last_undegraded":"2024-04-25T05:51:06.481435+0000","last_fullsized":"2024-04-25T05:51:06.481435+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:15.963221+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":242,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.470319+0000","last_change":"2024-04-25T05:50:46.862960+0000","last_active":"2024-04-25T05:51:06.470319+0000","last_peered":"2024-04-25T05:51:06.470319+0000","last_clean":"2024-04-25T05:51:06.470319+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:51:06.470319+0000","last_undegraded":"2024-04-25T05:51:06.470319+0000","last_fullsized":"2024-04-25T05:51:06.470319+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:46.862820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.862820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:55.777488+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":806,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473208+0000","last_change":"2024-04-25T05:50:06.715616+0000","last_active":"2024-04-25T05:51:06.473208+0000","last_peered":"2024-04-25T05:51:06.473208+0000","last_clean":"2024-04-25T05:51:06.473208+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:51:06.473208+0000","last_undegraded":"2024-04-25T05:51:06.473208+0000","last_fullsized":"2024-04-25T05:51:06.473208+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:50:06.715476+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.715476+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:29.983548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"480'5252","reported_seq":5309,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.561557+0000","last_change":"2024-04-25T05:50:12.035436+0000","last_active":"2024-04-25T05:51:08.561557+0000","last_peered":"2024-04-25T05:51:08.561557+0000","last_clean":"2024-04-25T05:51:08.561557+0000","last_became_active":"2024-04-25T05:50:12.035111+0000","last_became_peered":"2024-04-25T05:50:12.035111+0000","last_unstale":"2024-04-25T05:51:08.561557+0000","last_undegraded":"2024-04-25T05:51:08.561557+0000","last_fullsized":"2024-04-25T05:51:08.561557+0000","mapping_epoch":456,"log_start":"463'400","ondisk_log_start":"463'400","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.368286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344195072,"num_objects":5252,"num_object_clones":0,"num_object_copies":15756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5252,"num_write_kb":336128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":684,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469498+0000","last_change":"2024-04-25T05:50:11.713907+0000","last_active":"2024-04-25T05:51:06.469498+0000","last_peered":"2024-04-25T05:51:06.469498+0000","last_clean":"2024-04-25T05:51:06.469498+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:51:06.469498+0000","last_undegraded":"2024-04-25T05:51:06.469498+0000","last_fullsized":"2024-04-25T05:51:06.469498+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:50:11.713794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.713794+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.646485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":911,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468275+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:51:06.468275+0000","last_peered":"2024-04-25T05:51:06.468275+0000","last_clean":"2024-04-25T05:51:06.468275+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:51:06.468275+0000","last_undegraded":"2024-04-25T05:51:06.468275+0000","last_fullsized":"2024-04-25T05:51:06.468275+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716196+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":209,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475063+0000","last_change":"2024-04-25T05:49:29.951087+0000","last_active":"2024-04-25T05:51:06.475063+0000","last_peered":"2024-04-25T05:51:06.475063+0000","last_clean":"2024-04-25T05:51:06.475063+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:51:06.475063+0000","last_undegraded":"2024-04-25T05:51:06.475063+0000","last_fullsized":"2024-04-25T05:51:06.475063+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:29.950964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:49:29.950964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.969131+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":887,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471800+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:51:06.471800+0000","last_peered":"2024-04-25T05:51:06.471800+0000","last_clean":"2024-04-25T05:51:06.471800+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:51:06.471800+0000","last_undegraded":"2024-04-25T05:51:06.471800+0000","last_fullsized":"2024-04-25T05:51:06.471800+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.640676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"480'5254","reported_seq":5310,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.098363+0000","last_change":"2024-04-25T05:50:12.041469+0000","last_active":"2024-04-25T05:51:09.098363+0000","last_peered":"2024-04-25T05:51:09.098363+0000","last_clean":"2024-04-25T05:51:09.098363+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:51:09.098363+0000","last_undegraded":"2024-04-25T05:51:09.098363+0000","last_fullsized":"2024-04-25T05:51:09.098363+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4054,"log_dups_size":0,"ondisk_log_size":4054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:12.357670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344326144,"num_objects":5254,"num_object_clones":0,"num_object_copies":15762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5254,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":928,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.482661+0000","last_change":"2024-04-25T05:50:10.992094+0000","last_active":"2024-04-25T05:51:06.482661+0000","last_peered":"2024-04-25T05:51:06.482661+0000","last_clean":"2024-04-25T05:51:06.482661+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:51:06.482661+0000","last_undegraded":"2024-04-25T05:51:06.482661+0000","last_fullsized":"2024-04-25T05:51:06.482661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:10.991997+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:50:10.991997+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.924649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":875,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468130+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:51:06.468130+0000","last_peered":"2024-04-25T05:51:06.468130+0000","last_clean":"2024-04-25T05:51:06.468130+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:51:06.468130+0000","last_undegraded":"2024-04-25T05:51:06.468130+0000","last_fullsized":"2024-04-25T05:51:06.468130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"433'16","reported_seq":542,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469848+0000","last_change":"2024-04-25T05:49:09.610647+0000","last_active":"2024-04-25T05:51:06.469848+0000","last_peered":"2024-04-25T05:51:06.469848+0000","last_clean":"2024-04-25T05:51:06.469848+0000","last_became_active":"2024-04-25T05:49:09.561406+0000","last_became_peered":"2024-04-25T05:49:09.561406+0000","last_unstale":"2024-04-25T05:51:06.469848+0000","last_undegraded":"2024-04-25T05:51:06.469848+0000","last_fullsized":"2024-04-25T05:51:06.469848+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-25T05:48:21.090099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.090099+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716297+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":209,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475013+0000","last_change":"2024-04-25T05:48:30.199408+0000","last_active":"2024-04-25T05:51:06.475013+0000","last_peered":"2024-04-25T05:51:06.475013+0000","last_clean":"2024-04-25T05:51:06.475013+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:51:06.475013+0000","last_undegraded":"2024-04-25T05:51:06.475013+0000","last_fullsized":"2024-04-25T05:51:06.475013+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:30.199282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.199282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968893+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":770,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475470+0000","last_change":"2024-04-25T05:50:47.941283+0000","last_active":"2024-04-25T05:51:06.475470+0000","last_peered":"2024-04-25T05:51:06.475470+0000","last_clean":"2024-04-25T05:51:06.475470+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:51:06.475470+0000","last_undegraded":"2024-04-25T05:51:06.475470+0000","last_fullsized":"2024-04-25T05:51:06.475470+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:47.940906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.940906+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.568875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"480'5299","reported_seq":5356,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.171351+0000","last_change":"2024-04-25T05:50:12.042142+0000","last_active":"2024-04-25T05:51:08.171351+0000","last_peered":"2024-04-25T05:51:08.171351+0000","last_clean":"2024-04-25T05:51:08.171351+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:51:08.171351+0000","last_undegraded":"2024-04-25T05:51:08.171351+0000","last_fullsized":"2024-04-25T05:51:08.171351+0000","mapping_epoch":456,"log_start":"464'800","ondisk_log_start":"464'800","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.962178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5299,"num_write_kb":339136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468186+0000","last_change":"2024-04-25T05:49:03.437727+0000","last_active":"2024-04-25T05:51:06.468186+0000","last_peered":"2024-04-25T05:51:06.468186+0000","last_clean":"2024-04-25T05:51:06.468186+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:51:06.468186+0000","last_undegraded":"2024-04-25T05:51:06.468186+0000","last_fullsized":"2024-04-25T05:51:06.468186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:03.437604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:03.437604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716529+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1030,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468242+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:51:06.468242+0000","last_peered":"2024-04-25T05:51:06.468242+0000","last_clean":"2024-04-25T05:51:06.468242+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:51:06.468242+0000","last_undegraded":"2024-04-25T05:51:06.468242+0000","last_fullsized":"2024-04-25T05:51:06.468242+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":907,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472109+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:51:06.472109+0000","last_peered":"2024-04-25T05:51:06.472109+0000","last_clean":"2024-04-25T05:51:06.472109+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:51:06.472109+0000","last_undegraded":"2024-04-25T05:51:06.472109+0000","last_fullsized":"2024-04-25T05:51:06.472109+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":446,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468246+0000","last_change":"2024-04-25T05:50:41.801622+0000","last_active":"2024-04-25T05:51:06.468246+0000","last_peered":"2024-04-25T05:51:06.468246+0000","last_clean":"2024-04-25T05:51:06.468246+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:51:06.468246+0000","last_undegraded":"2024-04-25T05:51:06.468246+0000","last_fullsized":"2024-04-25T05:51:06.468246+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.801508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.801508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.561567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1121,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468116+0000","last_change":"2024-04-25T05:49:01.441992+0000","last_active":"2024-04-25T05:51:06.468116+0000","last_peered":"2024-04-25T05:51:06.468116+0000","last_clean":"2024-04-25T05:51:06.468116+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:51:06.468116+0000","last_undegraded":"2024-04-25T05:51:06.468116+0000","last_fullsized":"2024-04-25T05:51:06.468116+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:01.440957+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.440957+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"480'5135","reported_seq":5193,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.116799+0000","last_change":"2024-04-25T05:50:12.042923+0000","last_active":"2024-04-25T05:51:08.116799+0000","last_peered":"2024-04-25T05:51:08.116799+0000","last_clean":"2024-04-25T05:51:08.116799+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:51:08.116799+0000","last_undegraded":"2024-04-25T05:51:08.116799+0000","last_fullsized":"2024-04-25T05:51:08.116799+0000","mapping_epoch":456,"log_start":"464'900","ondisk_log_start":"464'900","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.401319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336527360,"num_objects":5135,"num_object_clones":0,"num_object_copies":15405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5135,"num_write_kb":328640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1093,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468406+0000","last_change":"2024-04-25T05:48:25.100906+0000","last_active":"2024-04-25T05:51:06.468406+0000","last_peered":"2024-04-25T05:51:06.468406+0000","last_clean":"2024-04-25T05:51:06.468406+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:51:06.468406+0000","last_undegraded":"2024-04-25T05:51:06.468406+0000","last_fullsized":"2024-04-25T05:51:06.468406+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:48:25.100752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.100752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":576,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468509+0000","last_change":"2024-04-25T05:50:46.875806+0000","last_active":"2024-04-25T05:51:06.468509+0000","last_peered":"2024-04-25T05:51:06.468509+0000","last_clean":"2024-04-25T05:51:06.468509+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:51:06.468509+0000","last_undegraded":"2024-04-25T05:51:06.468509+0000","last_fullsized":"2024-04-25T05:51:06.468509+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:46.875662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.875662+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:49.181320+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":587,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.480811+0000","last_change":"2024-04-25T05:50:15.054217+0000","last_active":"2024-04-25T05:51:06.480811+0000","last_peered":"2024-04-25T05:51:06.480811+0000","last_clean":"2024-04-25T05:51:06.480811+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:51:06.480811+0000","last_undegraded":"2024-04-25T05:51:06.480811+0000","last_fullsized":"2024-04-25T05:51:06.480811+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.219601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":265,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.481528+0000","last_change":"2024-04-25T05:50:12.172126+0000","last_active":"2024-04-25T05:51:06.481528+0000","last_peered":"2024-04-25T05:51:06.481528+0000","last_clean":"2024-04-25T05:51:06.481528+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:51:06.481528+0000","last_undegraded":"2024-04-25T05:51:06.481528+0000","last_fullsized":"2024-04-25T05:51:06.481528+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:12.172001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.172001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.707108+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1077,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471688+0000","last_change":"2024-04-25T05:48:27.160787+0000","last_active":"2024-04-25T05:51:06.471688+0000","last_peered":"2024-04-25T05:51:06.471688+0000","last_clean":"2024-04-25T05:51:06.471688+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:51:06.471688+0000","last_undegraded":"2024-04-25T05:51:06.471688+0000","last_fullsized":"2024-04-25T05:51:06.471688+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:48:27.160639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.160639+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"480'5132","reported_seq":5190,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:07.689704+0000","last_change":"2024-04-25T05:50:12.035986+0000","last_active":"2024-04-25T05:51:07.689704+0000","last_peered":"2024-04-25T05:51:07.689704+0000","last_clean":"2024-04-25T05:51:07.689704+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:51:07.689704+0000","last_undegraded":"2024-04-25T05:51:07.689704+0000","last_fullsized":"2024-04-25T05:51:07.689704+0000","mapping_epoch":456,"log_start":"462'200","ondisk_log_start":"462'200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.423843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5132,"num_write_kb":328448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":690,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475595+0000","last_change":"2024-04-25T05:50:45.130550+0000","last_active":"2024-04-25T05:51:06.475595+0000","last_peered":"2024-04-25T05:51:06.475595+0000","last_clean":"2024-04-25T05:51:06.475595+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:51:06.475595+0000","last_undegraded":"2024-04-25T05:51:06.475595+0000","last_fullsized":"2024-04-25T05:51:06.475595+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:51.401906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":624,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479303+0000","last_change":"2024-04-25T05:49:02.359488+0000","last_active":"2024-04-25T05:51:06.479303+0000","last_peered":"2024-04-25T05:51:06.479303+0000","last_clean":"2024-04-25T05:51:06.479303+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:51:06.479303+0000","last_undegraded":"2024-04-25T05:51:06.479303+0000","last_fullsized":"2024-04-25T05:51:06.479303+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:49:02.359173+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.359173+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":217,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474991+0000","last_change":"2024-04-25T05:50:08.761599+0000","last_active":"2024-04-25T05:51:06.474991+0000","last_peered":"2024-04-25T05:51:06.474991+0000","last_clean":"2024-04-25T05:51:06.474991+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:51:06.474991+0000","last_undegraded":"2024-04-25T05:51:06.474991+0000","last_fullsized":"2024-04-25T05:51:06.474991+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.520460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":522,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.467694+0000","last_change":"2024-04-25T05:50:09.086597+0000","last_active":"2024-04-25T05:51:06.467694+0000","last_peered":"2024-04-25T05:51:06.467694+0000","last_clean":"2024-04-25T05:51:06.467694+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:51:06.467694+0000","last_undegraded":"2024-04-25T05:51:06.467694+0000","last_fullsized":"2024-04-25T05:51:06.467694+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.086479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.086479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.899606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1069,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473815+0000","last_change":"2024-04-25T05:50:07.959527+0000","last_active":"2024-04-25T05:51:06.473815+0000","last_peered":"2024-04-25T05:51:06.473815+0000","last_clean":"2024-04-25T05:51:06.473815+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:51:06.473815+0000","last_undegraded":"2024-04-25T05:51:06.473815+0000","last_fullsized":"2024-04-25T05:51:06.473815+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.959446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.959446+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:24.723125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"480'5444","reported_seq":5501,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.098193+0000","last_change":"2024-04-25T05:50:12.038131+0000","last_active":"2024-04-25T05:51:09.098193+0000","last_peered":"2024-04-25T05:51:09.098193+0000","last_clean":"2024-04-25T05:51:09.098193+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:51:09.098193+0000","last_undegraded":"2024-04-25T05:51:09.098193+0000","last_fullsized":"2024-04-25T05:51:09.098193+0000","mapping_epoch":456,"log_start":"465'1400","ondisk_log_start":"465'1400","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.865063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356777984,"num_objects":5444,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5444,"num_write_kb":348416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":644,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475151+0000","last_change":"2024-04-25T05:50:42.148569+0000","last_active":"2024-04-25T05:51:06.475151+0000","last_peered":"2024-04-25T05:51:06.475151+0000","last_clean":"2024-04-25T05:51:06.475151+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:51:06.475151+0000","last_undegraded":"2024-04-25T05:51:06.475151+0000","last_fullsized":"2024-04-25T05:51:06.475151+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:42.148429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:50:42.148429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:11.882512+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":255,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474017+0000","last_change":"2024-04-25T05:48:59.056819+0000","last_active":"2024-04-25T05:51:06.474017+0000","last_peered":"2024-04-25T05:51:06.474017+0000","last_clean":"2024-04-25T05:51:06.474017+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:51:06.474017+0000","last_undegraded":"2024-04-25T05:51:06.474017+0000","last_fullsized":"2024-04-25T05:51:06.474017+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:48:59.056469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.056469+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.969105+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":921,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.481152+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:51:06.481152+0000","last_peered":"2024-04-25T05:51:06.481152+0000","last_clean":"2024-04-25T05:51:06.481152+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:51:06.481152+0000","last_undegraded":"2024-04-25T05:51:06.481152+0000","last_fullsized":"2024-04-25T05:51:06.481152+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.818280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":895,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.480436+0000","last_change":"2024-04-25T05:49:32.024307+0000","last_active":"2024-04-25T05:51:06.480436+0000","last_peered":"2024-04-25T05:51:06.480436+0000","last_clean":"2024-04-25T05:51:06.480436+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:51:06.480436+0000","last_undegraded":"2024-04-25T05:51:06.480436+0000","last_fullsized":"2024-04-25T05:51:06.480436+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.024156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.024156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1087,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474686+0000","last_change":"2024-04-25T05:50:01.929134+0000","last_active":"2024-04-25T05:51:06.474686+0000","last_peered":"2024-04-25T05:51:06.474686+0000","last_clean":"2024-04-25T05:51:06.474686+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:51:06.474686+0000","last_undegraded":"2024-04-25T05:51:06.474686+0000","last_fullsized":"2024-04-25T05:51:06.474686+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:01.929038+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:50:01.929038+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:20.992655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"480'5364","reported_seq":5421,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.113269+0000","last_change":"2024-04-25T05:50:12.035979+0000","last_active":"2024-04-25T05:51:09.113269+0000","last_peered":"2024-04-25T05:51:09.113269+0000","last_clean":"2024-04-25T05:51:09.113269+0000","last_became_active":"2024-04-25T05:50:12.035482+0000","last_became_peered":"2024-04-25T05:50:12.035482+0000","last_unstale":"2024-04-25T05:51:09.113269+0000","last_undegraded":"2024-04-25T05:51:09.113269+0000","last_fullsized":"2024-04-25T05:51:09.113269+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.703340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351535104,"num_objects":5364,"num_object_clones":0,"num_object_copies":16092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5364,"num_write_kb":343296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":209,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474929+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:51:06.474929+0000","last_peered":"2024-04-25T05:51:06.474929+0000","last_clean":"2024-04-25T05:51:06.474929+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:51:06.474929+0000","last_undegraded":"2024-04-25T05:51:06.474929+0000","last_fullsized":"2024-04-25T05:51:06.474929+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968838+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":231,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474422+0000","last_change":"2024-04-25T05:50:12.786445+0000","last_active":"2024-04-25T05:51:06.474422+0000","last_peered":"2024-04-25T05:51:06.474422+0000","last_clean":"2024-04-25T05:51:06.474422+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:51:06.474422+0000","last_undegraded":"2024-04-25T05:51:06.474422+0000","last_fullsized":"2024-04-25T05:51:06.474422+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:50:12.786333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.786333+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.425682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1945,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.959270+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:51:06.959270+0000","last_peered":"2024-04-25T05:51:06.959270+0000","last_clean":"2024-04-25T05:51:06.959270+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:51:06.959270+0000","last_undegraded":"2024-04-25T05:51:06.959270+0000","last_fullsized":"2024-04-25T05:51:06.959270+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1011,"num_read_kb":1006,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":517,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468359+0000","last_change":"2024-04-25T05:49:05.488606+0000","last_active":"2024-04-25T05:51:06.468359+0000","last_peered":"2024-04-25T05:51:06.468359+0000","last_clean":"2024-04-25T05:51:06.468359+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:51:06.468359+0000","last_undegraded":"2024-04-25T05:51:06.468359+0000","last_fullsized":"2024-04-25T05:51:06.468359+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.488514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.488514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.480818+0000","last_change":"2024-04-25T05:48:35.483025+0000","last_active":"2024-04-25T05:51:06.480818+0000","last_peered":"2024-04-25T05:51:06.480818+0000","last_clean":"2024-04-25T05:51:06.480818+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:51:06.480818+0000","last_undegraded":"2024-04-25T05:51:06.480818+0000","last_fullsized":"2024-04-25T05:51:06.480818+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.482886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.482886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1065,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.481419+0000","last_change":"2024-04-25T05:49:08.317583+0000","last_active":"2024-04-25T05:51:06.481419+0000","last_peered":"2024-04-25T05:51:06.481419+0000","last_clean":"2024-04-25T05:51:06.481419+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:51:06.481419+0000","last_undegraded":"2024-04-25T05:51:06.481419+0000","last_fullsized":"2024-04-25T05:51:06.481419+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_clean_scrub_stamp":"2024-04-25T05:49:08.317433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"480'5242","reported_seq":5300,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.105720+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:51:09.105720+0000","last_peered":"2024-04-25T05:51:09.105720+0000","last_clean":"2024-04-25T05:51:09.105720+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:51:09.105720+0000","last_undegraded":"2024-04-25T05:51:09.105720+0000","last_fullsized":"2024-04-25T05:51:09.105720+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.159052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343539712,"num_objects":5242,"num_object_clones":0,"num_object_copies":15726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5242,"num_write_kb":335488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":905,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472439+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:51:06.472439+0000","last_peered":"2024-04-25T05:51:06.472439+0000","last_clean":"2024-04-25T05:51:06.472439+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:51:06.472439+0000","last_undegraded":"2024-04-25T05:51:06.472439+0000","last_fullsized":"2024-04-25T05:51:06.472439+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":789,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474223+0000","last_change":"2024-04-25T05:50:13.028112+0000","last_active":"2024-04-25T05:51:06.474223+0000","last_peered":"2024-04-25T05:51:06.474223+0000","last_clean":"2024-04-25T05:51:06.474223+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:51:06.474223+0000","last_undegraded":"2024-04-25T05:51:06.474223+0000","last_fullsized":"2024-04-25T05:51:06.474223+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.027901+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.798452+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.477069+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:51:06.477069+0000","last_peered":"2024-04-25T05:51:06.477069+0000","last_clean":"2024-04-25T05:51:06.477069+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:51:06.477069+0000","last_undegraded":"2024-04-25T05:51:06.477069+0000","last_fullsized":"2024-04-25T05:51:06.477069+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1200,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471308+0000","last_change":"2024-04-25T05:50:07.888799+0000","last_active":"2024-04-25T05:51:06.471308+0000","last_peered":"2024-04-25T05:51:06.471308+0000","last_clean":"2024-04-25T05:51:06.471308+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:51:06.471308+0000","last_undegraded":"2024-04-25T05:51:06.471308+0000","last_fullsized":"2024-04-25T05:51:06.471308+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.888714+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.888714+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.057112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.485918+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:51:06.485918+0000","last_peered":"2024-04-25T05:51:06.485918+0000","last_clean":"2024-04-25T05:51:06.485918+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:51:06.485918+0000","last_undegraded":"2024-04-25T05:51:06.485918+0000","last_fullsized":"2024-04-25T05:51:06.485918+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.250098+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"480'5295","reported_seq":5352,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.109718+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:51:09.109718+0000","last_peered":"2024-04-25T05:51:09.109718+0000","last_clean":"2024-04-25T05:51:09.109718+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:51:09.109718+0000","last_undegraded":"2024-04-25T05:51:09.109718+0000","last_fullsized":"2024-04-25T05:51:09.109718+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.808327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5295,"num_write_kb":338880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":248,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469621+0000","last_change":"2024-04-25T05:50:45.550588+0000","last_active":"2024-04-25T05:51:06.469621+0000","last_peered":"2024-04-25T05:51:06.469621+0000","last_clean":"2024-04-25T05:51:06.469621+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:51:06.469621+0000","last_undegraded":"2024-04-25T05:51:06.469621+0000","last_fullsized":"2024-04-25T05:51:06.469621+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.550474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.550474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.043449+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1029,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.482530+0000","last_change":"2024-04-25T05:50:40.134940+0000","last_active":"2024-04-25T05:51:06.482530+0000","last_peered":"2024-04-25T05:51:06.482530+0000","last_clean":"2024-04-25T05:51:06.482530+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:51:06.482530+0000","last_undegraded":"2024-04-25T05:51:06.482530+0000","last_fullsized":"2024-04-25T05:51:06.482530+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:50:40.134604+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.134604+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:41.529906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":242,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468867+0000","last_change":"2024-04-25T05:50:14.628452+0000","last_active":"2024-04-25T05:51:06.468867+0000","last_peered":"2024-04-25T05:51:06.468867+0000","last_clean":"2024-04-25T05:51:06.468867+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:51:06.468867+0000","last_undegraded":"2024-04-25T05:51:06.468867+0000","last_fullsized":"2024-04-25T05:51:06.468867+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.601623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":927,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.478689+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:51:06.478689+0000","last_peered":"2024-04-25T05:51:06.478689+0000","last_clean":"2024-04-25T05:51:06.478689+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:51:06.478689+0000","last_undegraded":"2024-04-25T05:51:06.478689+0000","last_fullsized":"2024-04-25T05:51:06.478689+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.323525+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":883,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472490+0000","last_change":"2024-04-25T05:47:23.888942+0000","last_active":"2024-04-25T05:51:06.472490+0000","last_peered":"2024-04-25T05:51:06.472490+0000","last_clean":"2024-04-25T05:51:06.472490+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:51:06.472490+0000","last_undegraded":"2024-04-25T05:51:06.472490+0000","last_fullsized":"2024-04-25T05:51:06.472490+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:23.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907343+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1208,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471628+0000","last_change":"2024-04-25T05:49:32.818373+0000","last_active":"2024-04-25T05:51:06.471628+0000","last_peered":"2024-04-25T05:51:06.471628+0000","last_clean":"2024-04-25T05:51:06.471628+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:51:06.471628+0000","last_undegraded":"2024-04-25T05:51:06.471628+0000","last_fullsized":"2024-04-25T05:51:06.471628+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:32.818274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.818274+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"480'5359","reported_seq":5417,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:09.072763+0000","last_change":"2024-04-25T05:50:12.040668+0000","last_active":"2024-04-25T05:51:09.072763+0000","last_peered":"2024-04-25T05:51:09.072763+0000","last_clean":"2024-04-25T05:51:09.072763+0000","last_became_active":"2024-04-25T05:50:12.040023+0000","last_became_peered":"2024-04-25T05:50:12.040023+0000","last_unstale":"2024-04-25T05:51:09.072763+0000","last_undegraded":"2024-04-25T05:51:09.072763+0000","last_fullsized":"2024-04-25T05:51:09.072763+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:36.039935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351207424,"num_objects":5359,"num_object_clones":0,"num_object_copies":16077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":342976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":508,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479403+0000","last_change":"2024-04-25T05:48:29.428977+0000","last_active":"2024-04-25T05:51:06.479403+0000","last_peered":"2024-04-25T05:51:06.479403+0000","last_clean":"2024-04-25T05:51:06.479403+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:51:06.479403+0000","last_undegraded":"2024-04-25T05:51:06.479403+0000","last_fullsized":"2024-04-25T05:51:06.479403+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:29.428764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:29.428764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1235,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.481298+0000","last_change":"2024-04-25T05:50:07.985223+0000","last_active":"2024-04-25T05:51:06.481298+0000","last_peered":"2024-04-25T05:51:06.481298+0000","last_clean":"2024-04-25T05:51:06.481298+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:51:06.481298+0000","last_undegraded":"2024-04-25T05:51:06.481298+0000","last_fullsized":"2024-04-25T05:51:06.481298+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:50:07.985110+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.985110+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.172799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":451,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468778+0000","last_change":"2024-04-25T05:49:09.560837+0000","last_active":"2024-04-25T05:51:06.468778+0000","last_peered":"2024-04-25T05:51:06.468778+0000","last_clean":"2024-04-25T05:51:06.468778+0000","last_became_active":"2024-04-25T05:49:09.559684+0000","last_became_peered":"2024-04-25T05:49:09.559684+0000","last_unstale":"2024-04-25T05:51:06.468778+0000","last_undegraded":"2024-04-25T05:51:06.468778+0000","last_fullsized":"2024-04-25T05:51:06.468778+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:37.254738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:48:37.254738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716325+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":221,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475362+0000","last_change":"2024-04-25T05:50:04.735739+0000","last_active":"2024-04-25T05:51:06.475362+0000","last_peered":"2024-04-25T05:51:06.475362+0000","last_clean":"2024-04-25T05:51:06.475362+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:51:06.475362+0000","last_undegraded":"2024-04-25T05:51:06.475362+0000","last_fullsized":"2024-04-25T05:51:06.475362+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:50:04.735651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.735651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:21.992705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":863,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473509+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:51:06.473509+0000","last_peered":"2024-04-25T05:51:06.473509+0000","last_clean":"2024-04-25T05:51:06.473509+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:51:06.473509+0000","last_undegraded":"2024-04-25T05:51:06.473509+0000","last_fullsized":"2024-04-25T05:51:06.473509+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":909,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473174+0000","last_change":"2024-04-25T05:49:00.009223+0000","last_active":"2024-04-25T05:51:06.473174+0000","last_peered":"2024-04-25T05:51:06.473174+0000","last_clean":"2024-04-25T05:51:06.473174+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:51:06.473174+0000","last_undegraded":"2024-04-25T05:51:06.473174+0000","last_fullsized":"2024-04-25T05:51:06.473174+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:00.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:00.008984+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.969022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"480'5181","reported_seq":5237,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.220461+0000","last_change":"2024-04-25T05:50:12.043881+0000","last_active":"2024-04-25T05:51:08.220461+0000","last_peered":"2024-04-25T05:51:08.220461+0000","last_clean":"2024-04-25T05:51:08.220461+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:51:08.220461+0000","last_undegraded":"2024-04-25T05:51:08.220461+0000","last_fullsized":"2024-04-25T05:51:08.220461+0000","mapping_epoch":456,"log_start":"464'700","ondisk_log_start":"464'700","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:18.117782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542016,"num_objects":5181,"num_object_clones":0,"num_object_copies":15543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":331584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":600,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.486617+0000","last_change":"2024-04-25T05:50:14.065922+0000","last_active":"2024-04-25T05:51:06.486617+0000","last_peered":"2024-04-25T05:51:06.486617+0000","last_clean":"2024-04-25T05:51:06.486617+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:51:06.486617+0000","last_undegraded":"2024-04-25T05:51:06.486617+0000","last_fullsized":"2024-04-25T05:51:06.486617+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:22.097234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":209,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474132+0000","last_change":"2024-04-25T05:48:20.054569+0000","last_active":"2024-04-25T05:51:06.474132+0000","last_peered":"2024-04-25T05:51:06.474132+0000","last_clean":"2024-04-25T05:51:06.474132+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:51:06.474132+0000","last_undegraded":"2024-04-25T05:51:06.474132+0000","last_fullsized":"2024-04-25T05:51:06.474132+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:20.054446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:48:20.054446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968866+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":234,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471118+0000","last_change":"2024-04-25T05:50:11.521033+0000","last_active":"2024-04-25T05:51:06.471118+0000","last_peered":"2024-04-25T05:51:06.471118+0000","last_clean":"2024-04-25T05:51:06.471118+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:51:06.471118+0000","last_undegraded":"2024-04-25T05:51:06.471118+0000","last_fullsized":"2024-04-25T05:51:06.471118+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.520907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.520907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.942600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":500,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471860+0000","last_change":"2024-04-25T05:47:19.913147+0000","last_active":"2024-04-25T05:51:06.471860+0000","last_peered":"2024-04-25T05:51:06.471860+0000","last_clean":"2024-04-25T05:51:06.471860+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:51:06.471860+0000","last_undegraded":"2024-04-25T05:51:06.471860+0000","last_fullsized":"2024-04-25T05:51:06.471860+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.913041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.913041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907309+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"480'5326","reported_seq":5383,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.186491+0000","last_change":"2024-04-25T05:50:12.042685+0000","last_active":"2024-04-25T05:51:08.186491+0000","last_peered":"2024-04-25T05:51:08.186491+0000","last_clean":"2024-04-25T05:51:08.186491+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:51:08.186491+0000","last_undegraded":"2024-04-25T05:51:08.186491+0000","last_fullsized":"2024-04-25T05:51:08.186491+0000","mapping_epoch":456,"log_start":"464'900","ondisk_log_start":"464'900","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:35.841340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349044736,"num_objects":5326,"num_object_clones":0,"num_object_copies":15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5326,"num_write_kb":340864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":670,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474910+0000","last_change":"2024-04-25T05:50:15.942029+0000","last_active":"2024-04-25T05:51:06.474910+0000","last_peered":"2024-04-25T05:51:06.474910+0000","last_clean":"2024-04-25T05:51:06.474910+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:51:06.474910+0000","last_undegraded":"2024-04-25T05:51:06.474910+0000","last_fullsized":"2024-04-25T05:51:06.474910+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.941896+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.479186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":937,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472208+0000","last_change":"2024-04-25T05:50:47.124683+0000","last_active":"2024-04-25T05:51:06.472208+0000","last_peered":"2024-04-25T05:51:06.472208+0000","last_clean":"2024-04-25T05:51:06.472208+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:51:06.472208+0000","last_undegraded":"2024-04-25T05:51:06.472208+0000","last_fullsized":"2024-04-25T05:51:06.472208+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.124554+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.362413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":240,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469651+0000","last_change":"2024-04-25T05:50:05.416344+0000","last_active":"2024-04-25T05:51:06.469651+0000","last_peered":"2024-04-25T05:51:06.469651+0000","last_clean":"2024-04-25T05:51:06.469651+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:51:06.469651+0000","last_undegraded":"2024-04-25T05:51:06.469651+0000","last_fullsized":"2024-04-25T05:51:06.469651+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.416227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.416227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.653594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":882,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471198+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:51:06.471198+0000","last_peered":"2024-04-25T05:51:06.471198+0000","last_clean":"2024-04-25T05:51:06.471198+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:51:06.471198+0000","last_undegraded":"2024-04-25T05:51:06.471198+0000","last_fullsized":"2024-04-25T05:51:06.471198+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"480'5148","reported_seq":5203,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:07.713716+0000","last_change":"2024-04-25T05:50:12.035713+0000","last_active":"2024-04-25T05:51:07.713716+0000","last_peered":"2024-04-25T05:51:07.713716+0000","last_clean":"2024-04-25T05:51:07.713716+0000","last_became_active":"2024-04-25T05:50:12.035376+0000","last_became_peered":"2024-04-25T05:50:12.035376+0000","last_unstale":"2024-04-25T05:51:07.713716+0000","last_undegraded":"2024-04-25T05:51:07.713716+0000","last_fullsized":"2024-04-25T05:51:07.713716+0000","mapping_epoch":456,"log_start":"462'300","ondisk_log_start":"462'300","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.653834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":248,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468808+0000","last_change":"2024-04-25T05:50:39.566845+0000","last_active":"2024-04-25T05:51:06.468808+0000","last_peered":"2024-04-25T05:51:06.468808+0000","last_clean":"2024-04-25T05:51:06.468808+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:51:06.468808+0000","last_undegraded":"2024-04-25T05:51:06.468808+0000","last_fullsized":"2024-04-25T05:51:06.468808+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.566717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.566717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.356177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":298,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468688+0000","last_change":"2024-04-25T05:49:09.598004+0000","last_active":"2024-04-25T05:51:06.468688+0000","last_peered":"2024-04-25T05:51:06.468688+0000","last_clean":"2024-04-25T05:51:06.468688+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:51:06.468688+0000","last_undegraded":"2024-04-25T05:51:06.468688+0000","last_fullsized":"2024-04-25T05:51:06.468688+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:48:24.506969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.506969+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716383+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":890,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472403+0000","last_change":"2024-04-25T05:50:41.101003+0000","last_active":"2024-04-25T05:51:06.472403+0000","last_peered":"2024-04-25T05:51:06.472403+0000","last_clean":"2024-04-25T05:51:06.472403+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:51:06.472403+0000","last_undegraded":"2024-04-25T05:51:06.472403+0000","last_fullsized":"2024-04-25T05:51:06.472403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.100880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.100880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.503491+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":875,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472355+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:51:06.472355+0000","last_peered":"2024-04-25T05:51:06.472355+0000","last_clean":"2024-04-25T05:51:06.472355+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:51:06.472355+0000","last_undegraded":"2024-04-25T05:51:06.472355+0000","last_fullsized":"2024-04-25T05:51:06.472355+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.671987+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"480'5174","reported_seq":5231,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:08.107895+0000","last_change":"2024-04-25T05:50:12.042051+0000","last_active":"2024-04-25T05:51:08.107895+0000","last_peered":"2024-04-25T05:51:08.107895+0000","last_clean":"2024-04-25T05:51:08.107895+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:51:08.107895+0000","last_undegraded":"2024-04-25T05:51:08.107895+0000","last_fullsized":"2024-04-25T05:51:08.107895+0000","mapping_epoch":456,"log_start":"463'400","ondisk_log_start":"463'400","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.775770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339083264,"num_objects":5174,"num_object_clones":0,"num_object_copies":15522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":331136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1123,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471341+0000","last_change":"2024-04-25T05:50:39.051287+0000","last_active":"2024-04-25T05:51:06.471341+0000","last_peered":"2024-04-25T05:51:06.471341+0000","last_clean":"2024-04-25T05:51:06.471341+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:51:06.471341+0000","last_undegraded":"2024-04-25T05:51:06.471341+0000","last_fullsized":"2024-04-25T05:51:06.471341+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:50:39.050944+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.050944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:07.882347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":237,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473309+0000","last_change":"2024-04-25T05:50:11.779506+0000","last_active":"2024-04-25T05:51:06.473309+0000","last_peered":"2024-04-25T05:51:06.473309+0000","last_clean":"2024-04-25T05:51:06.473309+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:51:06.473309+0000","last_undegraded":"2024-04-25T05:51:06.473309+0000","last_fullsized":"2024-04-25T05:51:06.473309+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:50:11.779355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.779355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.341739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":930,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475944+0000","last_change":"2024-04-25T05:48:59.765915+0000","last_active":"2024-04-25T05:51:06.475944+0000","last_peered":"2024-04-25T05:51:06.475944+0000","last_clean":"2024-04-25T05:51:06.475944+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:51:06.475944+0000","last_undegraded":"2024-04-25T05:51:06.475944+0000","last_fullsized":"2024-04-25T05:51:06.475944+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:48:59.765781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.765781+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907589+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":209,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.475360+0000","last_change":"2024-04-25T05:49:01.017521+0000","last_active":"2024-04-25T05:51:06.475360+0000","last_peered":"2024-04-25T05:51:06.475360+0000","last_clean":"2024-04-25T05:51:06.475360+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:51:06.475360+0000","last_undegraded":"2024-04-25T05:51:06.475360+0000","last_fullsized":"2024-04-25T05:51:06.475360+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:01.017406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.017406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.969077+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":901,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472496+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:51:06.472496+0000","last_peered":"2024-04-25T05:51:06.472496+0000","last_clean":"2024-04-25T05:51:06.472496+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:51:06.472496+0000","last_undegraded":"2024-04-25T05:51:06.472496+0000","last_fullsized":"2024-04-25T05:51:06.472496+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":643,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473020+0000","last_change":"2024-04-25T05:50:43.106046+0000","last_active":"2024-04-25T05:51:06.473020+0000","last_peered":"2024-04-25T05:51:06.473020+0000","last_clean":"2024-04-25T05:51:06.473020+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:51:06.473020+0000","last_undegraded":"2024-04-25T05:51:06.473020+0000","last_fullsized":"2024-04-25T05:51:06.473020+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.105913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.105913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.313735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474323+0000","last_change":"2024-04-25T05:50:43.938330+0000","last_active":"2024-04-25T05:51:06.474323+0000","last_peered":"2024-04-25T05:51:06.474323+0000","last_clean":"2024-04-25T05:51:06.474323+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:51:06.474323+0000","last_undegraded":"2024-04-25T05:51:06.474323+0000","last_fullsized":"2024-04-25T05:51:06.474323+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.938246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.938246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:52.848134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":890,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471784+0000","last_change":"2024-04-25T05:48:57.925411+0000","last_active":"2024-04-25T05:51:06.471784+0000","last_peered":"2024-04-25T05:51:06.471784+0000","last_clean":"2024-04-25T05:51:06.471784+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:51:06.471784+0000","last_undegraded":"2024-04-25T05:51:06.471784+0000","last_fullsized":"2024-04-25T05:51:06.471784+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:57.925301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.925301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907619+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":589,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472877+0000","last_change":"2024-04-25T05:49:32.097695+0000","last_active":"2024-04-25T05:51:06.472877+0000","last_peered":"2024-04-25T05:51:06.472877+0000","last_clean":"2024-04-25T05:51:06.472877+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:51:06.472877+0000","last_undegraded":"2024-04-25T05:51:06.472877+0000","last_fullsized":"2024-04-25T05:51:06.472877+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:49:32.097448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.097448+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":908,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471575+0000","last_change":"2024-04-25T05:50:53.041228+0000","last_active":"2024-04-25T05:51:06.471575+0000","last_peered":"2024-04-25T05:51:06.471575+0000","last_clean":"2024-04-25T05:51:06.471575+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:51:06.471575+0000","last_undegraded":"2024-04-25T05:51:06.471575+0000","last_fullsized":"2024-04-25T05:51:06.471575+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:53.041118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:50:53.041118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.078933+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":859,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472283+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:51:06.472283+0000","last_peered":"2024-04-25T05:51:06.472283+0000","last_clean":"2024-04-25T05:51:06.472283+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:51:06.472283+0000","last_undegraded":"2024-04-25T05:51:06.472283+0000","last_fullsized":"2024-04-25T05:51:06.472283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":234,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.478153+0000","last_change":"2024-04-25T05:50:13.581116+0000","last_active":"2024-04-25T05:51:06.478153+0000","last_peered":"2024-04-25T05:51:06.478153+0000","last_clean":"2024-04-25T05:51:06.478153+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:51:06.478153+0000","last_undegraded":"2024-04-25T05:51:06.478153+0000","last_fullsized":"2024-04-25T05:51:06.478153+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.581028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.581028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.001393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":220,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474590+0000","last_change":"2024-04-25T05:48:26.172633+0000","last_active":"2024-04-25T05:51:06.474590+0000","last_peered":"2024-04-25T05:51:06.474590+0000","last_clean":"2024-04-25T05:51:06.474590+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:51:06.474590+0000","last_undegraded":"2024-04-25T05:51:06.474590+0000","last_fullsized":"2024-04-25T05:51:06.474590+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:48:26.172484+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:48:26.172484+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968948+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":493,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471407+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:51:06.471407+0000","last_peered":"2024-04-25T05:51:06.471407+0000","last_clean":"2024-04-25T05:51:06.471407+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:51:06.471407+0000","last_undegraded":"2024-04-25T05:51:06.471407+0000","last_fullsized":"2024-04-25T05:51:06.471407+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907555+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1102,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472520+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:51:06.472520+0000","last_peered":"2024-04-25T05:51:06.472520+0000","last_clean":"2024-04-25T05:51:06.472520+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:51:06.472520+0000","last_undegraded":"2024-04-25T05:51:06.472520+0000","last_fullsized":"2024-04-25T05:51:06.472520+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.466618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":429,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469458+0000","last_change":"2024-04-25T05:50:52.078015+0000","last_active":"2024-04-25T05:51:06.469458+0000","last_peered":"2024-04-25T05:51:06.469458+0000","last_clean":"2024-04-25T05:51:06.469458+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:51:06.469458+0000","last_undegraded":"2024-04-25T05:51:06.469458+0000","last_fullsized":"2024-04-25T05:51:06.469458+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.077914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.221398+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":925,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472037+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:51:06.472037+0000","last_peered":"2024-04-25T05:51:06.472037+0000","last_clean":"2024-04-25T05:51:06.472037+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:51:06.472037+0000","last_undegraded":"2024-04-25T05:51:06.472037+0000","last_fullsized":"2024-04-25T05:51:06.472037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":894,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471736+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:51:06.471736+0000","last_peered":"2024-04-25T05:51:06.471736+0000","last_clean":"2024-04-25T05:51:06.471736+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:51:06.471736+0000","last_undegraded":"2024-04-25T05:51:06.471736+0000","last_fullsized":"2024-04-25T05:51:06.471736+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907497+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":266,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474772+0000","last_change":"2024-04-25T05:49:01.967445+0000","last_active":"2024-04-25T05:51:06.474772+0000","last_peered":"2024-04-25T05:51:06.474772+0000","last_clean":"2024-04-25T05:51:06.474772+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:06.474772+0000","last_undegraded":"2024-04-25T05:51:06.474772+0000","last_fullsized":"2024-04-25T05:51:06.474772+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'13","last_scrub_stamp":"2024-04-25T05:49:01.967348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.967348+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.969051+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":493,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472218+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:51:06.472218+0000","last_peered":"2024-04-25T05:51:06.472218+0000","last_clean":"2024-04-25T05:51:06.472218+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:51:06.472218+0000","last_undegraded":"2024-04-25T05:51:06.472218+0000","last_fullsized":"2024-04-25T05:51:06.472218+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907404+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1021,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469009+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:51:06.469009+0000","last_peered":"2024-04-25T05:51:06.469009+0000","last_clean":"2024-04-25T05:51:06.469009+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:51:06.469009+0000","last_undegraded":"2024-04-25T05:51:06.469009+0000","last_fullsized":"2024-04-25T05:51:06.469009+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716161+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":475,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468149+0000","last_change":"2024-04-25T05:49:35.656164+0000","last_active":"2024-04-25T05:51:06.468149+0000","last_peered":"2024-04-25T05:51:06.468149+0000","last_clean":"2024-04-25T05:51:06.468149+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:51:06.468149+0000","last_undegraded":"2024-04-25T05:51:06.468149+0000","last_fullsized":"2024-04-25T05:51:06.468149+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:49:35.656070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.656070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":889,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.478899+0000","last_change":"2024-04-25T05:50:13.065576+0000","last_active":"2024-04-25T05:51:06.478899+0000","last_peered":"2024-04-25T05:51:06.478899+0000","last_clean":"2024-04-25T05:51:06.478899+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:51:06.478899+0000","last_undegraded":"2024-04-25T05:51:06.478899+0000","last_fullsized":"2024-04-25T05:51:06.478899+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.065464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.065464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.492742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":907,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468375+0000","last_change":"2024-04-25T05:50:06.095785+0000","last_active":"2024-04-25T05:51:06.468375+0000","last_peered":"2024-04-25T05:51:06.468375+0000","last_clean":"2024-04-25T05:51:06.468375+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:51:06.468375+0000","last_undegraded":"2024-04-25T05:51:06.468375+0000","last_fullsized":"2024-04-25T05:51:06.468375+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.604835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":436,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479774+0000","last_change":"2024-04-25T05:48:57.339166+0000","last_active":"2024-04-25T05:51:06.479774+0000","last_peered":"2024-04-25T05:51:06.479774+0000","last_clean":"2024-04-25T05:51:06.479774+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:51:06.479774+0000","last_undegraded":"2024-04-25T05:51:06.479774+0000","last_fullsized":"2024-04-25T05:51:06.479774+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:48:57.338784+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.338784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":890,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472002+0000","last_change":"2024-04-25T05:50:45.109485+0000","last_active":"2024-04-25T05:51:06.472002+0000","last_peered":"2024-04-25T05:51:06.472002+0000","last_clean":"2024-04-25T05:51:06.472002+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:51:06.472002+0000","last_undegraded":"2024-04-25T05:51:06.472002+0000","last_fullsized":"2024-04-25T05:51:06.472002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.109246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.109246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.734569+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":891,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471811+0000","last_change":"2024-04-25T05:50:49.427823+0000","last_active":"2024-04-25T05:51:06.471811+0000","last_peered":"2024-04-25T05:51:06.471811+0000","last_clean":"2024-04-25T05:51:06.471811+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:51:06.471811+0000","last_undegraded":"2024-04-25T05:51:06.471811+0000","last_fullsized":"2024-04-25T05:51:06.471811+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.080828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473103+0000","last_change":"2024-04-25T05:50:02.734274+0000","last_active":"2024-04-25T05:51:06.473103+0000","last_peered":"2024-04-25T05:51:06.473103+0000","last_clean":"2024-04-25T05:51:06.473103+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:51:06.473103+0000","last_undegraded":"2024-04-25T05:51:06.473103+0000","last_fullsized":"2024-04-25T05:51:06.473103+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:02.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:02.734157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:12.617322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":968,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472308+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:51:06.472308+0000","last_peered":"2024-04-25T05:51:06.472308+0000","last_clean":"2024-04-25T05:51:06.472308+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:51:06.472308+0000","last_undegraded":"2024-04-25T05:51:06.472308+0000","last_fullsized":"2024-04-25T05:51:06.472308+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:20.599614+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474169+0000","last_change":"2024-04-25T05:50:05.705768+0000","last_active":"2024-04-25T05:51:06.474169+0000","last_peered":"2024-04-25T05:51:06.474169+0000","last_clean":"2024-04-25T05:51:06.474169+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:51:06.474169+0000","last_undegraded":"2024-04-25T05:51:06.474169+0000","last_fullsized":"2024-04-25T05:51:06.474169+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.705647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.705647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.259931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":892,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471610+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:51:06.471610+0000","last_peered":"2024-04-25T05:51:06.471610+0000","last_clean":"2024-04-25T05:51:06.471610+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:51:06.471610+0000","last_undegraded":"2024-04-25T05:51:06.471610+0000","last_fullsized":"2024-04-25T05:51:06.471610+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":899,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472379+0000","last_change":"2024-04-25T05:49:41.202012+0000","last_active":"2024-04-25T05:51:06.472379+0000","last_peered":"2024-04-25T05:51:06.472379+0000","last_clean":"2024-04-25T05:51:06.472379+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:51:06.472379+0000","last_undegraded":"2024-04-25T05:51:06.472379+0000","last_fullsized":"2024-04-25T05:51:06.472379+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":907,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.467976+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:51:06.467976+0000","last_peered":"2024-04-25T05:51:06.467976+0000","last_clean":"2024-04-25T05:51:06.467976+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:51:06.467976+0000","last_undegraded":"2024-04-25T05:51:06.467976+0000","last_fullsized":"2024-04-25T05:51:06.467976+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474808+0000","last_change":"2024-04-25T05:50:07.763957+0000","last_active":"2024-04-25T05:51:06.474808+0000","last_peered":"2024-04-25T05:51:06.474808+0000","last_clean":"2024-04-25T05:51:06.474808+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:51:06.474808+0000","last_undegraded":"2024-04-25T05:51:06.474808+0000","last_fullsized":"2024-04-25T05:51:06.474808+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:07.763810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.763810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.405363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":605,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473125+0000","last_change":"2024-04-25T05:50:14.959177+0000","last_active":"2024-04-25T05:51:06.473125+0000","last_peered":"2024-04-25T05:51:06.473125+0000","last_clean":"2024-04-25T05:51:06.473125+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:51:06.473125+0000","last_undegraded":"2024-04-25T05:51:06.473125+0000","last_fullsized":"2024-04-25T05:51:06.473125+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.959063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.959063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:44.398322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":891,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471882+0000","last_change":"2024-04-25T05:49:31.857568+0000","last_active":"2024-04-25T05:51:06.471882+0000","last_peered":"2024-04-25T05:51:06.471882+0000","last_clean":"2024-04-25T05:51:06.471882+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:51:06.471882+0000","last_undegraded":"2024-04-25T05:51:06.471882+0000","last_fullsized":"2024-04-25T05:51:06.471882+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:49:31.857452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:31.857452+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907743+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":726,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473935+0000","last_change":"2024-04-25T05:50:47.087717+0000","last_active":"2024-04-25T05:51:06.473935+0000","last_peered":"2024-04-25T05:51:06.473935+0000","last_clean":"2024-04-25T05:51:06.473935+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:51:06.473935+0000","last_undegraded":"2024-04-25T05:51:06.473935+0000","last_fullsized":"2024-04-25T05:51:06.473935+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:50:47.087575+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.087575+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.480370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":899,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.467788+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:51:06.467788+0000","last_peered":"2024-04-25T05:51:06.467788+0000","last_clean":"2024-04-25T05:51:06.467788+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:51:06.467788+0000","last_undegraded":"2024-04-25T05:51:06.467788+0000","last_fullsized":"2024-04-25T05:51:06.467788+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":500,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472504+0000","last_change":"2024-04-25T05:49:30.823791+0000","last_active":"2024-04-25T05:51:06.472504+0000","last_peered":"2024-04-25T05:51:06.472504+0000","last_clean":"2024-04-25T05:51:06.472504+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:51:06.472504+0000","last_undegraded":"2024-04-25T05:51:06.472504+0000","last_fullsized":"2024-04-25T05:51:06.472504+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:30.823661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:30.823661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907709+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":887,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479096+0000","last_change":"2024-04-25T05:50:17.140640+0000","last_active":"2024-04-25T05:51:06.479096+0000","last_peered":"2024-04-25T05:51:06.479096+0000","last_clean":"2024-04-25T05:51:06.479096+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:51:06.479096+0000","last_undegraded":"2024-04-25T05:51:06.479096+0000","last_fullsized":"2024-04-25T05:51:06.479096+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:50:17.140516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.143695+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":575,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472675+0000","last_change":"2024-04-25T05:50:48.098281+0000","last_active":"2024-04-25T05:51:06.472675+0000","last_peered":"2024-04-25T05:51:06.472675+0000","last_clean":"2024-04-25T05:51:06.472675+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:51:06.472675+0000","last_undegraded":"2024-04-25T05:51:06.472675+0000","last_fullsized":"2024-04-25T05:51:06.472675+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.351413+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":891,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.480276+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:51:06.480276+0000","last_peered":"2024-04-25T05:51:06.480276+0000","last_clean":"2024-04-25T05:51:06.480276+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:51:06.480276+0000","last_undegraded":"2024-04-25T05:51:06.480276+0000","last_fullsized":"2024-04-25T05:51:06.480276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.209136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":757,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473715+0000","last_change":"2024-04-25T05:50:41.100007+0000","last_active":"2024-04-25T05:51:06.473715+0000","last_peered":"2024-04-25T05:51:06.473715+0000","last_clean":"2024-04-25T05:51:06.473715+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:51:06.473715+0000","last_undegraded":"2024-04-25T05:51:06.473715+0000","last_fullsized":"2024-04-25T05:51:06.473715+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:50:41.099870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.099870+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.548999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":918,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472314+0000","last_change":"2024-04-25T05:50:04.837659+0000","last_active":"2024-04-25T05:51:06.472314+0000","last_peered":"2024-04-25T05:51:06.472314+0000","last_clean":"2024-04-25T05:51:06.472314+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:51:06.472314+0000","last_undegraded":"2024-04-25T05:51:06.472314+0000","last_fullsized":"2024-04-25T05:51:06.472314+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:50:04.837536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.837536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.301195+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":906,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472589+0000","last_change":"2024-04-25T05:47:16.525921+0000","last_active":"2024-04-25T05:51:06.472589+0000","last_peered":"2024-04-25T05:51:06.472589+0000","last_clean":"2024-04-25T05:51:06.472589+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:51:06.472589+0000","last_undegraded":"2024-04-25T05:51:06.472589+0000","last_fullsized":"2024-04-25T05:51:06.472589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:16.525560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:16.525560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":916,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468548+0000","last_change":"2024-04-25T05:48:00.830523+0000","last_active":"2024-04-25T05:51:06.468548+0000","last_peered":"2024-04-25T05:51:06.468548+0000","last_clean":"2024-04-25T05:51:06.468548+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:51:06.468548+0000","last_undegraded":"2024-04-25T05:51:06.468548+0000","last_fullsized":"2024-04-25T05:51:06.468548+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.830408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716122+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":447,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468405+0000","last_change":"2024-04-25T05:50:09.708374+0000","last_active":"2024-04-25T05:51:06.468405+0000","last_peered":"2024-04-25T05:51:06.468405+0000","last_clean":"2024-04-25T05:51:06.468405+0000","last_became_active":"2024-04-25T05:49:09.550767+0000","last_became_peered":"2024-04-25T05:49:09.550767+0000","last_unstale":"2024-04-25T05:51:06.468405+0000","last_undegraded":"2024-04-25T05:51:06.468405+0000","last_fullsized":"2024-04-25T05:51:06.468405+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.708291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.708291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.947355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":888,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472381+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:51:06.472381+0000","last_peered":"2024-04-25T05:51:06.472381+0000","last_clean":"2024-04-25T05:51:06.472381+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:51:06.472381+0000","last_undegraded":"2024-04-25T05:51:06.472381+0000","last_fullsized":"2024-04-25T05:51:06.472381+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.657262+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":257,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.477919+0000","last_change":"2024-04-25T05:50:44.583319+0000","last_active":"2024-04-25T05:51:06.477919+0000","last_peered":"2024-04-25T05:51:06.477919+0000","last_clean":"2024-04-25T05:51:06.477919+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:51:06.477919+0000","last_undegraded":"2024-04-25T05:51:06.477919+0000","last_fullsized":"2024-04-25T05:51:06.477919+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:44.583030+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:50:44.583030+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:45.044248+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":239,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468867+0000","last_change":"2024-04-25T05:49:09.557107+0000","last_active":"2024-04-25T05:51:06.468867+0000","last_peered":"2024-04-25T05:51:06.468867+0000","last_clean":"2024-04-25T05:51:06.468867+0000","last_became_active":"2024-04-25T05:49:09.556671+0000","last_became_peered":"2024-04-25T05:49:09.556671+0000","last_unstale":"2024-04-25T05:51:06.468867+0000","last_undegraded":"2024-04-25T05:51:06.468867+0000","last_fullsized":"2024-04-25T05:51:06.468867+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":897,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479309+0000","last_change":"2024-04-25T05:48:22.532473+0000","last_active":"2024-04-25T05:51:06.479309+0000","last_peered":"2024-04-25T05:51:06.479309+0000","last_clean":"2024-04-25T05:51:06.479309+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:51:06.479309+0000","last_undegraded":"2024-04-25T05:51:06.479309+0000","last_fullsized":"2024-04-25T05:51:06.479309+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:22.532394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:22.532394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":912,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.469753+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:51:06.469753+0000","last_peered":"2024-04-25T05:51:06.469753+0000","last_clean":"2024-04-25T05:51:06.469753+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:51:06.469753+0000","last_undegraded":"2024-04-25T05:51:06.469753+0000","last_fullsized":"2024-04-25T05:51:06.469753+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716230+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":390,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468889+0000","last_change":"2024-04-25T05:49:09.554222+0000","last_active":"2024-04-25T05:51:06.468889+0000","last_peered":"2024-04-25T05:51:06.468889+0000","last_clean":"2024-04-25T05:51:06.468889+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:51:06.468889+0000","last_undegraded":"2024-04-25T05:51:06.468889+0000","last_fullsized":"2024-04-25T05:51:06.468889+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:48:36.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:36.108290+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716413+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474186+0000","last_change":"2024-04-25T05:50:50.860753+0000","last_active":"2024-04-25T05:51:06.474186+0000","last_peered":"2024-04-25T05:51:06.474186+0000","last_clean":"2024-04-25T05:51:06.474186+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:51:06.474186+0000","last_undegraded":"2024-04-25T05:51:06.474186+0000","last_fullsized":"2024-04-25T05:51:06.474186+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_clean_scrub_stamp":"2024-04-25T05:50:50.860626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:18.625619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":879,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.471776+0000","last_change":"2024-04-25T05:49:35.840397+0000","last_active":"2024-04-25T05:51:06.471776+0000","last_peered":"2024-04-25T05:51:06.471776+0000","last_clean":"2024-04-25T05:51:06.471776+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:51:06.471776+0000","last_undegraded":"2024-04-25T05:51:06.471776+0000","last_fullsized":"2024-04-25T05:51:06.471776+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.840316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.840316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.907681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":208,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473285+0000","last_change":"2024-04-25T05:48:31.213742+0000","last_active":"2024-04-25T05:51:06.473285+0000","last_peered":"2024-04-25T05:51:06.473285+0000","last_clean":"2024-04-25T05:51:06.473285+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:51:06.473285+0000","last_undegraded":"2024-04-25T05:51:06.473285+0000","last_fullsized":"2024-04-25T05:51:06.473285+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:31.213611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:31.213611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968990+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":684,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468442+0000","last_change":"2024-04-25T05:49:09.555965+0000","last_active":"2024-04-25T05:51:06.468442+0000","last_peered":"2024-04-25T05:51:06.468442+0000","last_clean":"2024-04-25T05:51:06.468442+0000","last_became_active":"2024-04-25T05:49:09.554840+0000","last_became_peered":"2024-04-25T05:49:09.554840+0000","last_unstale":"2024-04-25T05:51:06.468442+0000","last_undegraded":"2024-04-25T05:51:06.468442+0000","last_fullsized":"2024-04-25T05:51:06.468442+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.169828+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":216,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474683+0000","last_change":"2024-04-25T05:50:45.906872+0000","last_active":"2024-04-25T05:51:06.474683+0000","last_peered":"2024-04-25T05:51:06.474683+0000","last_clean":"2024-04-25T05:51:06.474683+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:51:06.474683+0000","last_undegraded":"2024-04-25T05:51:06.474683+0000","last_fullsized":"2024-04-25T05:51:06.474683+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.906734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.906734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.651537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":208,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.473912+0000","last_change":"2024-04-25T05:48:24.173416+0000","last_active":"2024-04-25T05:51:06.473912+0000","last_peered":"2024-04-25T05:51:06.473912+0000","last_clean":"2024-04-25T05:51:06.473912+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:51:06.473912+0000","last_undegraded":"2024-04-25T05:51:06.473912+0000","last_fullsized":"2024-04-25T05:51:06.473912+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:24.173135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.173135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:09.968919+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":507,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.479398+0000","last_change":"2024-04-25T05:50:08.944539+0000","last_active":"2024-04-25T05:51:06.479398+0000","last_peered":"2024-04-25T05:51:06.479398+0000","last_clean":"2024-04-25T05:51:06.479398+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:51:06.479398+0000","last_undegraded":"2024-04-25T05:51:06.479398+0000","last_fullsized":"2024-04-25T05:51:06.479398+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.210582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1172,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472417+0000","last_change":"2024-04-25T05:50:40.381895+0000","last_active":"2024-04-25T05:51:06.472417+0000","last_peered":"2024-04-25T05:51:06.472417+0000","last_clean":"2024-04-25T05:51:06.472417+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:51:06.472417+0000","last_undegraded":"2024-04-25T05:51:06.472417+0000","last_fullsized":"2024-04-25T05:51:06.472417+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:02.543415+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":908,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472665+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:51:06.472665+0000","last_peered":"2024-04-25T05:51:06.472665+0000","last_clean":"2024-04-25T05:51:06.472665+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:51:06.472665+0000","last_undegraded":"2024-04-25T05:51:06.472665+0000","last_fullsized":"2024-04-25T05:51:06.472665+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":445,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.467800+0000","last_change":"2024-04-25T05:50:40.775527+0000","last_active":"2024-04-25T05:51:06.467800+0000","last_peered":"2024-04-25T05:51:06.467800+0000","last_clean":"2024-04-25T05:51:06.467800+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:51:06.467800+0000","last_undegraded":"2024-04-25T05:51:06.467800+0000","last_fullsized":"2024-04-25T05:51:06.467800+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:40.775406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.775406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:57.019250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":874,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.472257+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:51:06.472257+0000","last_peered":"2024-04-25T05:51:06.472257+0000","last_clean":"2024-04-25T05:51:06.472257+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:51:06.472257+0000","last_undegraded":"2024-04-25T05:51:06.472257+0000","last_fullsized":"2024-04-25T05:51:06.472257+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.291166+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":981,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.467610+0000","last_change":"2024-04-25T05:50:43.844399+0000","last_active":"2024-04-25T05:51:06.467610+0000","last_peered":"2024-04-25T05:51:06.467610+0000","last_clean":"2024-04-25T05:51:06.467610+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:51:06.467610+0000","last_undegraded":"2024-04-25T05:51:06.467610+0000","last_fullsized":"2024-04-25T05:51:06.467610+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:43.844256+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.844256+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.303125+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":217,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.474720+0000","last_change":"2024-04-25T05:49:35.890470+0000","last_active":"2024-04-25T05:51:06.474720+0000","last_peered":"2024-04-25T05:51:06.474720+0000","last_clean":"2024-04-25T05:51:06.474720+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:51:06.474720+0000","last_undegraded":"2024-04-25T05:51:06.474720+0000","last_fullsized":"2024-04-25T05:51:06.474720+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.890392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.890392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":900,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.480361+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:51:06.480361+0000","last_peered":"2024-04-25T05:51:06.480361+0000","last_clean":"2024-04-25T05:51:06.480361+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:06.480361+0000","last_undegraded":"2024-04-25T05:51:06.480361+0000","last_fullsized":"2024-04-25T05:51:06.480361+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:16.929190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":421,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-04-25T05:51:06.468141+0000","last_change":"2024-04-25T05:49:09.559432+0000","last_active":"2024-04-25T05:51:06.468141+0000","last_peered":"2024-04-25T05:51:06.468141+0000","last_clean":"2024-04-25T05:51:06.468141+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:51:06.468141+0000","last_undegraded":"2024-04-25T05:51:06.468141+0000","last_fullsized":"2024-04-25T05:51:06.468141+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.227001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.227001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:07.716352+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5507514368,"num_objects":84038,"num_object_clones":0,"num_object_copies":252114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84038,"num_write_kb":5378432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16522084352,"data_stored":16522084352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71638,"ondisk_log_size":71638,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1011,"num_read_kb":1006,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":10594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070675,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3187304,"kb_used_data":2330076,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":90557848,"statfs":{"total":95995035648,"available":92731236352,"internally_reserved":0,"allocated":2385997824,"data_stored":2385405924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":877787594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365015,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3413560,"kb_used_data":2678512,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90331592,"statfs":{"total":95995035648,"available":92499550208,"internally_reserved":0,"allocated":2742796288,"data_stored":2742247860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":752679366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4032624,"kb_used_data":3058792,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":89712528,"statfs":{"total":95995035648,"available":91865628672,"internally_reserved":0,"allocated":3132203008,"data_stored":3131597464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411563,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2976144,"kb_used_data":2323016,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90769008,"statfs":{"total":95995035648,"available":92947464192,"internally_reserved":0,"allocated":2378768384,"data_stored":2378195426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149925,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3462176,"kb_used_data":2352280,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":90282976,"statfs":{"total":95995035648,"available":92449767424,"internally_reserved":0,"allocated":2408734720,"data_stored":2408145837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1136523719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986400,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2693332,"kb_used_data":1994380,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91051820,"statfs":{"total":95995035648,"available":93237063680,"internally_reserved":0,"allocated":2042245120,"data_stored":2041668202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692060,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1926084,"kb_used_data":1335420,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":91819068,"statfs":{"total":95995035648,"available":94022725632,"internally_reserved":0,"allocated":1367470080,"data_stored":1366909494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3131572224,"data_stored":3131572224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2385969152,"data_stored":2385969152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2432827392,"data_stored":2432827392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2051342336,"data_stored":2051342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1368588288,"data_stored":1368588288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2754150400,"data_stored":2754150400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2397634560,"data_stored":2397634560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:51:12.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:51:12.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:12.641 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:12.743 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:12.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:12.841 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:12 smithi043 ceph-mon[27010]: from='client.25966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:12.842 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:12 smithi043 ceph-mon[37492]: from='client.25966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:12.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:12.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:12.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:12.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:12 smithi060 ceph-mon[30898]: from='client.25966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:13.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:13.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:13.136 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:13.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:13.220 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 53 15 90100 90085 106.178 77.375 0.0091425 0.0094128 2024-04-25T05:51:13.224 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:13.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:13.356 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:13.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:13.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.644 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:13.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:13.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:13.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:13 smithi043 ceph-mon[27010]: pgmap v906: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 776 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:51:13.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:13 smithi043 ceph-mon[37492]: pgmap v906: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 776 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:51:13.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:13.780 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:13.869 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:51:13.870 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:51:13.870 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:51:13.870 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:51:13.870 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:51:13.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:13.926 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:13.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:13.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:51:13.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:13 smithi060 ceph-mon[30898]: pgmap v906: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 776 B/s rd, 61 MiB/s wr, 978 op/s 2024-04-25T05:51:13.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:14.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:14.059 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:14.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:14.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:14.196 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:14.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:14.220 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 54 16 91258 91242 105.551 72.3125 0.010569 0.009469 2024-04-25T05:51:14.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:14.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:14.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:14.433 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:14.473 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:14.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:14.819 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:51:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[27010]: osdmap e482: 8 total, 8 up, 7 in 2024-04-25T05:51:14.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[37492]: osdmap e482: 8 total, 8 up, 7 in 2024-04-25T05:51:14.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:14 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:14.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:14.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:14 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:14 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:14 smithi060 ceph-mon[30898]: osdmap e482: 8 total, 8 up, 7 in 2024-04-25T05:51:14.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:14 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:15.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:51:15.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:15.112 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:15.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:15.220 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 55 16 92591 92575 105.147 83.3125 0.00998077 0.0095056 2024-04-25T05:51:15.244 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:15.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:15.394 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:15.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:15.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:15.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: pgmap v908: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:51:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: osdmap e483: 8 total, 8 up, 7 in 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[27010]: osdmap e484: 8 total, 8 up, 7 in 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: pgmap v908: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: osdmap e483: 8 total, 8 up, 7 in 2024-04-25T05:51:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:51:15.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:51:15.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:15 smithi043 ceph-mon[37492]: osdmap e484: 8 total, 8 up, 7 in 2024-04-25T05:51:15.665 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:15.767 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:15.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:15.911 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:15.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:15.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: pgmap v908: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1162601435' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: osdmap e483: 8 total, 8 up, 7 in 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:51:15.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:15 smithi060 ceph-mon[30898]: osdmap e484: 8 total, 8 up, 7 in 2024-04-25T05:51:16.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:16.072 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:16.107 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:16.123 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:16.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:16.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:16.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 56 16 93876 93860 104.703 80.3125 0.0159853 0.00954626 2024-04-25T05:51:16.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:16.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:16.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.529 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:16.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:16.533 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:51:16.538 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:51:16.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:16.657 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:16.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:16.795 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[27010]: pgmap v911: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[27010]: from='client.26375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[37492]: pgmap v911: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:51:16.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:16 smithi043 ceph-mon[37492]: from='client.26375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:16.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:16.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:16.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:16 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:51:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:16 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:51:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:16 smithi060 ceph-mon[30898]: pgmap v911: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 920 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-25T05:51:16.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:16 smithi060 ceph-mon[30898]: from='client.26375 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:17.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.082 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:17.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:17.159 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:17.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:17.205 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:17.213 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:17.222 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 57 16 95055 95039 104.159 73.6875 0.0162012 0.00959565 2024-04-25T05:51:17.243 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:17.285 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:17.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:17.330 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:17.377 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:17.418 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:17.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:17.468 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:17.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:17.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:17.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:17.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:17.471 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:17.472 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:17.472 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:17.472 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:17.485 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:17.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:17.617 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:17.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:17.754 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:17.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1226278512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:51:17.830 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1226278512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:17.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:17.888 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:17.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:17.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:17.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:17 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:51:17.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:17 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:51:17.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1226278512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:18.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:18.144 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:18.144 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:18.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:18.220 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 58 16 96249 96233 103.649 74.625 0.00506361 0.00963719 2024-04-25T05:51:18.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:18.315 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:18.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:18.455 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:18.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:18.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:18.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:18.723 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:18.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:18.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:18.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:51:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:51:18.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[27010]: pgmap v912: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 851 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:51:18.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:51:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:51:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[37492]: pgmap v912: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 851 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:51:18.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:18 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:18.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:18.998 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:18 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:51:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:18 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:51:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:18 smithi060 ceph-mon[30898]: pgmap v912: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 851 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-25T05:51:18.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:18 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:19.039 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":911,"stamp":"2024-04-25T05:51:16.033014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5924067867,"num_objects":90615,"num_object_clones":0,"num_object_copies":271845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90615,"num_whiteouts":0,"num_read":2429,"num_read_kb":2149,"num_write":91299,"num_write_kb":5784685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":626,"num_bytes_recovered":1198290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74298,"ondisk_log_size":74298,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22952256,"kb_used_data":17300220,"kb_used_omap":10,"kb_used_meta":5651957,"kb_avail":633263808,"statfs":{"total":671965249536,"available":648462139392,"internally_reserved":0,"allocated":17715425280,"data_stored":17711339191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5787604075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,3,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":415956992,"num_objects":6347,"num_object_clones":0,"num_object_copies":19041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6347,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6347,"num_write_kb":406208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.560427"},"pg_stats":[{"pgid":"11.3","version":"482'5647","reported_seq":5708,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.573580+0000","last_change":"2024-04-25T05:50:12.040793+0000","last_active":"2024-04-25T05:51:13.573580+0000","last_peered":"2024-04-25T05:51:13.573580+0000","last_clean":"2024-04-25T05:51:13.573580+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:51:13.573580+0000","last_undegraded":"2024-04-25T05:51:13.573580+0000","last_fullsized":"2024-04-25T05:51:13.573580+0000","mapping_epoch":456,"log_start":"464'800","ondisk_log_start":"464'800","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.267931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370081792,"num_objects":5647,"num_object_clones":0,"num_object_copies":16941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5647,"num_write_kb":361408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":377,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470830+0000","last_change":"2024-04-25T05:50:08.705811+0000","last_active":"2024-04-25T05:51:10.470830+0000","last_peered":"2024-04-25T05:51:10.470830+0000","last_clean":"2024-04-25T05:51:10.470830+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:51:10.470830+0000","last_undegraded":"2024-04-25T05:51:10.470830+0000","last_fullsized":"2024-04-25T05:51:10.470830+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:50:08.705677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.705677+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.222090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":897,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533795+0000","last_change":"2024-04-25T05:49:05.117179+0000","last_active":"2024-04-25T05:51:13.533795+0000","last_peered":"2024-04-25T05:51:13.533795+0000","last_clean":"2024-04-25T05:51:13.533795+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:51:13.533795+0000","last_undegraded":"2024-04-25T05:51:13.533795+0000","last_fullsized":"2024-04-25T05:51:13.533795+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391123+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":308,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490154+0000","last_change":"2024-04-25T05:50:12.546520+0000","last_active":"2024-04-25T05:51:10.490154+0000","last_peered":"2024-04-25T05:51:10.490154+0000","last_clean":"2024-04-25T05:51:10.490154+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:51:10.490154+0000","last_undegraded":"2024-04-25T05:51:10.490154+0000","last_fullsized":"2024-04-25T05:51:10.490154+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:50:12.546342+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.546342+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.973854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":454,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471914+0000","last_change":"2024-04-25T05:50:13.709776+0000","last_active":"2024-04-25T05:51:10.471914+0000","last_peered":"2024-04-25T05:51:10.471914+0000","last_clean":"2024-04-25T05:51:10.471914+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:51:10.471914+0000","last_undegraded":"2024-04-25T05:51:10.471914+0000","last_fullsized":"2024-04-25T05:51:10.471914+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.709675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.709675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.101183+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1209,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484621+0000","last_change":"2024-04-25T05:49:33.878907+0000","last_active":"2024-04-25T05:51:10.484621+0000","last_peered":"2024-04-25T05:51:10.484621+0000","last_clean":"2024-04-25T05:51:10.484621+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:51:10.484621+0000","last_undegraded":"2024-04-25T05:51:10.484621+0000","last_fullsized":"2024-04-25T05:51:10.484621+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:33.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:33.878785+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918857+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"481'5620","reported_seq":5680,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:12.711440+0000","last_change":"2024-04-25T05:50:12.040238+0000","last_active":"2024-04-25T05:51:12.711440+0000","last_peered":"2024-04-25T05:51:12.711440+0000","last_clean":"2024-04-25T05:51:12.711440+0000","last_became_active":"2024-04-25T05:50:12.039934+0000","last_became_peered":"2024-04-25T05:50:12.039934+0000","last_unstale":"2024-04-25T05:51:12.711440+0000","last_undegraded":"2024-04-25T05:51:12.711440+0000","last_fullsized":"2024-04-25T05:51:12.711440+0000","mapping_epoch":456,"log_start":"464'700","ondisk_log_start":"464'700","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.997447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368312320,"num_objects":5620,"num_object_clones":0,"num_object_copies":16860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":359680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":378,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471518+0000","last_change":"2024-04-25T05:49:09.556929+0000","last_active":"2024-04-25T05:51:10.471518+0000","last_peered":"2024-04-25T05:51:10.471518+0000","last_clean":"2024-04-25T05:51:10.471518+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:51:10.471518+0000","last_undegraded":"2024-04-25T05:51:10.471518+0000","last_fullsized":"2024-04-25T05:51:10.471518+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:48:59.073429+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.073429+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":610,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539239+0000","last_change":"2024-04-25T05:49:32.886598+0000","last_active":"2024-04-25T05:51:13.539239+0000","last_peered":"2024-04-25T05:51:13.539239+0000","last_clean":"2024-04-25T05:51:13.539239+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:51:13.539239+0000","last_undegraded":"2024-04-25T05:51:13.539239+0000","last_fullsized":"2024-04-25T05:51:13.539239+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.886504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.886504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":910,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535641+0000","last_change":"2024-04-25T05:50:55.073084+0000","last_active":"2024-04-25T05:51:13.535641+0000","last_peered":"2024-04-25T05:51:13.535641+0000","last_clean":"2024-04-25T05:51:13.535641+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:51:13.535641+0000","last_undegraded":"2024-04-25T05:51:13.535641+0000","last_fullsized":"2024-04-25T05:51:13.535641+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:15.963221+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":244,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483207+0000","last_change":"2024-04-25T05:50:46.862960+0000","last_active":"2024-04-25T05:51:10.483207+0000","last_peered":"2024-04-25T05:51:10.483207+0000","last_clean":"2024-04-25T05:51:10.483207+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:51:10.483207+0000","last_undegraded":"2024-04-25T05:51:10.483207+0000","last_fullsized":"2024-04-25T05:51:10.483207+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:46.862820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.862820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:55.777488+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":808,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489423+0000","last_change":"2024-04-25T05:50:06.715616+0000","last_active":"2024-04-25T05:51:10.489423+0000","last_peered":"2024-04-25T05:51:10.489423+0000","last_clean":"2024-04-25T05:51:10.489423+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:51:10.489423+0000","last_undegraded":"2024-04-25T05:51:10.489423+0000","last_fullsized":"2024-04-25T05:51:10.489423+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:50:06.715476+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.715476+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:29.983548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"482'5661","reported_seq":5722,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.546890+0000","last_change":"2024-04-25T05:50:12.035436+0000","last_active":"2024-04-25T05:51:13.546890+0000","last_peered":"2024-04-25T05:51:13.546890+0000","last_clean":"2024-04-25T05:51:13.546890+0000","last_became_active":"2024-04-25T05:50:12.035111+0000","last_became_peered":"2024-04-25T05:50:12.035111+0000","last_unstale":"2024-04-25T05:51:13.546890+0000","last_undegraded":"2024-04-25T05:51:13.546890+0000","last_fullsized":"2024-04-25T05:51:13.546890+0000","mapping_epoch":456,"log_start":"464'800","ondisk_log_start":"464'800","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.368286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":16983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5661,"num_write_kb":362304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":686,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471714+0000","last_change":"2024-04-25T05:50:11.713907+0000","last_active":"2024-04-25T05:51:10.471714+0000","last_peered":"2024-04-25T05:51:10.471714+0000","last_clean":"2024-04-25T05:51:10.471714+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:51:10.471714+0000","last_undegraded":"2024-04-25T05:51:10.471714+0000","last_fullsized":"2024-04-25T05:51:10.471714+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:50:11.713794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.713794+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.646485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":913,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470528+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:51:10.470528+0000","last_peered":"2024-04-25T05:51:10.470528+0000","last_clean":"2024-04-25T05:51:10.470528+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:51:10.470528+0000","last_undegraded":"2024-04-25T05:51:10.470528+0000","last_fullsized":"2024-04-25T05:51:10.470528+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491742+0000","last_change":"2024-04-25T05:49:29.951087+0000","last_active":"2024-04-25T05:51:10.491742+0000","last_peered":"2024-04-25T05:51:10.491742+0000","last_clean":"2024-04-25T05:51:10.491742+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:51:10.491742+0000","last_undegraded":"2024-04-25T05:51:10.491742+0000","last_fullsized":"2024-04-25T05:51:10.491742+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:29.950964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:49:29.950964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":891,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.527105+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:51:13.527105+0000","last_peered":"2024-04-25T05:51:13.527105+0000","last_clean":"2024-04-25T05:51:13.527105+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:51:13.527105+0000","last_undegraded":"2024-04-25T05:51:13.527105+0000","last_fullsized":"2024-04-25T05:51:13.527105+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.640676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"482'5687","reported_seq":5748,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.122144+0000","last_change":"2024-04-25T05:50:12.041469+0000","last_active":"2024-04-25T05:51:14.122144+0000","last_peered":"2024-04-25T05:51:14.122144+0000","last_clean":"2024-04-25T05:51:14.122144+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:51:14.122144+0000","last_undegraded":"2024-04-25T05:51:14.122144+0000","last_fullsized":"2024-04-25T05:51:14.122144+0000","mapping_epoch":456,"log_start":"466'1600","ondisk_log_start":"466'1600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372703232,"num_objects":5687,"num_object_clones":0,"num_object_copies":17061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5687,"num_write_kb":363968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":932,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536346+0000","last_change":"2024-04-25T05:50:10.992094+0000","last_active":"2024-04-25T05:51:13.536346+0000","last_peered":"2024-04-25T05:51:13.536346+0000","last_clean":"2024-04-25T05:51:13.536346+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:51:13.536346+0000","last_undegraded":"2024-04-25T05:51:13.536346+0000","last_fullsized":"2024-04-25T05:51:13.536346+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:10.991997+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:50:10.991997+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.924649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":879,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536662+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:51:13.536662+0000","last_peered":"2024-04-25T05:51:13.536662+0000","last_clean":"2024-04-25T05:51:13.536662+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:51:13.536662+0000","last_undegraded":"2024-04-25T05:51:13.536662+0000","last_fullsized":"2024-04-25T05:51:13.536662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390808+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"433'16","reported_seq":545,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471831+0000","last_change":"2024-04-25T05:49:09.610647+0000","last_active":"2024-04-25T05:51:10.471831+0000","last_peered":"2024-04-25T05:51:10.471831+0000","last_clean":"2024-04-25T05:51:10.471831+0000","last_became_active":"2024-04-25T05:49:09.561406+0000","last_became_peered":"2024-04-25T05:49:09.561406+0000","last_unstale":"2024-04-25T05:51:10.471831+0000","last_undegraded":"2024-04-25T05:51:10.471831+0000","last_fullsized":"2024-04-25T05:51:10.471831+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-25T05:48:21.090099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.090099+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491294+0000","last_change":"2024-04-25T05:48:30.199408+0000","last_active":"2024-04-25T05:51:10.491294+0000","last_peered":"2024-04-25T05:51:10.491294+0000","last_clean":"2024-04-25T05:51:10.491294+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:51:10.491294+0000","last_undegraded":"2024-04-25T05:51:10.491294+0000","last_fullsized":"2024-04-25T05:51:10.491294+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:30.199282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.199282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":772,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490089+0000","last_change":"2024-04-25T05:50:47.941283+0000","last_active":"2024-04-25T05:51:10.490089+0000","last_peered":"2024-04-25T05:51:10.490089+0000","last_clean":"2024-04-25T05:51:10.490089+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:51:10.490089+0000","last_undegraded":"2024-04-25T05:51:10.490089+0000","last_fullsized":"2024-04-25T05:51:10.490089+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:47.940906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.940906+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.568875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"481'5708","reported_seq":5766,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.232607+0000","last_change":"2024-04-25T05:50:12.042142+0000","last_active":"2024-04-25T05:51:13.232607+0000","last_peered":"2024-04-25T05:51:13.232607+0000","last_clean":"2024-04-25T05:51:13.232607+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:51:13.232607+0000","last_undegraded":"2024-04-25T05:51:13.232607+0000","last_fullsized":"2024-04-25T05:51:13.232607+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.962178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":365312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":894,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470437+0000","last_change":"2024-04-25T05:49:03.437727+0000","last_active":"2024-04-25T05:51:10.470437+0000","last_peered":"2024-04-25T05:51:10.470437+0000","last_clean":"2024-04-25T05:51:10.470437+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:51:10.470437+0000","last_undegraded":"2024-04-25T05:51:10.470437+0000","last_fullsized":"2024-04-25T05:51:10.470437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:03.437604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:03.437604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1034,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536757+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:51:13.536757+0000","last_peered":"2024-04-25T05:51:13.536757+0000","last_clean":"2024-04-25T05:51:13.536757+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:51:13.536757+0000","last_undegraded":"2024-04-25T05:51:13.536757+0000","last_fullsized":"2024-04-25T05:51:13.536757+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390876+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":911,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.532991+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:51:13.532991+0000","last_peered":"2024-04-25T05:51:13.532991+0000","last_clean":"2024-04-25T05:51:13.532991+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:51:13.532991+0000","last_undegraded":"2024-04-25T05:51:13.532991+0000","last_fullsized":"2024-04-25T05:51:13.532991+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":448,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471006+0000","last_change":"2024-04-25T05:50:41.801622+0000","last_active":"2024-04-25T05:51:10.471006+0000","last_peered":"2024-04-25T05:51:10.471006+0000","last_clean":"2024-04-25T05:51:10.471006+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:51:10.471006+0000","last_undegraded":"2024-04-25T05:51:10.471006+0000","last_fullsized":"2024-04-25T05:51:10.471006+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.801508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.801508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.561567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1123,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470445+0000","last_change":"2024-04-25T05:49:01.441992+0000","last_active":"2024-04-25T05:51:10.470445+0000","last_peered":"2024-04-25T05:51:10.470445+0000","last_clean":"2024-04-25T05:51:10.470445+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:51:10.470445+0000","last_undegraded":"2024-04-25T05:51:10.470445+0000","last_fullsized":"2024-04-25T05:51:10.470445+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:01.440957+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.440957+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"481'5485","reported_seq":5545,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.085521+0000","last_change":"2024-04-25T05:50:12.042923+0000","last_active":"2024-04-25T05:51:13.085521+0000","last_peered":"2024-04-25T05:51:13.085521+0000","last_clean":"2024-04-25T05:51:13.085521+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:51:13.085521+0000","last_undegraded":"2024-04-25T05:51:13.085521+0000","last_fullsized":"2024-04-25T05:51:13.085521+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.401319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359464960,"num_objects":5485,"num_object_clones":0,"num_object_copies":16455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":351040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1098,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539685+0000","last_change":"2024-04-25T05:48:25.100906+0000","last_active":"2024-04-25T05:51:13.539685+0000","last_peered":"2024-04-25T05:51:13.539685+0000","last_clean":"2024-04-25T05:51:13.539685+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:51:13.539685+0000","last_undegraded":"2024-04-25T05:51:13.539685+0000","last_fullsized":"2024-04-25T05:51:13.539685+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:48:25.100752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.100752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391029+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":578,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470799+0000","last_change":"2024-04-25T05:50:46.875806+0000","last_active":"2024-04-25T05:51:10.470799+0000","last_peered":"2024-04-25T05:51:10.470799+0000","last_clean":"2024-04-25T05:51:10.470799+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:51:10.470799+0000","last_undegraded":"2024-04-25T05:51:10.470799+0000","last_fullsized":"2024-04-25T05:51:10.470799+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:46.875662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.875662+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:49.181320+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":591,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536396+0000","last_change":"2024-04-25T05:50:15.054217+0000","last_active":"2024-04-25T05:51:13.536396+0000","last_peered":"2024-04-25T05:51:13.536396+0000","last_clean":"2024-04-25T05:51:13.536396+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:51:13.536396+0000","last_undegraded":"2024-04-25T05:51:13.536396+0000","last_fullsized":"2024-04-25T05:51:13.536396+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.219601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":269,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.537680+0000","last_change":"2024-04-25T05:50:12.172126+0000","last_active":"2024-04-25T05:51:13.537680+0000","last_peered":"2024-04-25T05:51:13.537680+0000","last_clean":"2024-04-25T05:51:13.537680+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:51:13.537680+0000","last_undegraded":"2024-04-25T05:51:13.537680+0000","last_fullsized":"2024-04-25T05:51:13.537680+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:12.172001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.172001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.707108+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1081,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.526903+0000","last_change":"2024-04-25T05:48:27.160787+0000","last_active":"2024-04-25T05:51:13.526903+0000","last_peered":"2024-04-25T05:51:13.526903+0000","last_clean":"2024-04-25T05:51:13.526903+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:51:13.526903+0000","last_undegraded":"2024-04-25T05:51:13.526903+0000","last_fullsized":"2024-04-25T05:51:13.526903+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:48:27.160639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.160639+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391061+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"481'5522","reported_seq":5582,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:12.727570+0000","last_change":"2024-04-25T05:50:12.035986+0000","last_active":"2024-04-25T05:51:12.727570+0000","last_peered":"2024-04-25T05:51:12.727570+0000","last_clean":"2024-04-25T05:51:12.727570+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:51:12.727570+0000","last_undegraded":"2024-04-25T05:51:12.727570+0000","last_fullsized":"2024-04-25T05:51:12.727570+0000","mapping_epoch":456,"log_start":"464'600","ondisk_log_start":"464'600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.423843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361889792,"num_objects":5522,"num_object_clones":0,"num_object_copies":16566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":353408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":695,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539837+0000","last_change":"2024-04-25T05:50:45.130550+0000","last_active":"2024-04-25T05:51:13.539837+0000","last_peered":"2024-04-25T05:51:13.539837+0000","last_clean":"2024-04-25T05:51:13.539837+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:51:13.539837+0000","last_undegraded":"2024-04-25T05:51:13.539837+0000","last_fullsized":"2024-04-25T05:51:13.539837+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:51.401906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":628,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.532919+0000","last_change":"2024-04-25T05:49:02.359488+0000","last_active":"2024-04-25T05:51:13.532919+0000","last_peered":"2024-04-25T05:51:13.532919+0000","last_clean":"2024-04-25T05:51:13.532919+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:51:13.532919+0000","last_undegraded":"2024-04-25T05:51:13.532919+0000","last_fullsized":"2024-04-25T05:51:13.532919+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:49:02.359173+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.359173+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":219,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491385+0000","last_change":"2024-04-25T05:50:08.761599+0000","last_active":"2024-04-25T05:51:10.491385+0000","last_peered":"2024-04-25T05:51:10.491385+0000","last_clean":"2024-04-25T05:51:10.491385+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:51:10.491385+0000","last_undegraded":"2024-04-25T05:51:10.491385+0000","last_fullsized":"2024-04-25T05:51:10.491385+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.520460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":526,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536159+0000","last_change":"2024-04-25T05:50:09.086597+0000","last_active":"2024-04-25T05:51:13.536159+0000","last_peered":"2024-04-25T05:51:13.536159+0000","last_clean":"2024-04-25T05:51:13.536159+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:51:13.536159+0000","last_undegraded":"2024-04-25T05:51:13.536159+0000","last_fullsized":"2024-04-25T05:51:13.536159+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.086479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.086479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.899606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1073,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539753+0000","last_change":"2024-04-25T05:50:07.959527+0000","last_active":"2024-04-25T05:51:13.539753+0000","last_peered":"2024-04-25T05:51:13.539753+0000","last_clean":"2024-04-25T05:51:13.539753+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:51:13.539753+0000","last_undegraded":"2024-04-25T05:51:13.539753+0000","last_fullsized":"2024-04-25T05:51:13.539753+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.959446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.959446+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:24.723125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"482'5850","reported_seq":5911,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.124133+0000","last_change":"2024-04-25T05:50:12.038131+0000","last_active":"2024-04-25T05:51:14.124133+0000","last_peered":"2024-04-25T05:51:14.124133+0000","last_clean":"2024-04-25T05:51:14.124133+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:51:14.124133+0000","last_undegraded":"2024-04-25T05:51:14.124133+0000","last_fullsized":"2024-04-25T05:51:14.124133+0000","mapping_epoch":456,"log_start":"466'1800","ondisk_log_start":"466'1800","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4050,"log_dups_size":0,"ondisk_log_size":4050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.865063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383385600,"num_objects":5850,"num_object_clones":0,"num_object_copies":17550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":374400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":648,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539318+0000","last_change":"2024-04-25T05:50:42.148569+0000","last_active":"2024-04-25T05:51:13.539318+0000","last_peered":"2024-04-25T05:51:13.539318+0000","last_clean":"2024-04-25T05:51:13.539318+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:51:13.539318+0000","last_undegraded":"2024-04-25T05:51:13.539318+0000","last_fullsized":"2024-04-25T05:51:13.539318+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:42.148429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:50:42.148429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:11.882512+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":257,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490236+0000","last_change":"2024-04-25T05:48:59.056819+0000","last_active":"2024-04-25T05:51:10.490236+0000","last_peered":"2024-04-25T05:51:10.490236+0000","last_clean":"2024-04-25T05:51:10.490236+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:51:10.490236+0000","last_undegraded":"2024-04-25T05:51:10.490236+0000","last_fullsized":"2024-04-25T05:51:10.490236+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:48:59.056469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.056469+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":925,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535388+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:51:13.535388+0000","last_peered":"2024-04-25T05:51:13.535388+0000","last_clean":"2024-04-25T05:51:13.535388+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:51:13.535388+0000","last_undegraded":"2024-04-25T05:51:13.535388+0000","last_fullsized":"2024-04-25T05:51:13.535388+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.818280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":899,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.534259+0000","last_change":"2024-04-25T05:49:32.024307+0000","last_active":"2024-04-25T05:51:13.534259+0000","last_peered":"2024-04-25T05:51:13.534259+0000","last_clean":"2024-04-25T05:51:13.534259+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:51:13.534259+0000","last_undegraded":"2024-04-25T05:51:13.534259+0000","last_fullsized":"2024-04-25T05:51:13.534259+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.024156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.024156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1091,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.538457+0000","last_change":"2024-04-25T05:50:01.929134+0000","last_active":"2024-04-25T05:51:13.538457+0000","last_peered":"2024-04-25T05:51:13.538457+0000","last_clean":"2024-04-25T05:51:13.538457+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:51:13.538457+0000","last_undegraded":"2024-04-25T05:51:13.538457+0000","last_fullsized":"2024-04-25T05:51:13.538457+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:01.929038+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:50:01.929038+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:20.992655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"482'5752","reported_seq":5814,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.113929+0000","last_change":"2024-04-25T05:50:12.035979+0000","last_active":"2024-04-25T05:51:14.113929+0000","last_peered":"2024-04-25T05:51:14.113929+0000","last_clean":"2024-04-25T05:51:14.113929+0000","last_became_active":"2024-04-25T05:50:12.035482+0000","last_became_peered":"2024-04-25T05:50:12.035482+0000","last_unstale":"2024-04-25T05:51:14.113929+0000","last_undegraded":"2024-04-25T05:51:14.113929+0000","last_fullsized":"2024-04-25T05:51:14.113929+0000","mapping_epoch":456,"log_start":"465'1600","ondisk_log_start":"465'1600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.703340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376963072,"num_objects":5752,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5752,"num_write_kb":368128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491831+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:51:10.491831+0000","last_peered":"2024-04-25T05:51:10.491831+0000","last_clean":"2024-04-25T05:51:10.491831+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:51:10.491831+0000","last_undegraded":"2024-04-25T05:51:10.491831+0000","last_fullsized":"2024-04-25T05:51:10.491831+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":233,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491239+0000","last_change":"2024-04-25T05:50:12.786445+0000","last_active":"2024-04-25T05:51:10.491239+0000","last_peered":"2024-04-25T05:51:10.491239+0000","last_clean":"2024-04-25T05:51:10.491239+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:51:10.491239+0000","last_undegraded":"2024-04-25T05:51:10.491239+0000","last_fullsized":"2024-04-25T05:51:10.491239+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:50:12.786333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.786333+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.425682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1952,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:11.969328+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:51:11.969328+0000","last_peered":"2024-04-25T05:51:11.969328+0000","last_clean":"2024-04-25T05:51:11.969328+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:51:11.969328+0000","last_undegraded":"2024-04-25T05:51:11.969328+0000","last_fullsized":"2024-04-25T05:51:11.969328+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918698+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1016,"num_read_kb":1011,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":519,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470629+0000","last_change":"2024-04-25T05:49:05.488606+0000","last_active":"2024-04-25T05:51:10.470629+0000","last_peered":"2024-04-25T05:51:10.470629+0000","last_clean":"2024-04-25T05:51:10.470629+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:51:10.470629+0000","last_undegraded":"2024-04-25T05:51:10.470629+0000","last_fullsized":"2024-04-25T05:51:10.470629+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.488514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.488514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":887,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535057+0000","last_change":"2024-04-25T05:48:35.483025+0000","last_active":"2024-04-25T05:51:13.535057+0000","last_peered":"2024-04-25T05:51:13.535057+0000","last_clean":"2024-04-25T05:51:13.535057+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:51:13.535057+0000","last_undegraded":"2024-04-25T05:51:13.535057+0000","last_fullsized":"2024-04-25T05:51:13.535057+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.482886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.482886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1069,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.537570+0000","last_change":"2024-04-25T05:49:08.317583+0000","last_active":"2024-04-25T05:51:13.537570+0000","last_peered":"2024-04-25T05:51:13.537570+0000","last_clean":"2024-04-25T05:51:13.537570+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:51:13.537570+0000","last_undegraded":"2024-04-25T05:51:13.537570+0000","last_fullsized":"2024-04-25T05:51:13.537570+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_clean_scrub_stamp":"2024-04-25T05:49:08.317433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"482'5627","reported_seq":5689,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.107643+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:51:14.107643+0000","last_peered":"2024-04-25T05:51:14.107643+0000","last_clean":"2024-04-25T05:51:14.107643+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:51:14.107643+0000","last_undegraded":"2024-04-25T05:51:14.107643+0000","last_fullsized":"2024-04-25T05:51:14.107643+0000","mapping_epoch":456,"log_start":"465'1500","ondisk_log_start":"465'1500","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.159052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368771072,"num_objects":5627,"num_object_clones":0,"num_object_copies":16881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":360128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":907,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.485186+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:51:10.485186+0000","last_peered":"2024-04-25T05:51:10.485186+0000","last_clean":"2024-04-25T05:51:10.485186+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:51:10.485186+0000","last_undegraded":"2024-04-25T05:51:10.485186+0000","last_fullsized":"2024-04-25T05:51:10.485186+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918680+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":794,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539680+0000","last_change":"2024-04-25T05:50:13.028112+0000","last_active":"2024-04-25T05:51:13.539680+0000","last_peered":"2024-04-25T05:51:13.539680+0000","last_clean":"2024-04-25T05:51:13.539680+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:51:13.539680+0000","last_undegraded":"2024-04-25T05:51:13.539680+0000","last_fullsized":"2024-04-25T05:51:13.539680+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.027901+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.798452+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":904,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535886+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:51:13.535886+0000","last_peered":"2024-04-25T05:51:13.535886+0000","last_clean":"2024-04-25T05:51:13.535886+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:51:13.535886+0000","last_undegraded":"2024-04-25T05:51:13.535886+0000","last_fullsized":"2024-04-25T05:51:13.535886+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1202,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484050+0000","last_change":"2024-04-25T05:50:07.888799+0000","last_active":"2024-04-25T05:51:10.484050+0000","last_peered":"2024-04-25T05:51:10.484050+0000","last_clean":"2024-04-25T05:51:10.484050+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:51:10.484050+0000","last_undegraded":"2024-04-25T05:51:10.484050+0000","last_fullsized":"2024-04-25T05:51:10.484050+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.888714+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.888714+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.057112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":193,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536354+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:51:13.536354+0000","last_peered":"2024-04-25T05:51:13.536354+0000","last_clean":"2024-04-25T05:51:13.536354+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:51:13.536354+0000","last_undegraded":"2024-04-25T05:51:13.536354+0000","last_fullsized":"2024-04-25T05:51:13.536354+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.250098+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"482'5697","reported_seq":5759,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.116659+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:51:14.116659+0000","last_peered":"2024-04-25T05:51:14.116659+0000","last_clean":"2024-04-25T05:51:14.116659+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:51:14.116659+0000","last_undegraded":"2024-04-25T05:51:14.116659+0000","last_fullsized":"2024-04-25T05:51:14.116659+0000","mapping_epoch":456,"log_start":"465'1500","ondisk_log_start":"465'1500","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.808327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373358592,"num_objects":5697,"num_object_clones":0,"num_object_copies":17091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5697,"num_write_kb":364608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":250,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483149+0000","last_change":"2024-04-25T05:50:45.550588+0000","last_active":"2024-04-25T05:51:10.483149+0000","last_peered":"2024-04-25T05:51:10.483149+0000","last_clean":"2024-04-25T05:51:10.483149+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:51:10.483149+0000","last_undegraded":"2024-04-25T05:51:10.483149+0000","last_fullsized":"2024-04-25T05:51:10.483149+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.550474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.550474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.043449+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1033,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536212+0000","last_change":"2024-04-25T05:50:40.134940+0000","last_active":"2024-04-25T05:51:13.536212+0000","last_peered":"2024-04-25T05:51:13.536212+0000","last_clean":"2024-04-25T05:51:13.536212+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:51:13.536212+0000","last_undegraded":"2024-04-25T05:51:13.536212+0000","last_fullsized":"2024-04-25T05:51:13.536212+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:50:40.134604+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.134604+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:41.529906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":244,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491939+0000","last_change":"2024-04-25T05:50:14.628452+0000","last_active":"2024-04-25T05:51:10.491939+0000","last_peered":"2024-04-25T05:51:10.491939+0000","last_clean":"2024-04-25T05:51:10.491939+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:51:10.491939+0000","last_undegraded":"2024-04-25T05:51:10.491939+0000","last_fullsized":"2024-04-25T05:51:10.491939+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.601623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":931,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535260+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:51:13.535260+0000","last_peered":"2024-04-25T05:51:13.535260+0000","last_clean":"2024-04-25T05:51:13.535260+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:51:13.535260+0000","last_undegraded":"2024-04-25T05:51:13.535260+0000","last_fullsized":"2024-04-25T05:51:13.535260+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.323525+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":885,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484675+0000","last_change":"2024-04-25T05:47:23.888942+0000","last_active":"2024-04-25T05:51:10.484675+0000","last_peered":"2024-04-25T05:51:10.484675+0000","last_clean":"2024-04-25T05:51:10.484675+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:51:10.484675+0000","last_undegraded":"2024-04-25T05:51:10.484675+0000","last_fullsized":"2024-04-25T05:51:10.484675+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:23.888825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:47:23.888825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918631+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484758+0000","last_change":"2024-04-25T05:49:32.818373+0000","last_active":"2024-04-25T05:51:10.484758+0000","last_peered":"2024-04-25T05:51:10.484758+0000","last_clean":"2024-04-25T05:51:10.484758+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:51:10.484758+0000","last_undegraded":"2024-04-25T05:51:10.484758+0000","last_fullsized":"2024-04-25T05:51:10.484758+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:32.818274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.818274+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"482'5744","reported_seq":5806,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:14.045384+0000","last_change":"2024-04-25T05:50:12.040668+0000","last_active":"2024-04-25T05:51:14.045384+0000","last_peered":"2024-04-25T05:51:14.045384+0000","last_clean":"2024-04-25T05:51:14.045384+0000","last_became_active":"2024-04-25T05:50:12.040023+0000","last_became_peered":"2024-04-25T05:50:12.040023+0000","last_unstale":"2024-04-25T05:51:14.045384+0000","last_undegraded":"2024-04-25T05:51:14.045384+0000","last_fullsized":"2024-04-25T05:51:14.045384+0000","mapping_epoch":456,"log_start":"465'1600","ondisk_log_start":"465'1600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4144,"log_dups_size":0,"ondisk_log_size":4144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:36.039935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":512,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533319+0000","last_change":"2024-04-25T05:48:29.428977+0000","last_active":"2024-04-25T05:51:13.533319+0000","last_peered":"2024-04-25T05:51:13.533319+0000","last_clean":"2024-04-25T05:51:13.533319+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:51:13.533319+0000","last_undegraded":"2024-04-25T05:51:13.533319+0000","last_fullsized":"2024-04-25T05:51:13.533319+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:29.428764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:29.428764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1240,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535524+0000","last_change":"2024-04-25T05:50:07.985223+0000","last_active":"2024-04-25T05:51:13.535524+0000","last_peered":"2024-04-25T05:51:13.535524+0000","last_clean":"2024-04-25T05:51:13.535524+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:51:13.535524+0000","last_undegraded":"2024-04-25T05:51:13.535524+0000","last_fullsized":"2024-04-25T05:51:13.535524+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:50:07.985110+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.985110+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.172799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":453,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470611+0000","last_change":"2024-04-25T05:49:09.560837+0000","last_active":"2024-04-25T05:51:10.470611+0000","last_peered":"2024-04-25T05:51:10.470611+0000","last_clean":"2024-04-25T05:51:10.470611+0000","last_became_active":"2024-04-25T05:49:09.559684+0000","last_became_peered":"2024-04-25T05:49:09.559684+0000","last_unstale":"2024-04-25T05:51:10.470611+0000","last_undegraded":"2024-04-25T05:51:10.470611+0000","last_fullsized":"2024-04-25T05:51:10.470611+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:37.254738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:48:37.254738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":223,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489982+0000","last_change":"2024-04-25T05:50:04.735739+0000","last_active":"2024-04-25T05:51:10.489982+0000","last_peered":"2024-04-25T05:51:10.489982+0000","last_clean":"2024-04-25T05:51:10.489982+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:51:10.489982+0000","last_undegraded":"2024-04-25T05:51:10.489982+0000","last_fullsized":"2024-04-25T05:51:10.489982+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:50:04.735651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.735651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:21.992705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":867,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533539+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:51:13.533539+0000","last_peered":"2024-04-25T05:51:13.533539+0000","last_clean":"2024-04-25T05:51:13.533539+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:51:13.533539+0000","last_undegraded":"2024-04-25T05:51:13.533539+0000","last_fullsized":"2024-04-25T05:51:13.533539+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391237+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":911,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489490+0000","last_change":"2024-04-25T05:49:00.009223+0000","last_active":"2024-04-25T05:51:10.489490+0000","last_peered":"2024-04-25T05:51:10.489490+0000","last_clean":"2024-04-25T05:51:10.489490+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:51:10.489490+0000","last_undegraded":"2024-04-25T05:51:10.489490+0000","last_fullsized":"2024-04-25T05:51:10.489490+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:00.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:00.008984+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"481'5594","reported_seq":5653,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.196948+0000","last_change":"2024-04-25T05:50:12.043881+0000","last_active":"2024-04-25T05:51:13.196948+0000","last_peered":"2024-04-25T05:51:13.196948+0000","last_clean":"2024-04-25T05:51:13.196948+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:51:13.196948+0000","last_undegraded":"2024-04-25T05:51:13.196948+0000","last_fullsized":"2024-04-25T05:51:13.196948+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:18.117782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366608384,"num_objects":5594,"num_object_clones":0,"num_object_copies":16782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":358016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":605,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536114+0000","last_change":"2024-04-25T05:50:14.065922+0000","last_active":"2024-04-25T05:51:13.536114+0000","last_peered":"2024-04-25T05:51:13.536114+0000","last_clean":"2024-04-25T05:51:13.536114+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:51:13.536114+0000","last_undegraded":"2024-04-25T05:51:13.536114+0000","last_fullsized":"2024-04-25T05:51:13.536114+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:22.097234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490467+0000","last_change":"2024-04-25T05:48:20.054569+0000","last_active":"2024-04-25T05:51:10.490467+0000","last_peered":"2024-04-25T05:51:10.490467+0000","last_clean":"2024-04-25T05:51:10.490467+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:51:10.490467+0000","last_undegraded":"2024-04-25T05:51:10.490467+0000","last_fullsized":"2024-04-25T05:51:10.490467+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:20.054446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:48:20.054446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":236,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483151+0000","last_change":"2024-04-25T05:50:11.521033+0000","last_active":"2024-04-25T05:51:10.483151+0000","last_peered":"2024-04-25T05:51:10.483151+0000","last_clean":"2024-04-25T05:51:10.483151+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:51:10.483151+0000","last_undegraded":"2024-04-25T05:51:10.483151+0000","last_fullsized":"2024-04-25T05:51:10.483151+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.520907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.520907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.942600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":502,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484525+0000","last_change":"2024-04-25T05:47:19.913147+0000","last_active":"2024-04-25T05:51:10.484525+0000","last_peered":"2024-04-25T05:51:10.484525+0000","last_clean":"2024-04-25T05:51:10.484525+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:51:10.484525+0000","last_undegraded":"2024-04-25T05:51:10.484525+0000","last_fullsized":"2024-04-25T05:51:10.484525+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:19.913041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:19.913041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918614+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"481'5746","reported_seq":5805,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.226671+0000","last_change":"2024-04-25T05:50:12.042685+0000","last_active":"2024-04-25T05:51:13.226671+0000","last_peered":"2024-04-25T05:51:13.226671+0000","last_clean":"2024-04-25T05:51:13.226671+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:51:13.226671+0000","last_undegraded":"2024-04-25T05:51:13.226671+0000","last_fullsized":"2024-04-25T05:51:13.226671+0000","mapping_epoch":456,"log_start":"465'1300","ondisk_log_start":"465'1300","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:35.841340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":674,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539139+0000","last_change":"2024-04-25T05:50:15.942029+0000","last_active":"2024-04-25T05:51:13.539139+0000","last_peered":"2024-04-25T05:51:13.539139+0000","last_clean":"2024-04-25T05:51:13.539139+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:51:13.539139+0000","last_undegraded":"2024-04-25T05:51:13.539139+0000","last_fullsized":"2024-04-25T05:51:13.539139+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.941896+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.479186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":939,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484635+0000","last_change":"2024-04-25T05:50:47.124683+0000","last_active":"2024-04-25T05:51:10.484635+0000","last_peered":"2024-04-25T05:51:10.484635+0000","last_clean":"2024-04-25T05:51:10.484635+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:51:10.484635+0000","last_undegraded":"2024-04-25T05:51:10.484635+0000","last_fullsized":"2024-04-25T05:51:10.484635+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.124554+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.362413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":242,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483439+0000","last_change":"2024-04-25T05:50:05.416344+0000","last_active":"2024-04-25T05:51:10.483439+0000","last_peered":"2024-04-25T05:51:10.483439+0000","last_clean":"2024-04-25T05:51:10.483439+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:51:10.483439+0000","last_undegraded":"2024-04-25T05:51:10.483439+0000","last_fullsized":"2024-04-25T05:51:10.483439+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.416227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.416227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.653594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":884,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483912+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:51:10.483912+0000","last_peered":"2024-04-25T05:51:10.483912+0000","last_clean":"2024-04-25T05:51:10.483912+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:51:10.483912+0000","last_undegraded":"2024-04-25T05:51:10.483912+0000","last_fullsized":"2024-04-25T05:51:10.483912+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"481'5491","reported_seq":5551,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:12.650872+0000","last_change":"2024-04-25T05:50:12.035713+0000","last_active":"2024-04-25T05:51:12.650872+0000","last_peered":"2024-04-25T05:51:12.650872+0000","last_clean":"2024-04-25T05:51:12.650872+0000","last_became_active":"2024-04-25T05:50:12.035376+0000","last_became_peered":"2024-04-25T05:50:12.035376+0000","last_unstale":"2024-04-25T05:51:12.650872+0000","last_undegraded":"2024-04-25T05:51:12.650872+0000","last_fullsized":"2024-04-25T05:51:12.650872+0000","mapping_epoch":456,"log_start":"464'600","ondisk_log_start":"464'600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.653834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359858176,"num_objects":5491,"num_object_clones":0,"num_object_copies":16473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":250,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.483047+0000","last_change":"2024-04-25T05:50:39.566845+0000","last_active":"2024-04-25T05:51:10.483047+0000","last_peered":"2024-04-25T05:51:10.483047+0000","last_clean":"2024-04-25T05:51:10.483047+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:51:10.483047+0000","last_undegraded":"2024-04-25T05:51:10.483047+0000","last_fullsized":"2024-04-25T05:51:10.483047+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.566717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.566717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.356177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":300,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470962+0000","last_change":"2024-04-25T05:49:09.598004+0000","last_active":"2024-04-25T05:51:10.470962+0000","last_peered":"2024-04-25T05:51:10.470962+0000","last_clean":"2024-04-25T05:51:10.470962+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:51:10.470962+0000","last_undegraded":"2024-04-25T05:51:10.470962+0000","last_fullsized":"2024-04-25T05:51:10.470962+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:48:24.506969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.506969+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":892,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484817+0000","last_change":"2024-04-25T05:50:41.101003+0000","last_active":"2024-04-25T05:51:10.484817+0000","last_peered":"2024-04-25T05:51:10.484817+0000","last_clean":"2024-04-25T05:51:10.484817+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:51:10.484817+0000","last_undegraded":"2024-04-25T05:51:10.484817+0000","last_fullsized":"2024-04-25T05:51:10.484817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.100880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.100880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.503491+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":879,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.534069+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:51:13.534069+0000","last_peered":"2024-04-25T05:51:13.534069+0000","last_clean":"2024-04-25T05:51:13.534069+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:51:13.534069+0000","last_undegraded":"2024-04-25T05:51:13.534069+0000","last_fullsized":"2024-04-25T05:51:13.534069+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.671987+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"481'5554","reported_seq":5613,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.111693+0000","last_change":"2024-04-25T05:50:12.042051+0000","last_active":"2024-04-25T05:51:13.111693+0000","last_peered":"2024-04-25T05:51:13.111693+0000","last_clean":"2024-04-25T05:51:13.111693+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:51:13.111693+0000","last_undegraded":"2024-04-25T05:51:13.111693+0000","last_fullsized":"2024-04-25T05:51:13.111693+0000","mapping_epoch":456,"log_start":"464'700","ondisk_log_start":"464'700","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.775770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363986944,"num_objects":5554,"num_object_clones":0,"num_object_copies":16662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":355456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1126,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484159+0000","last_change":"2024-04-25T05:50:39.051287+0000","last_active":"2024-04-25T05:51:10.484159+0000","last_peered":"2024-04-25T05:51:10.484159+0000","last_clean":"2024-04-25T05:51:10.484159+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:51:10.484159+0000","last_undegraded":"2024-04-25T05:51:10.484159+0000","last_fullsized":"2024-04-25T05:51:10.484159+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:50:39.050944+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.050944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:07.882347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":239,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489487+0000","last_change":"2024-04-25T05:50:11.779506+0000","last_active":"2024-04-25T05:51:10.489487+0000","last_peered":"2024-04-25T05:51:10.489487+0000","last_clean":"2024-04-25T05:51:10.489487+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:51:10.489487+0000","last_undegraded":"2024-04-25T05:51:10.489487+0000","last_fullsized":"2024-04-25T05:51:10.489487+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:50:11.779355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.779355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.341739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":932,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484960+0000","last_change":"2024-04-25T05:48:59.765915+0000","last_active":"2024-04-25T05:51:10.484960+0000","last_peered":"2024-04-25T05:51:10.484960+0000","last_clean":"2024-04-25T05:51:10.484960+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:51:10.484960+0000","last_undegraded":"2024-04-25T05:51:10.484960+0000","last_fullsized":"2024-04-25T05:51:10.484960+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:48:59.765781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.765781+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918761+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":211,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490007+0000","last_change":"2024-04-25T05:49:01.017521+0000","last_active":"2024-04-25T05:51:10.490007+0000","last_peered":"2024-04-25T05:51:10.490007+0000","last_clean":"2024-04-25T05:51:10.490007+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:51:10.490007+0000","last_undegraded":"2024-04-25T05:51:10.490007+0000","last_fullsized":"2024-04-25T05:51:10.490007+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:01.017406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.017406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":905,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533324+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:51:13.533324+0000","last_peered":"2024-04-25T05:51:13.533324+0000","last_clean":"2024-04-25T05:51:13.533324+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:51:13.533324+0000","last_undegraded":"2024-04-25T05:51:13.533324+0000","last_fullsized":"2024-04-25T05:51:13.533324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":648,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.538297+0000","last_change":"2024-04-25T05:50:43.106046+0000","last_active":"2024-04-25T05:51:13.538297+0000","last_peered":"2024-04-25T05:51:13.538297+0000","last_clean":"2024-04-25T05:51:13.538297+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:51:13.538297+0000","last_undegraded":"2024-04-25T05:51:13.538297+0000","last_fullsized":"2024-04-25T05:51:13.538297+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.105913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.105913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.313735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491144+0000","last_change":"2024-04-25T05:50:43.938330+0000","last_active":"2024-04-25T05:51:10.491144+0000","last_peered":"2024-04-25T05:51:10.491144+0000","last_clean":"2024-04-25T05:51:10.491144+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:51:10.491144+0000","last_undegraded":"2024-04-25T05:51:10.491144+0000","last_fullsized":"2024-04-25T05:51:10.491144+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.938246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.938246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:52.848134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":892,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484307+0000","last_change":"2024-04-25T05:48:57.925411+0000","last_active":"2024-04-25T05:51:10.484307+0000","last_peered":"2024-04-25T05:51:10.484307+0000","last_clean":"2024-04-25T05:51:10.484307+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:51:10.484307+0000","last_undegraded":"2024-04-25T05:51:10.484307+0000","last_fullsized":"2024-04-25T05:51:10.484307+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:57.925301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.925301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918776+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":593,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533441+0000","last_change":"2024-04-25T05:49:32.097695+0000","last_active":"2024-04-25T05:51:13.533441+0000","last_peered":"2024-04-25T05:51:13.533441+0000","last_clean":"2024-04-25T05:51:13.533441+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:51:13.533441+0000","last_undegraded":"2024-04-25T05:51:13.533441+0000","last_fullsized":"2024-04-25T05:51:13.533441+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:49:32.097448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.097448+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391212+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":910,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484172+0000","last_change":"2024-04-25T05:50:53.041228+0000","last_active":"2024-04-25T05:51:10.484172+0000","last_peered":"2024-04-25T05:51:10.484172+0000","last_clean":"2024-04-25T05:51:10.484172+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:51:10.484172+0000","last_undegraded":"2024-04-25T05:51:10.484172+0000","last_fullsized":"2024-04-25T05:51:10.484172+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:53.041118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:50:53.041118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.078933+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":863,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533147+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:51:13.533147+0000","last_peered":"2024-04-25T05:51:13.533147+0000","last_clean":"2024-04-25T05:51:13.533147+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:51:13.533147+0000","last_undegraded":"2024-04-25T05:51:13.533147+0000","last_fullsized":"2024-04-25T05:51:13.533147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390905+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":236,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.486595+0000","last_change":"2024-04-25T05:50:13.581116+0000","last_active":"2024-04-25T05:51:10.486595+0000","last_peered":"2024-04-25T05:51:10.486595+0000","last_clean":"2024-04-25T05:51:10.486595+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:51:10.486595+0000","last_undegraded":"2024-04-25T05:51:10.486595+0000","last_fullsized":"2024-04-25T05:51:10.486595+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.581028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.581028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.001393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":222,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490971+0000","last_change":"2024-04-25T05:48:26.172633+0000","last_active":"2024-04-25T05:51:10.490971+0000","last_peered":"2024-04-25T05:51:10.490971+0000","last_clean":"2024-04-25T05:51:10.490971+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:51:10.490971+0000","last_undegraded":"2024-04-25T05:51:10.490971+0000","last_fullsized":"2024-04-25T05:51:10.490971+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:48:26.172484+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:48:26.172484+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":495,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484080+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:51:10.484080+0000","last_peered":"2024-04-25T05:51:10.484080+0000","last_clean":"2024-04-25T05:51:10.484080+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:51:10.484080+0000","last_undegraded":"2024-04-25T05:51:10.484080+0000","last_fullsized":"2024-04-25T05:51:10.484080+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918744+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1107,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.527659+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:51:13.527659+0000","last_peered":"2024-04-25T05:51:13.527659+0000","last_clean":"2024-04-25T05:51:13.527659+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:51:13.527659+0000","last_undegraded":"2024-04-25T05:51:13.527659+0000","last_fullsized":"2024-04-25T05:51:13.527659+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.466618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":431,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471656+0000","last_change":"2024-04-25T05:50:52.078015+0000","last_active":"2024-04-25T05:51:10.471656+0000","last_peered":"2024-04-25T05:51:10.471656+0000","last_clean":"2024-04-25T05:51:10.471656+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:51:10.471656+0000","last_undegraded":"2024-04-25T05:51:10.471656+0000","last_fullsized":"2024-04-25T05:51:10.471656+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.077914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.221398+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":929,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.527205+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:51:13.527205+0000","last_peered":"2024-04-25T05:51:13.527205+0000","last_clean":"2024-04-25T05:51:13.527205+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:51:13.527205+0000","last_undegraded":"2024-04-25T05:51:13.527205+0000","last_fullsized":"2024-04-25T05:51:13.527205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390997+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":896,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.485162+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:51:10.485162+0000","last_peered":"2024-04-25T05:51:10.485162+0000","last_clean":"2024-04-25T05:51:10.485162+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:51:10.485162+0000","last_undegraded":"2024-04-25T05:51:10.485162+0000","last_fullsized":"2024-04-25T05:51:10.485162+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918714+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":269,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491187+0000","last_change":"2024-04-25T05:49:01.967445+0000","last_active":"2024-04-25T05:51:10.491187+0000","last_peered":"2024-04-25T05:51:10.491187+0000","last_clean":"2024-04-25T05:51:10.491187+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:10.491187+0000","last_undegraded":"2024-04-25T05:51:10.491187+0000","last_fullsized":"2024-04-25T05:51:10.491187+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'13","last_scrub_stamp":"2024-04-25T05:49:01.967348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.967348+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":495,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484643+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:51:10.484643+0000","last_peered":"2024-04-25T05:51:10.484643+0000","last_clean":"2024-04-25T05:51:10.484643+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:51:10.484643+0000","last_undegraded":"2024-04-25T05:51:10.484643+0000","last_fullsized":"2024-04-25T05:51:10.484643+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918663+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1023,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470994+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:51:10.470994+0000","last_peered":"2024-04-25T05:51:10.470994+0000","last_clean":"2024-04-25T05:51:10.470994+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:51:10.470994+0000","last_undegraded":"2024-04-25T05:51:10.470994+0000","last_fullsized":"2024-04-25T05:51:10.470994+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":477,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470680+0000","last_change":"2024-04-25T05:49:35.656164+0000","last_active":"2024-04-25T05:51:10.470680+0000","last_peered":"2024-04-25T05:51:10.470680+0000","last_clean":"2024-04-25T05:51:10.470680+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:51:10.470680+0000","last_undegraded":"2024-04-25T05:51:10.470680+0000","last_fullsized":"2024-04-25T05:51:10.470680+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:49:35.656070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.656070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":893,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.532801+0000","last_change":"2024-04-25T05:50:13.065576+0000","last_active":"2024-04-25T05:51:13.532801+0000","last_peered":"2024-04-25T05:51:13.532801+0000","last_clean":"2024-04-25T05:51:13.532801+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:51:13.532801+0000","last_undegraded":"2024-04-25T05:51:13.532801+0000","last_fullsized":"2024-04-25T05:51:13.532801+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.065464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.065464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.492742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":911,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536373+0000","last_change":"2024-04-25T05:50:06.095785+0000","last_active":"2024-04-25T05:51:13.536373+0000","last_peered":"2024-04-25T05:51:13.536373+0000","last_clean":"2024-04-25T05:51:13.536373+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:51:13.536373+0000","last_undegraded":"2024-04-25T05:51:13.536373+0000","last_fullsized":"2024-04-25T05:51:13.536373+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.604835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":440,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533389+0000","last_change":"2024-04-25T05:48:57.339166+0000","last_active":"2024-04-25T05:51:13.533389+0000","last_peered":"2024-04-25T05:51:13.533389+0000","last_clean":"2024-04-25T05:51:13.533389+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:51:13.533389+0000","last_undegraded":"2024-04-25T05:51:13.533389+0000","last_fullsized":"2024-04-25T05:51:13.533389+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:48:57.338784+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.338784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":892,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484359+0000","last_change":"2024-04-25T05:50:45.109485+0000","last_active":"2024-04-25T05:51:10.484359+0000","last_peered":"2024-04-25T05:51:10.484359+0000","last_clean":"2024-04-25T05:51:10.484359+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:51:10.484359+0000","last_undegraded":"2024-04-25T05:51:10.484359+0000","last_fullsized":"2024-04-25T05:51:10.484359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.109246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.109246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.734569+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":895,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533892+0000","last_change":"2024-04-25T05:50:49.427823+0000","last_active":"2024-04-25T05:51:13.533892+0000","last_peered":"2024-04-25T05:51:13.533892+0000","last_clean":"2024-04-25T05:51:13.533892+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:51:13.533892+0000","last_undegraded":"2024-04-25T05:51:13.533892+0000","last_fullsized":"2024-04-25T05:51:13.533892+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.080828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489380+0000","last_change":"2024-04-25T05:50:02.734274+0000","last_active":"2024-04-25T05:51:10.489380+0000","last_peered":"2024-04-25T05:51:10.489380+0000","last_clean":"2024-04-25T05:51:10.489380+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:51:10.489380+0000","last_undegraded":"2024-04-25T05:51:10.489380+0000","last_fullsized":"2024-04-25T05:51:10.489380+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:02.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:02.734157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:12.617322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":972,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536580+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:51:13.536580+0000","last_peered":"2024-04-25T05:51:13.536580+0000","last_clean":"2024-04-25T05:51:13.536580+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:51:13.536580+0000","last_undegraded":"2024-04-25T05:51:13.536580+0000","last_fullsized":"2024-04-25T05:51:13.536580+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:20.599614+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490983+0000","last_change":"2024-04-25T05:50:05.705768+0000","last_active":"2024-04-25T05:51:10.490983+0000","last_peered":"2024-04-25T05:51:10.490983+0000","last_clean":"2024-04-25T05:51:10.490983+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:51:10.490983+0000","last_undegraded":"2024-04-25T05:51:10.490983+0000","last_fullsized":"2024-04-25T05:51:10.490983+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.705647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.705647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.259931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":894,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484822+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:51:10.484822+0000","last_peered":"2024-04-25T05:51:10.484822+0000","last_clean":"2024-04-25T05:51:10.484822+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:51:10.484822+0000","last_undegraded":"2024-04-25T05:51:10.484822+0000","last_fullsized":"2024-04-25T05:51:10.484822+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918647+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":903,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.534158+0000","last_change":"2024-04-25T05:49:41.202012+0000","last_active":"2024-04-25T05:51:13.534158+0000","last_peered":"2024-04-25T05:51:13.534158+0000","last_clean":"2024-04-25T05:51:13.534158+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:51:13.534158+0000","last_undegraded":"2024-04-25T05:51:13.534158+0000","last_fullsized":"2024-04-25T05:51:13.534158+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391263+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":911,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536485+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:51:13.536485+0000","last_peered":"2024-04-25T05:51:13.536485+0000","last_clean":"2024-04-25T05:51:13.536485+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:51:13.536485+0000","last_undegraded":"2024-04-25T05:51:13.536485+0000","last_fullsized":"2024-04-25T05:51:13.536485+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391091+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.492152+0000","last_change":"2024-04-25T05:50:07.763957+0000","last_active":"2024-04-25T05:51:10.492152+0000","last_peered":"2024-04-25T05:51:10.492152+0000","last_clean":"2024-04-25T05:51:10.492152+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:51:10.492152+0000","last_undegraded":"2024-04-25T05:51:10.492152+0000","last_fullsized":"2024-04-25T05:51:10.492152+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:07.763810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.763810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.405363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":610,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.538535+0000","last_change":"2024-04-25T05:50:14.959177+0000","last_active":"2024-04-25T05:51:13.538535+0000","last_peered":"2024-04-25T05:51:13.538535+0000","last_clean":"2024-04-25T05:51:13.538535+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:51:13.538535+0000","last_undegraded":"2024-04-25T05:51:13.538535+0000","last_fullsized":"2024-04-25T05:51:13.538535+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.959063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.959063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:44.398322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":893,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484950+0000","last_change":"2024-04-25T05:49:31.857568+0000","last_active":"2024-04-25T05:51:10.484950+0000","last_peered":"2024-04-25T05:51:10.484950+0000","last_clean":"2024-04-25T05:51:10.484950+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:51:10.484950+0000","last_undegraded":"2024-04-25T05:51:10.484950+0000","last_fullsized":"2024-04-25T05:51:10.484950+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:49:31.857452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:31.857452+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918840+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":730,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.539916+0000","last_change":"2024-04-25T05:50:47.087717+0000","last_active":"2024-04-25T05:51:13.539916+0000","last_peered":"2024-04-25T05:51:13.539916+0000","last_clean":"2024-04-25T05:51:13.539916+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:51:13.539916+0000","last_undegraded":"2024-04-25T05:51:13.539916+0000","last_fullsized":"2024-04-25T05:51:13.539916+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:50:47.087575+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.087575+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.480370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":903,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536259+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:51:13.536259+0000","last_peered":"2024-04-25T05:51:13.536259+0000","last_clean":"2024-04-25T05:51:13.536259+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:51:13.536259+0000","last_undegraded":"2024-04-25T05:51:13.536259+0000","last_fullsized":"2024-04-25T05:51:13.536259+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":502,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484865+0000","last_change":"2024-04-25T05:49:30.823791+0000","last_active":"2024-04-25T05:51:10.484865+0000","last_peered":"2024-04-25T05:51:10.484865+0000","last_clean":"2024-04-25T05:51:10.484865+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:51:10.484865+0000","last_undegraded":"2024-04-25T05:51:10.484865+0000","last_fullsized":"2024-04-25T05:51:10.484865+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:30.823661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:30.823661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918823+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":891,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.532993+0000","last_change":"2024-04-25T05:50:17.140640+0000","last_active":"2024-04-25T05:51:13.532993+0000","last_peered":"2024-04-25T05:51:13.532993+0000","last_clean":"2024-04-25T05:51:13.532993+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:51:13.532993+0000","last_undegraded":"2024-04-25T05:51:13.532993+0000","last_fullsized":"2024-04-25T05:51:13.532993+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:50:17.140516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.143695+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":577,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484964+0000","last_change":"2024-04-25T05:50:48.098281+0000","last_active":"2024-04-25T05:51:10.484964+0000","last_peered":"2024-04-25T05:51:10.484964+0000","last_clean":"2024-04-25T05:51:10.484964+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:51:10.484964+0000","last_undegraded":"2024-04-25T05:51:10.484964+0000","last_fullsized":"2024-04-25T05:51:10.484964+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.351413+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":895,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.534075+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:51:13.534075+0000","last_peered":"2024-04-25T05:51:13.534075+0000","last_clean":"2024-04-25T05:51:13.534075+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:51:13.534075+0000","last_undegraded":"2024-04-25T05:51:13.534075+0000","last_fullsized":"2024-04-25T05:51:13.534075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.209136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":762,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.538909+0000","last_change":"2024-04-25T05:50:41.100007+0000","last_active":"2024-04-25T05:51:13.538909+0000","last_peered":"2024-04-25T05:51:13.538909+0000","last_clean":"2024-04-25T05:51:13.538909+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:51:13.538909+0000","last_undegraded":"2024-04-25T05:51:13.538909+0000","last_fullsized":"2024-04-25T05:51:13.538909+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:50:41.099870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.099870+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.548999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":920,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.485082+0000","last_change":"2024-04-25T05:50:04.837659+0000","last_active":"2024-04-25T05:51:10.485082+0000","last_peered":"2024-04-25T05:51:10.485082+0000","last_clean":"2024-04-25T05:51:10.485082+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:51:10.485082+0000","last_undegraded":"2024-04-25T05:51:10.485082+0000","last_fullsized":"2024-04-25T05:51:10.485082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:50:04.837536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.837536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.301195+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":910,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.527919+0000","last_change":"2024-04-25T05:47:16.525921+0000","last_active":"2024-04-25T05:51:13.527919+0000","last_peered":"2024-04-25T05:51:13.527919+0000","last_clean":"2024-04-25T05:51:13.527919+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:51:13.527919+0000","last_undegraded":"2024-04-25T05:51:13.527919+0000","last_fullsized":"2024-04-25T05:51:13.527919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:16.525560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:16.525560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390778+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":918,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470921+0000","last_change":"2024-04-25T05:48:00.830523+0000","last_active":"2024-04-25T05:51:10.470921+0000","last_peered":"2024-04-25T05:51:10.470921+0000","last_clean":"2024-04-25T05:51:10.470921+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:51:10.470921+0000","last_undegraded":"2024-04-25T05:51:10.470921+0000","last_fullsized":"2024-04-25T05:51:10.470921+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.830408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":449,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470920+0000","last_change":"2024-04-25T05:50:09.708374+0000","last_active":"2024-04-25T05:51:10.470920+0000","last_peered":"2024-04-25T05:51:10.470920+0000","last_clean":"2024-04-25T05:51:10.470920+0000","last_became_active":"2024-04-25T05:49:09.550767+0000","last_became_peered":"2024-04-25T05:49:09.550767+0000","last_unstale":"2024-04-25T05:51:10.470920+0000","last_undegraded":"2024-04-25T05:51:10.470920+0000","last_fullsized":"2024-04-25T05:51:10.470920+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.708291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.708291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":892,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533232+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:51:13.533232+0000","last_peered":"2024-04-25T05:51:13.533232+0000","last_clean":"2024-04-25T05:51:13.533232+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:51:13.533232+0000","last_undegraded":"2024-04-25T05:51:13.533232+0000","last_fullsized":"2024-04-25T05:51:13.533232+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.657262+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":259,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.486460+0000","last_change":"2024-04-25T05:50:44.583319+0000","last_active":"2024-04-25T05:51:10.486460+0000","last_peered":"2024-04-25T05:51:10.486460+0000","last_clean":"2024-04-25T05:51:10.486460+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:51:10.486460+0000","last_undegraded":"2024-04-25T05:51:10.486460+0000","last_fullsized":"2024-04-25T05:51:10.486460+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:44.583030+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:50:44.583030+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:45.044248+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":241,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470753+0000","last_change":"2024-04-25T05:49:09.557107+0000","last_active":"2024-04-25T05:51:10.470753+0000","last_peered":"2024-04-25T05:51:10.470753+0000","last_clean":"2024-04-25T05:51:10.470753+0000","last_became_active":"2024-04-25T05:49:09.556671+0000","last_became_peered":"2024-04-25T05:49:09.556671+0000","last_unstale":"2024-04-25T05:51:10.470753+0000","last_undegraded":"2024-04-25T05:51:10.470753+0000","last_fullsized":"2024-04-25T05:51:10.470753+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":901,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533222+0000","last_change":"2024-04-25T05:48:22.532473+0000","last_active":"2024-04-25T05:51:13.533222+0000","last_peered":"2024-04-25T05:51:13.533222+0000","last_clean":"2024-04-25T05:51:13.533222+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:51:13.533222+0000","last_undegraded":"2024-04-25T05:51:13.533222+0000","last_fullsized":"2024-04-25T05:51:13.533222+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:22.532394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:22.532394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":914,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.471810+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:51:10.471810+0000","last_peered":"2024-04-25T05:51:10.471810+0000","last_clean":"2024-04-25T05:51:10.471810+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:51:10.471810+0000","last_undegraded":"2024-04-25T05:51:10.471810+0000","last_fullsized":"2024-04-25T05:51:10.471810+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":392,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470842+0000","last_change":"2024-04-25T05:49:09.554222+0000","last_active":"2024-04-25T05:51:10.470842+0000","last_peered":"2024-04-25T05:51:10.470842+0000","last_clean":"2024-04-25T05:51:10.470842+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:51:10.470842+0000","last_undegraded":"2024-04-25T05:51:10.470842+0000","last_fullsized":"2024-04-25T05:51:10.470842+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:48:36.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:36.108290+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490557+0000","last_change":"2024-04-25T05:50:50.860753+0000","last_active":"2024-04-25T05:51:10.490557+0000","last_peered":"2024-04-25T05:51:10.490557+0000","last_clean":"2024-04-25T05:51:10.490557+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:51:10.490557+0000","last_undegraded":"2024-04-25T05:51:10.490557+0000","last_fullsized":"2024-04-25T05:51:10.490557+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_clean_scrub_stamp":"2024-04-25T05:50:50.860626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:18.625619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":881,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.484251+0000","last_change":"2024-04-25T05:49:35.840397+0000","last_active":"2024-04-25T05:51:10.484251+0000","last_peered":"2024-04-25T05:51:10.484251+0000","last_clean":"2024-04-25T05:51:10.484251+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:51:10.484251+0000","last_undegraded":"2024-04-25T05:51:10.484251+0000","last_fullsized":"2024-04-25T05:51:10.484251+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.840316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.840316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:14.918808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.489594+0000","last_change":"2024-04-25T05:48:31.213742+0000","last_active":"2024-04-25T05:51:10.489594+0000","last_peered":"2024-04-25T05:51:10.489594+0000","last_clean":"2024-04-25T05:51:10.489594+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:51:10.489594+0000","last_undegraded":"2024-04-25T05:51:10.489594+0000","last_fullsized":"2024-04-25T05:51:10.489594+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:31.213611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:31.213611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":686,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470822+0000","last_change":"2024-04-25T05:49:09.555965+0000","last_active":"2024-04-25T05:51:10.470822+0000","last_peered":"2024-04-25T05:51:10.470822+0000","last_clean":"2024-04-25T05:51:10.470822+0000","last_became_active":"2024-04-25T05:49:09.554840+0000","last_became_peered":"2024-04-25T05:49:09.554840+0000","last_unstale":"2024-04-25T05:51:10.470822+0000","last_undegraded":"2024-04-25T05:51:10.470822+0000","last_fullsized":"2024-04-25T05:51:10.470822+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.169828+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":218,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491089+0000","last_change":"2024-04-25T05:50:45.906872+0000","last_active":"2024-04-25T05:51:10.491089+0000","last_peered":"2024-04-25T05:51:10.491089+0000","last_clean":"2024-04-25T05:51:10.491089+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:51:10.491089+0000","last_undegraded":"2024-04-25T05:51:10.491089+0000","last_fullsized":"2024-04-25T05:51:10.491089+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.906734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.906734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.651537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":210,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.490130+0000","last_change":"2024-04-25T05:48:24.173416+0000","last_active":"2024-04-25T05:51:10.490130+0000","last_peered":"2024-04-25T05:51:10.490130+0000","last_clean":"2024-04-25T05:51:10.490130+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:51:10.490130+0000","last_undegraded":"2024-04-25T05:51:10.490130+0000","last_fullsized":"2024-04-25T05:51:10.490130+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:24.173135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.173135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":511,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.535698+0000","last_change":"2024-04-25T05:50:08.944539+0000","last_active":"2024-04-25T05:51:13.535698+0000","last_peered":"2024-04-25T05:51:13.535698+0000","last_clean":"2024-04-25T05:51:13.535698+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:51:13.535698+0000","last_undegraded":"2024-04-25T05:51:13.535698+0000","last_fullsized":"2024-04-25T05:51:13.535698+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.210582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1176,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.536647+0000","last_change":"2024-04-25T05:50:40.381895+0000","last_active":"2024-04-25T05:51:13.536647+0000","last_peered":"2024-04-25T05:51:13.536647+0000","last_clean":"2024-04-25T05:51:13.536647+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:51:13.536647+0000","last_undegraded":"2024-04-25T05:51:13.536647+0000","last_fullsized":"2024-04-25T05:51:13.536647+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:02.543415+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":912,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.528700+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:51:13.528700+0000","last_peered":"2024-04-25T05:51:13.528700+0000","last_clean":"2024-04-25T05:51:13.528700+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:51:13.528700+0000","last_undegraded":"2024-04-25T05:51:13.528700+0000","last_fullsized":"2024-04-25T05:51:13.528700+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":447,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470320+0000","last_change":"2024-04-25T05:50:40.775527+0000","last_active":"2024-04-25T05:51:10.470320+0000","last_peered":"2024-04-25T05:51:10.470320+0000","last_clean":"2024-04-25T05:51:10.470320+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:51:10.470320+0000","last_undegraded":"2024-04-25T05:51:10.470320+0000","last_fullsized":"2024-04-25T05:51:10.470320+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:40.775406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.775406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:57.019250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":878,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.533980+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:51:13.533980+0000","last_peered":"2024-04-25T05:51:13.533980+0000","last_clean":"2024-04-25T05:51:13.533980+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:51:13.533980+0000","last_undegraded":"2024-04-25T05:51:13.533980+0000","last_fullsized":"2024-04-25T05:51:13.533980+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.291166+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":983,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470130+0000","last_change":"2024-04-25T05:50:43.844399+0000","last_active":"2024-04-25T05:51:10.470130+0000","last_peered":"2024-04-25T05:51:10.470130+0000","last_clean":"2024-04-25T05:51:10.470130+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:51:10.470130+0000","last_undegraded":"2024-04-25T05:51:10.470130+0000","last_fullsized":"2024-04-25T05:51:10.470130+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:43.844256+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.844256+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.303125+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":219,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.491583+0000","last_change":"2024-04-25T05:49:35.890470+0000","last_active":"2024-04-25T05:51:10.491583+0000","last_peered":"2024-04-25T05:51:10.491583+0000","last_clean":"2024-04-25T05:51:10.491583+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:51:10.491583+0000","last_undegraded":"2024-04-25T05:51:10.491583+0000","last_fullsized":"2024-04-25T05:51:10.491583+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.890392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.890392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":904,"reported_epoch":482,"state":"active+clean","last_fresh":"2024-04-25T05:51:13.534169+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:51:13.534169+0000","last_peered":"2024-04-25T05:51:13.534169+0000","last_clean":"2024-04-25T05:51:13.534169+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:13.534169+0000","last_undegraded":"2024-04-25T05:51:13.534169+0000","last_fullsized":"2024-04-25T05:51:13.534169+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:16.929190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":423,"reported_epoch":481,"state":"active+clean","last_fresh":"2024-04-25T05:51:10.470581+0000","last_change":"2024-04-25T05:49:09.559432+0000","last_active":"2024-04-25T05:51:10.470581+0000","last_peered":"2024-04-25T05:51:10.470581+0000","last_clean":"2024-04-25T05:51:10.470581+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:51:10.470581+0000","last_undegraded":"2024-04-25T05:51:10.470581+0000","last_fullsized":"2024-04-25T05:51:10.470581+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.227001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.227001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5923471360,"num_objects":90385,"num_object_clones":0,"num_object_copies":271155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90385,"num_write_kb":5784640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17775591424,"data_stored":17775591424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71785,"ondisk_log_size":71785,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1016,"num_read_kb":1011,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":10594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070676,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3387888,"kb_used_data":2518184,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90357264,"statfs":{"total":95995035648,"available":92525838336,"internally_reserved":0,"allocated":2578620416,"data_stored":2578023407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365016,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3631620,"kb_used_data":2888252,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":90113532,"statfs":{"total":95995035648,"available":92276256768,"internally_reserved":0,"allocated":2957570048,"data_stored":2957016511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":761199046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4249356,"kb_used_data":3279680,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":89495796,"statfs":{"total":95995035648,"available":91643695104,"internally_reserved":0,"allocated":3358392320,"data_stored":3357776558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738615,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3151648,"kb_used_data":2502676,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":90593504,"statfs":{"total":95995035648,"available":92767748096,"internally_reserved":0,"allocated":2562740224,"data_stored":2562162157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149926,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3651244,"kb_used_data":2533028,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":90093908,"statfs":{"total":95995035648,"available":92256161792,"internally_reserved":0,"allocated":2593820672,"data_stored":2593226680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1145043399},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986401,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2848548,"kb_used_data":2141272,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":90896604,"statfs":{"total":95995035648,"available":93078122496,"internally_reserved":0,"allocated":2192662528,"data_stored":2192080501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692061,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031952,"kb_used_data":1437128,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91713200,"statfs":{"total":95995035648,"available":93914316800,"internally_reserved":0,"allocated":1471619072,"data_stored":1471053377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3361603584,"data_stored":3361603584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2570256384,"data_stored":2570256384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2618884096,"data_stored":2618884096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2201092096,"data_stored":2201092096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1473445888,"data_stored":1473445888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2964127744,"data_stored":2964127744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2586181632,"data_stored":2586181632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:51:19.052 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:51:19.053 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:51:19.053 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-25T05:51:19.053 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:51:19.101 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:19.135 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:19.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 59 16 97601 97585 103.325 84.5 0.0163634 0.00967367 2024-04-25T05:51:19.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.280 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:19.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:19.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:19.414 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:19.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:19.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:19.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:19.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:19.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:19.823 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:19.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:19.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[27010]: 4.1d scrub starts 2024-04-25T05:51:19.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[27010]: 4.1d scrub ok 2024-04-25T05:51:19.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:51:19.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:51:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[37492]: 4.1d scrub starts 2024-04-25T05:51:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[37492]: 4.1d scrub ok 2024-04-25T05:51:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:51:19.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:19 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:51:19.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:19.957 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:51:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:19.961 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:19.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:19 smithi060 ceph-mon[30898]: 4.1d scrub starts 2024-04-25T05:51:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:19 smithi060 ceph-mon[30898]: 4.1d scrub ok 2024-04-25T05:51:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:19 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:51:19.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:19 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:51:20.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.096 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:20.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:20.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:20.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:51:20.222212+0000 min lat: 0.00134748 max lat: 0.2399 avg lat: 0.00970076 2024-04-25T05:51:20.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:51:20.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 60 16 98949 98933 103.007 84.25 0.0401728 0.00970076 2024-04-25T05:51:20.222 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:20.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:20.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:20.427 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:20.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:20.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:20.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:20.634 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:20.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:20.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:20.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:20.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:20 smithi043 ceph-mon[27010]: pgmap v913: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 786 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:51:20.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:20 smithi043 ceph-mon[37492]: pgmap v913: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 786 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:51:20.910 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:20.911 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:20.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:20.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:20 smithi060 ceph-mon[30898]: pgmap v913: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 786 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-04-25T05:51:21.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:21.046 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:21.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:21.182 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:21.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 61 16 100133 100117 102.531 74 0.00634327 0.00974822 2024-04-25T05:51:21.283 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.344 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:21.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:21.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:21.486 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:21.588 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:21.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:21.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:21.737 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:21 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:51:21.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:21 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:51:21.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:21 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:51:21.738 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:21 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:51:21.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:21.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:21.902 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:21.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:21.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:21 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:51:21.932 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:21 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:51:22.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:22.053 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:22.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:22.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 62 16 101393 101377 102.148 78.75 0.00611942 0.00978486 2024-04-25T05:51:22.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:22.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:22.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:22.465 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:22.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:22.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:22.612 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:51:22.627 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:51:22.641 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:51:22.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.750 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:22.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:22.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: pgmap v914: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 638 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:51:22.842 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1922468080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: pgmap v914: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 638 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1922468080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:22 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:22.877 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:22.877 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: pgmap v914: 158 pgs: 158 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 638 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1922468080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.878 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:22 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:22.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:22.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:22.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:23.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:23.133 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:23.153 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:23.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 63 16 102388 102372 101.514 62.1875 0.00748347 0.00984351 2024-04-25T05:51:23.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:23.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:23.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:23.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:23.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:23.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:23.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:23.701 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:23.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:23.838 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:23.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: osdmap e485: 8 total, 8 up, 7 in 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: from='client.16509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:51:23.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: osdmap e485: 8 total, 8 up, 7 in 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: from='client.16509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:51:23.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:23 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:51:23.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:23.961 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:23.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:23.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:51:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:51:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:23.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: osdmap e485: 8 total, 8 up, 7 in 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: from='client.16509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:51:24.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:23 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:51:24.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.100 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:24.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:24.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:24.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 64 16 103421 103405 100.936 64.5625 0.0372088 0.00990182 2024-04-25T05:51:24.235 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:24.336 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:24.372 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:24.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:24.510 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:24.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:24.647 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:24.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:24.786 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:24.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 11.3 scrub starts 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 4.13 deep-scrub starts 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 4.13 deep-scrub ok 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:51:24.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[27010]: pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 11.3 scrub starts 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 4.13 deep-scrub starts 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 4.13 deep-scrub ok 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:51:24.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:24 smithi043 ceph-mon[37492]: pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T05:51:24.925 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:24.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 11.3 scrub starts 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 4.13 deep-scrub starts 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 4.13 deep-scrub ok 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:51:24.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:51:25.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:51:25.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:51:25.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:51:25.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:24 smithi060 ceph-mon[30898]: pgmap v916: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-25T05:51:25.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.061 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:25.161 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:25.201 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:25.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 65 16 104684 104668 100.598 78.9375 0.00512008 0.00993531 2024-04-25T05:51:25.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:25.323 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:25.425 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:25.464 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:25.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:25.597 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:25.699 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:25.735 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:25.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:25.869 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:25.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:25.971 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:26.005 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:26.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:26.146 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:26.180 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:51:26.222 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 66 16 105923 105907 100.247 77.4375 0.0118653 0.00997083 2024-04-25T05:51:26.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.301 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:26.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:51:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:51:26.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[27010]: osdmap e486: 8 total, 8 up, 7 in 2024-04-25T05:51:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:51:26.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:51:26.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:26 smithi043 ceph-mon[37492]: osdmap e486: 8 total, 8 up, 7 in 2024-04-25T05:51:26.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:26.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:26.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:26 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:51:26.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:26 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:51:26.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:26 smithi060 ceph-mon[30898]: osdmap e486: 8 total, 8 up, 7 in 2024-04-25T05:51:26.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:26.591 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:26.672 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:26.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:26.725 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:26.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:26.863 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:26.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:26.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:27.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:27.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:27.144 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:27.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 67 16 107115 107099 99.8629 74.5 0.00355116 0.0100063 2024-04-25T05:51:27.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:27.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:27 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:51:27.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:27 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:51:27.372 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:27 smithi060 ceph-mon[30898]: 11.3 scrub ok 2024-04-25T05:51:27.372 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:27 smithi060 ceph-mon[30898]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.39k op/s 2024-04-25T05:51:27.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:51:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:51:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[27010]: 11.3 scrub ok 2024-04-25T05:51:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[27010]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.39k op/s 2024-04-25T05:51:27.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:51:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:51:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[37492]: 11.3 scrub ok 2024-04-25T05:51:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:27 smithi043 ceph-mon[37492]: pgmap v918: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.39k op/s 2024-04-25T05:51:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:27.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:27.481 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:27.481 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:27.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.566 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:27.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:27.667 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:27.701 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:27.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:27.837 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:27.939 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:27.966 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:27.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:28.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:28.101 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:28.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:28 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:51:28.202 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:28 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:51:28.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:28.221 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 68 16 108316 108300 99.4982 75.0625 0.0133672 0.0100462 2024-04-25T05:51:28.247 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:28.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:51:28.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:28.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:28 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:51:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:28 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:51:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:28 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:51:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:28 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:51:28.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:28.519 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:28.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:28.675 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:28.771 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:28.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.826 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:28.927 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:28.970 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:28.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:28.988 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":915,"stamp":"2024-04-25T05:51:22.589118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6335109659,"num_objects":96887,"num_object_clones":0,"num_object_copies":290661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96887,"num_whiteouts":0,"num_read":2434,"num_read_kb":2154,"num_write":97571,"num_write_kb":6186093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":626,"num_bytes_recovered":1198290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74270,"ondisk_log_size":74270,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24160740,"kb_used_data":18489260,"kb_used_omap":10,"kb_used_meta":5671413,"kb_avail":632055324,"statfs":{"total":671965249536,"available":647224651776,"internally_reserved":0,"allocated":18933002240,"data_stored":18928813271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":5807527019},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,6,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":411041792,"num_objects":6272,"num_object_clones":0,"num_object_copies":18816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6272,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.060552"},"pg_stats":[{"pgid":"11.3","version":"484'6017","reported_seq":6082,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.544414+0000","last_change":"2024-04-25T05:50:12.040793+0000","last_active":"2024-04-25T05:51:18.544414+0000","last_peered":"2024-04-25T05:51:18.544414+0000","last_clean":"2024-04-25T05:51:18.544414+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:51:18.544414+0000","last_undegraded":"2024-04-25T05:51:18.544414+0000","last_fullsized":"2024-04-25T05:51:18.544414+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394330112,"num_objects":6017,"num_object_clones":0,"num_object_copies":18051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":385088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":383,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064496+0000","last_change":"2024-04-25T05:50:08.705811+0000","last_active":"2024-04-25T05:51:15.064496+0000","last_peered":"2024-04-25T05:51:15.064496+0000","last_clean":"2024-04-25T05:51:15.064496+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:51:15.064496+0000","last_undegraded":"2024-04-25T05:51:15.064496+0000","last_fullsized":"2024-04-25T05:51:15.064496+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:50:08.705677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.705677+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.222090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":901,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060805+0000","last_change":"2024-04-25T05:49:05.117179+0000","last_active":"2024-04-25T05:51:15.060805+0000","last_peered":"2024-04-25T05:51:15.060805+0000","last_clean":"2024-04-25T05:51:15.060805+0000","last_became_active":"2024-04-25T05:44:28.171662+0000","last_became_peered":"2024-04-25T05:44:28.171662+0000","last_unstale":"2024-04-25T05:51:15.060805+0000","last_undegraded":"2024-04-25T05:51:15.060805+0000","last_fullsized":"2024-04-25T05:51:15.060805+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391123+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":315,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061348+0000","last_change":"2024-04-25T05:50:12.546520+0000","last_active":"2024-04-25T05:51:15.061348+0000","last_peered":"2024-04-25T05:51:15.061348+0000","last_clean":"2024-04-25T05:51:15.061348+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:51:15.061348+0000","last_undegraded":"2024-04-25T05:51:15.061348+0000","last_fullsized":"2024-04-25T05:51:15.061348+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:50:12.546342+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.546342+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.973854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":460,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064465+0000","last_change":"2024-04-25T05:50:13.709776+0000","last_active":"2024-04-25T05:51:15.064465+0000","last_peered":"2024-04-25T05:51:15.064465+0000","last_clean":"2024-04-25T05:51:15.064465+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:51:15.064465+0000","last_undegraded":"2024-04-25T05:51:15.064465+0000","last_fullsized":"2024-04-25T05:51:15.064465+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.709675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.709675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.101183+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1215,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054755+0000","last_change":"2024-04-25T05:49:33.878907+0000","last_active":"2024-04-25T05:51:15.054755+0000","last_peered":"2024-04-25T05:51:15.054755+0000","last_clean":"2024-04-25T05:51:15.054755+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:51:15.054755+0000","last_undegraded":"2024-04-25T05:51:15.054755+0000","last_fullsized":"2024-04-25T05:51:15.054755+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:33.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:33.878785+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"484'6044","reported_seq":6110,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:17.716787+0000","last_change":"2024-04-25T05:50:12.040238+0000","last_active":"2024-04-25T05:51:17.716787+0000","last_peered":"2024-04-25T05:51:17.716787+0000","last_clean":"2024-04-25T05:51:17.716787+0000","last_became_active":"2024-04-25T05:50:12.039934+0000","last_became_peered":"2024-04-25T05:50:12.039934+0000","last_unstale":"2024-04-25T05:51:17.716787+0000","last_undegraded":"2024-04-25T05:51:17.716787+0000","last_fullsized":"2024-04-25T05:51:17.716787+0000","mapping_epoch":456,"log_start":"465'1200","ondisk_log_start":"465'1200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.997447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396099584,"num_objects":6044,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":386816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":384,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064924+0000","last_change":"2024-04-25T05:49:09.556929+0000","last_active":"2024-04-25T05:51:15.064924+0000","last_peered":"2024-04-25T05:51:15.064924+0000","last_clean":"2024-04-25T05:51:15.064924+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:51:15.064924+0000","last_undegraded":"2024-04-25T05:51:15.064924+0000","last_fullsized":"2024-04-25T05:51:15.064924+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:48:59.073429+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.073429+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811525+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":614,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059278+0000","last_change":"2024-04-25T05:49:32.886598+0000","last_active":"2024-04-25T05:51:15.059278+0000","last_peered":"2024-04-25T05:51:15.059278+0000","last_clean":"2024-04-25T05:51:15.059278+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:51:15.059278+0000","last_undegraded":"2024-04-25T05:51:15.059278+0000","last_fullsized":"2024-04-25T05:51:15.059278+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.886504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.886504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":914,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.067819+0000","last_change":"2024-04-25T05:50:55.073084+0000","last_active":"2024-04-25T05:51:15.067819+0000","last_peered":"2024-04-25T05:51:15.067819+0000","last_clean":"2024-04-25T05:51:15.067819+0000","last_became_active":"2024-04-25T05:44:28.167740+0000","last_became_peered":"2024-04-25T05:44:28.167740+0000","last_unstale":"2024-04-25T05:51:15.067819+0000","last_undegraded":"2024-04-25T05:51:15.067819+0000","last_fullsized":"2024-04-25T05:51:15.067819+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:15.963221+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":250,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059107+0000","last_change":"2024-04-25T05:50:46.862960+0000","last_active":"2024-04-25T05:51:15.059107+0000","last_peered":"2024-04-25T05:51:15.059107+0000","last_clean":"2024-04-25T05:51:15.059107+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:51:15.059107+0000","last_undegraded":"2024-04-25T05:51:15.059107+0000","last_fullsized":"2024-04-25T05:51:15.059107+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:46.862820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.862820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:55.777488+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":814,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058195+0000","last_change":"2024-04-25T05:50:06.715616+0000","last_active":"2024-04-25T05:51:15.058195+0000","last_peered":"2024-04-25T05:51:15.058195+0000","last_clean":"2024-04-25T05:51:15.058195+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:51:15.058195+0000","last_undegraded":"2024-04-25T05:51:15.058195+0000","last_fullsized":"2024-04-25T05:51:15.058195+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:50:06.715476+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.715476+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:29.983548+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"484'6024","reported_seq":6087,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.526483+0000","last_change":"2024-04-25T05:50:12.035436+0000","last_active":"2024-04-25T05:51:18.526483+0000","last_peered":"2024-04-25T05:51:18.526483+0000","last_clean":"2024-04-25T05:51:18.526483+0000","last_became_active":"2024-04-25T05:50:12.035111+0000","last_became_peered":"2024-04-25T05:50:12.035111+0000","last_unstale":"2024-04-25T05:51:18.526483+0000","last_undegraded":"2024-04-25T05:51:18.526483+0000","last_fullsized":"2024-04-25T05:51:18.526483+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.368286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394788864,"num_objects":6024,"num_object_clones":0,"num_object_copies":18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":692,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064124+0000","last_change":"2024-04-25T05:50:11.713907+0000","last_active":"2024-04-25T05:51:15.064124+0000","last_peered":"2024-04-25T05:51:15.064124+0000","last_clean":"2024-04-25T05:51:15.064124+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:51:15.064124+0000","last_undegraded":"2024-04-25T05:51:15.064124+0000","last_fullsized":"2024-04-25T05:51:15.064124+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:50:11.713794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.713794+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.646485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":920,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063312+0000","last_change":"2024-04-25T05:47:52.970556+0000","last_active":"2024-04-25T05:51:15.063312+0000","last_peered":"2024-04-25T05:51:15.063312+0000","last_clean":"2024-04-25T05:51:15.063312+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:51:15.063312+0000","last_undegraded":"2024-04-25T05:51:15.063312+0000","last_fullsized":"2024-04-25T05:51:15.063312+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:52.970450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:52.970450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":217,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059980+0000","last_change":"2024-04-25T05:49:29.951087+0000","last_active":"2024-04-25T05:51:15.059980+0000","last_peered":"2024-04-25T05:51:15.059980+0000","last_clean":"2024-04-25T05:51:15.059980+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:51:15.059980+0000","last_undegraded":"2024-04-25T05:51:15.059980+0000","last_fullsized":"2024-04-25T05:51:15.059980+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:29.950964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:49:29.950964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994788+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":895,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060464+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:51:15.060464+0000","last_peered":"2024-04-25T05:51:15.060464+0000","last_clean":"2024-04-25T05:51:15.060464+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:51:15.060464+0000","last_undegraded":"2024-04-25T05:51:15.060464+0000","last_fullsized":"2024-04-25T05:51:15.060464+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.640676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"484'6051","reported_seq":6116,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.081714+0000","last_change":"2024-04-25T05:50:12.041469+0000","last_active":"2024-04-25T05:51:19.081714+0000","last_peered":"2024-04-25T05:51:19.081714+0000","last_clean":"2024-04-25T05:51:19.081714+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:51:19.081714+0000","last_undegraded":"2024-04-25T05:51:19.081714+0000","last_fullsized":"2024-04-25T05:51:19.081714+0000","mapping_epoch":456,"log_start":"467'2000","ondisk_log_start":"467'2000","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":18153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6051,"num_write_kb":387264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":936,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063700+0000","last_change":"2024-04-25T05:50:10.992094+0000","last_active":"2024-04-25T05:51:15.063700+0000","last_peered":"2024-04-25T05:51:15.063700+0000","last_clean":"2024-04-25T05:51:15.063700+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:51:15.063700+0000","last_undegraded":"2024-04-25T05:51:15.063700+0000","last_fullsized":"2024-04-25T05:51:15.063700+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:10.991997+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:50:10.991997+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.924649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":883,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062520+0000","last_change":"2024-04-25T05:47:24.459369+0000","last_active":"2024-04-25T05:51:15.062520+0000","last_peered":"2024-04-25T05:51:15.062520+0000","last_clean":"2024-04-25T05:51:15.062520+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:51:15.062520+0000","last_undegraded":"2024-04-25T05:51:15.062520+0000","last_fullsized":"2024-04-25T05:51:15.062520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:24.459254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:24.459254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390808+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"433'16","reported_seq":552,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064347+0000","last_change":"2024-04-25T05:49:09.610647+0000","last_active":"2024-04-25T05:51:15.064347+0000","last_peered":"2024-04-25T05:51:15.064347+0000","last_clean":"2024-04-25T05:51:15.064347+0000","last_became_active":"2024-04-25T05:49:09.561406+0000","last_became_peered":"2024-04-25T05:49:09.561406+0000","last_unstale":"2024-04-25T05:51:15.064347+0000","last_undegraded":"2024-04-25T05:51:15.064347+0000","last_fullsized":"2024-04-25T05:51:15.064347+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-25T05:48:21.090099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:21.090099+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811386+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":217,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059101+0000","last_change":"2024-04-25T05:48:30.199408+0000","last_active":"2024-04-25T05:51:15.059101+0000","last_peered":"2024-04-25T05:51:15.059101+0000","last_clean":"2024-04-25T05:51:15.059101+0000","last_became_active":"2024-04-25T05:46:34.989636+0000","last_became_peered":"2024-04-25T05:46:34.989636+0000","last_unstale":"2024-04-25T05:51:15.059101+0000","last_undegraded":"2024-04-25T05:51:15.059101+0000","last_fullsized":"2024-04-25T05:51:15.059101+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:30.199282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:48:30.199282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994552+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":778,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058617+0000","last_change":"2024-04-25T05:50:47.941283+0000","last_active":"2024-04-25T05:51:15.058617+0000","last_peered":"2024-04-25T05:51:15.058617+0000","last_clean":"2024-04-25T05:51:15.058617+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:51:15.058617+0000","last_undegraded":"2024-04-25T05:51:15.058617+0000","last_fullsized":"2024-04-25T05:51:15.058617+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:47.940906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.940906+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.568875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"484'6087","reported_seq":6152,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.142822+0000","last_change":"2024-04-25T05:50:12.042142+0000","last_active":"2024-04-25T05:51:18.142822+0000","last_peered":"2024-04-25T05:51:18.142822+0000","last_clean":"2024-04-25T05:51:18.142822+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:51:18.142822+0000","last_undegraded":"2024-04-25T05:51:18.142822+0000","last_fullsized":"2024-04-25T05:51:18.142822+0000","mapping_epoch":456,"log_start":"465'1600","ondisk_log_start":"465'1600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.962178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398917632,"num_objects":6087,"num_object_clones":0,"num_object_copies":18261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":389568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063197+0000","last_change":"2024-04-25T05:49:03.437727+0000","last_active":"2024-04-25T05:51:15.063197+0000","last_peered":"2024-04-25T05:51:15.063197+0000","last_clean":"2024-04-25T05:51:15.063197+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:51:15.063197+0000","last_undegraded":"2024-04-25T05:51:15.063197+0000","last_fullsized":"2024-04-25T05:51:15.063197+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:03.437604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:03.437604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.812213+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1038,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062617+0000","last_change":"2024-04-25T05:47:31.493128+0000","last_active":"2024-04-25T05:51:15.062617+0000","last_peered":"2024-04-25T05:51:15.062617+0000","last_clean":"2024-04-25T05:51:15.062617+0000","last_became_active":"2024-04-25T05:44:28.171918+0000","last_became_peered":"2024-04-25T05:44:28.171918+0000","last_unstale":"2024-04-25T05:51:15.062617+0000","last_undegraded":"2024-04-25T05:51:15.062617+0000","last_fullsized":"2024-04-25T05:51:15.062617+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:47:31.492960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:31.492960+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390876+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":916,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061907+0000","last_change":"2024-04-25T05:48:01.460845+0000","last_active":"2024-04-25T05:51:15.061907+0000","last_peered":"2024-04-25T05:51:15.061907+0000","last_clean":"2024-04-25T05:51:15.061907+0000","last_became_active":"2024-04-25T05:46:34.971958+0000","last_became_peered":"2024-04-25T05:46:34.971958+0000","last_unstale":"2024-04-25T05:51:15.061907+0000","last_undegraded":"2024-04-25T05:51:15.061907+0000","last_fullsized":"2024-04-25T05:51:15.061907+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":455,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063996+0000","last_change":"2024-04-25T05:50:41.801622+0000","last_active":"2024-04-25T05:51:15.063996+0000","last_peered":"2024-04-25T05:51:15.063996+0000","last_clean":"2024-04-25T05:51:15.063996+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:51:15.063996+0000","last_undegraded":"2024-04-25T05:51:15.063996+0000","last_fullsized":"2024-04-25T05:51:15.063996+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.801508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.801508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.561567+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1129,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064405+0000","last_change":"2024-04-25T05:49:01.441992+0000","last_active":"2024-04-25T05:51:15.064405+0000","last_peered":"2024-04-25T05:51:15.064405+0000","last_clean":"2024-04-25T05:51:15.064405+0000","last_became_active":"2024-04-25T05:46:23.130238+0000","last_became_peered":"2024-04-25T05:46:23.130238+0000","last_unstale":"2024-04-25T05:51:15.064405+0000","last_undegraded":"2024-04-25T05:51:15.064405+0000","last_fullsized":"2024-04-25T05:51:15.064405+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:01.440957+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.440957+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.812145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"484'5884","reported_seq":5948,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.109649+0000","last_change":"2024-04-25T05:50:12.042923+0000","last_active":"2024-04-25T05:51:18.109649+0000","last_peered":"2024-04-25T05:51:18.109649+0000","last_clean":"2024-04-25T05:51:18.109649+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:51:18.109649+0000","last_undegraded":"2024-04-25T05:51:18.109649+0000","last_fullsized":"2024-04-25T05:51:18.109649+0000","mapping_epoch":456,"log_start":"465'1600","ondisk_log_start":"465'1600","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:40.401319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385613824,"num_objects":5884,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":376576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1103,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062770+0000","last_change":"2024-04-25T05:48:25.100906+0000","last_active":"2024-04-25T05:51:15.062770+0000","last_peered":"2024-04-25T05:51:15.062770+0000","last_clean":"2024-04-25T05:51:15.062770+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:51:15.062770+0000","last_undegraded":"2024-04-25T05:51:15.062770+0000","last_fullsized":"2024-04-25T05:51:15.062770+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:48:25.100752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.100752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":585,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063847+0000","last_change":"2024-04-25T05:50:46.875806+0000","last_active":"2024-04-25T05:51:15.063847+0000","last_peered":"2024-04-25T05:51:15.063847+0000","last_clean":"2024-04-25T05:51:15.063847+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:51:15.063847+0000","last_undegraded":"2024-04-25T05:51:15.063847+0000","last_fullsized":"2024-04-25T05:51:15.063847+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:46.875662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.875662+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:49.181320+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":595,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063438+0000","last_change":"2024-04-25T05:50:15.054217+0000","last_active":"2024-04-25T05:51:15.063438+0000","last_peered":"2024-04-25T05:51:15.063438+0000","last_clean":"2024-04-25T05:51:15.063438+0000","last_became_active":"2024-04-25T05:44:28.177169+0000","last_became_peered":"2024-04-25T05:44:28.177169+0000","last_unstale":"2024-04-25T05:51:15.063438+0000","last_undegraded":"2024-04-25T05:51:15.063438+0000","last_fullsized":"2024-04-25T05:51:15.063438+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.219601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":273,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063231+0000","last_change":"2024-04-25T05:50:12.172126+0000","last_active":"2024-04-25T05:51:15.063231+0000","last_peered":"2024-04-25T05:51:15.063231+0000","last_clean":"2024-04-25T05:51:15.063231+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:51:15.063231+0000","last_undegraded":"2024-04-25T05:51:15.063231+0000","last_fullsized":"2024-04-25T05:51:15.063231+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:12.172001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.172001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.707108+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1085,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060276+0000","last_change":"2024-04-25T05:48:27.160787+0000","last_active":"2024-04-25T05:51:15.060276+0000","last_peered":"2024-04-25T05:51:15.060276+0000","last_clean":"2024-04-25T05:51:15.060276+0000","last_became_active":"2024-04-25T05:46:37.983371+0000","last_became_peered":"2024-04-25T05:46:37.983371+0000","last_unstale":"2024-04-25T05:51:15.060276+0000","last_undegraded":"2024-04-25T05:51:15.060276+0000","last_fullsized":"2024-04-25T05:51:15.060276+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:48:27.160639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.160639+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391061+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"484'5926","reported_seq":5992,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:17.731859+0000","last_change":"2024-04-25T05:50:12.035986+0000","last_active":"2024-04-25T05:51:17.731859+0000","last_peered":"2024-04-25T05:51:17.731859+0000","last_clean":"2024-04-25T05:51:17.731859+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:51:17.731859+0000","last_undegraded":"2024-04-25T05:51:17.731859+0000","last_fullsized":"2024-04-25T05:51:17.731859+0000","mapping_epoch":456,"log_start":"465'1000","ondisk_log_start":"465'1000","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:27.423843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388366336,"num_objects":5926,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":379264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"388'16","reported_seq":700,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061105+0000","last_change":"2024-04-25T05:50:45.130550+0000","last_active":"2024-04-25T05:51:15.061105+0000","last_peered":"2024-04-25T05:51:15.061105+0000","last_clean":"2024-04-25T05:51:15.061105+0000","last_became_active":"2024-04-25T05:46:34.975426+0000","last_became_peered":"2024-04-25T05:46:34.975426+0000","last_unstale":"2024-04-25T05:51:15.061105+0000","last_undegraded":"2024-04-25T05:51:15.061105+0000","last_fullsized":"2024-04-25T05:51:15.061105+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:51.401906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":632,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062277+0000","last_change":"2024-04-25T05:49:02.359488+0000","last_active":"2024-04-25T05:51:15.062277+0000","last_peered":"2024-04-25T05:51:15.062277+0000","last_clean":"2024-04-25T05:51:15.062277+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:51:15.062277+0000","last_undegraded":"2024-04-25T05:51:15.062277+0000","last_fullsized":"2024-04-25T05:51:15.062277+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:49:02.359173+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.359173+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":225,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059204+0000","last_change":"2024-04-25T05:50:08.761599+0000","last_active":"2024-04-25T05:51:15.059204+0000","last_peered":"2024-04-25T05:51:15.059204+0000","last_clean":"2024-04-25T05:51:15.059204+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:51:15.059204+0000","last_undegraded":"2024-04-25T05:51:15.059204+0000","last_fullsized":"2024-04-25T05:51:15.059204+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.520460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":530,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061972+0000","last_change":"2024-04-25T05:50:09.086597+0000","last_active":"2024-04-25T05:51:15.061972+0000","last_peered":"2024-04-25T05:51:15.061972+0000","last_clean":"2024-04-25T05:51:15.061972+0000","last_became_active":"2024-04-25T05:46:34.960364+0000","last_became_peered":"2024-04-25T05:46:34.960364+0000","last_unstale":"2024-04-25T05:51:15.061972+0000","last_undegraded":"2024-04-25T05:51:15.061972+0000","last_fullsized":"2024-04-25T05:51:15.061972+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.086479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.086479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.899606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1077,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060610+0000","last_change":"2024-04-25T05:50:07.959527+0000","last_active":"2024-04-25T05:51:15.060610+0000","last_peered":"2024-04-25T05:51:15.060610+0000","last_clean":"2024-04-25T05:51:15.060610+0000","last_became_active":"2024-04-25T05:39:32.805326+0000","last_became_peered":"2024-04-25T05:39:32.805326+0000","last_unstale":"2024-04-25T05:51:15.060610+0000","last_undegraded":"2024-04-25T05:51:15.060610+0000","last_fullsized":"2024-04-25T05:51:15.060610+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.959446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.959446+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:24.723125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"484'6231","reported_seq":6296,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.103871+0000","last_change":"2024-04-25T05:50:12.038131+0000","last_active":"2024-04-25T05:51:19.103871+0000","last_peered":"2024-04-25T05:51:19.103871+0000","last_clean":"2024-04-25T05:51:19.103871+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:51:19.103871+0000","last_undegraded":"2024-04-25T05:51:19.103871+0000","last_fullsized":"2024-04-25T05:51:19.103871+0000","mapping_epoch":456,"log_start":"467'2200","ondisk_log_start":"467'2200","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408354816,"num_objects":6231,"num_object_clones":0,"num_object_copies":18693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":398784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":652,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059360+0000","last_change":"2024-04-25T05:50:42.148569+0000","last_active":"2024-04-25T05:51:15.059360+0000","last_peered":"2024-04-25T05:51:15.059360+0000","last_clean":"2024-04-25T05:51:15.059360+0000","last_became_active":"2024-04-25T05:46:34.965922+0000","last_became_peered":"2024-04-25T05:46:34.965922+0000","last_unstale":"2024-04-25T05:51:15.059360+0000","last_undegraded":"2024-04-25T05:51:15.059360+0000","last_fullsized":"2024-04-25T05:51:15.059360+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:42.148429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:50:42.148429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:11.882512+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":263,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058957+0000","last_change":"2024-04-25T05:48:59.056819+0000","last_active":"2024-04-25T05:51:15.058957+0000","last_peered":"2024-04-25T05:51:15.058957+0000","last_clean":"2024-04-25T05:51:15.058957+0000","last_became_active":"2024-04-25T05:46:36.983724+0000","last_became_peered":"2024-04-25T05:46:36.983724+0000","last_unstale":"2024-04-25T05:51:15.058957+0000","last_undegraded":"2024-04-25T05:51:15.058957+0000","last_fullsized":"2024-04-25T05:51:15.058957+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:48:59.056469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.056469+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994761+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":929,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.067567+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:51:15.067567+0000","last_peered":"2024-04-25T05:51:15.067567+0000","last_clean":"2024-04-25T05:51:15.067567+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:51:15.067567+0000","last_undegraded":"2024-04-25T05:51:15.067567+0000","last_fullsized":"2024-04-25T05:51:15.067567+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.818280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":903,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064138+0000","last_change":"2024-04-25T05:49:32.024307+0000","last_active":"2024-04-25T05:51:15.064138+0000","last_peered":"2024-04-25T05:51:15.064138+0000","last_clean":"2024-04-25T05:51:15.064138+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:51:15.064138+0000","last_undegraded":"2024-04-25T05:51:15.064138+0000","last_fullsized":"2024-04-25T05:51:15.064138+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.024156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.024156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1095,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059575+0000","last_change":"2024-04-25T05:50:01.929134+0000","last_active":"2024-04-25T05:51:15.059575+0000","last_peered":"2024-04-25T05:51:15.059575+0000","last_clean":"2024-04-25T05:51:15.059575+0000","last_became_active":"2024-04-25T05:44:30.917921+0000","last_became_peered":"2024-04-25T05:44:30.917921+0000","last_unstale":"2024-04-25T05:51:15.059575+0000","last_undegraded":"2024-04-25T05:51:15.059575+0000","last_fullsized":"2024-04-25T05:51:15.059575+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:01.929038+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:50:01.929038+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:20.992655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"484'6162","reported_seq":6228,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.127563+0000","last_change":"2024-04-25T05:50:12.035979+0000","last_active":"2024-04-25T05:51:19.127563+0000","last_peered":"2024-04-25T05:51:19.127563+0000","last_clean":"2024-04-25T05:51:19.127563+0000","last_became_active":"2024-04-25T05:50:12.035482+0000","last_became_peered":"2024-04-25T05:50:12.035482+0000","last_unstale":"2024-04-25T05:51:19.127563+0000","last_undegraded":"2024-04-25T05:51:19.127563+0000","last_fullsized":"2024-04-25T05:51:19.127563+0000","mapping_epoch":456,"log_start":"466'2000","ondisk_log_start":"466'2000","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.703340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":217,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060066+0000","last_change":"2024-04-25T05:48:02.052034+0000","last_active":"2024-04-25T05:51:15.060066+0000","last_peered":"2024-04-25T05:51:15.060066+0000","last_clean":"2024-04-25T05:51:15.060066+0000","last_became_active":"2024-04-25T05:46:34.985687+0000","last_became_peered":"2024-04-25T05:46:34.985687+0000","last_unstale":"2024-04-25T05:51:15.060066+0000","last_undegraded":"2024-04-25T05:51:15.060066+0000","last_fullsized":"2024-04-25T05:51:15.060066+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:02.051790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.051790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":239,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059321+0000","last_change":"2024-04-25T05:50:12.786445+0000","last_active":"2024-04-25T05:51:15.059321+0000","last_peered":"2024-04-25T05:51:15.059321+0000","last_clean":"2024-04-25T05:51:15.059321+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:51:15.059321+0000","last_undegraded":"2024-04-25T05:51:15.059321+0000","last_fullsized":"2024-04-25T05:51:15.059321+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:50:12.786333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.786333+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:33.425682+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1963,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:16.980146+0000","last_change":"2024-04-25T05:48:02.759509+0000","last_active":"2024-04-25T05:51:16.980146+0000","last_peered":"2024-04-25T05:51:16.980146+0000","last_clean":"2024-04-25T05:51:16.980146+0000","last_became_active":"2024-04-25T05:44:28.173921+0000","last_became_peered":"2024-04-25T05:44:28.173921+0000","last_unstale":"2024-04-25T05:51:16.980146+0000","last_undegraded":"2024-04-25T05:51:16.980146+0000","last_fullsized":"2024-04-25T05:51:16.980146+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:48:02.759343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:48:02.759343+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1021,"num_read_kb":1016,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":525,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063355+0000","last_change":"2024-04-25T05:49:05.488606+0000","last_active":"2024-04-25T05:51:15.063355+0000","last_peered":"2024-04-25T05:51:15.063355+0000","last_clean":"2024-04-25T05:51:15.063355+0000","last_became_active":"2024-04-25T05:46:34.967118+0000","last_became_peered":"2024-04-25T05:46:34.967118+0000","last_unstale":"2024-04-25T05:51:15.063355+0000","last_undegraded":"2024-04-25T05:51:15.063355+0000","last_fullsized":"2024-04-25T05:51:15.063355+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.488514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.488514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.812182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":891,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.067282+0000","last_change":"2024-04-25T05:48:35.483025+0000","last_active":"2024-04-25T05:51:15.067282+0000","last_peered":"2024-04-25T05:51:15.067282+0000","last_clean":"2024-04-25T05:51:15.067282+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:51:15.067282+0000","last_undegraded":"2024-04-25T05:51:15.067282+0000","last_fullsized":"2024-04-25T05:51:15.067282+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.482886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.482886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1073,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063030+0000","last_change":"2024-04-25T05:49:08.317583+0000","last_active":"2024-04-25T05:51:15.063030+0000","last_peered":"2024-04-25T05:51:15.063030+0000","last_clean":"2024-04-25T05:51:15.063030+0000","last_became_active":"2024-04-25T05:39:32.812683+0000","last_became_peered":"2024-04-25T05:39:32.812683+0000","last_unstale":"2024-04-25T05:51:15.063030+0000","last_undegraded":"2024-04-25T05:51:15.063030+0000","last_fullsized":"2024-04-25T05:51:15.063030+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:49:08.317433+0000","last_clean_scrub_stamp":"2024-04-25T05:49:08.317433+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"484'6030","reported_seq":6091,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.132708+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:51:19.132708+0000","last_peered":"2024-04-25T05:51:19.132708+0000","last_clean":"2024-04-25T05:51:19.132708+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:51:19.132708+0000","last_undegraded":"2024-04-25T05:51:19.132708+0000","last_fullsized":"2024-04-25T05:51:19.132708+0000","mapping_epoch":456,"log_start":"466'1900","ondisk_log_start":"466'1900","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:38.159052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395182080,"num_objects":6030,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":385920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":913,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054137+0000","last_change":"2024-04-25T05:47:33.735483+0000","last_active":"2024-04-25T05:51:15.054137+0000","last_peered":"2024-04-25T05:51:15.054137+0000","last_clean":"2024-04-25T05:51:15.054137+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:51:15.054137+0000","last_undegraded":"2024-04-25T05:51:15.054137+0000","last_fullsized":"2024-04-25T05:51:15.054137+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:33.735345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:47:33.735345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":799,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061896+0000","last_change":"2024-04-25T05:50:13.028112+0000","last_active":"2024-04-25T05:51:15.061896+0000","last_peered":"2024-04-25T05:51:15.061896+0000","last_clean":"2024-04-25T05:51:15.061896+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:51:15.061896+0000","last_undegraded":"2024-04-25T05:51:15.061896+0000","last_fullsized":"2024-04-25T05:51:15.061896+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.027901+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.798452+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":908,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.074328+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:51:15.074328+0000","last_peered":"2024-04-25T05:51:15.074328+0000","last_clean":"2024-04-25T05:51:15.074328+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:51:15.074328+0000","last_undegraded":"2024-04-25T05:51:15.074328+0000","last_fullsized":"2024-04-25T05:51:15.074328+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1208,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053387+0000","last_change":"2024-04-25T05:50:07.888799+0000","last_active":"2024-04-25T05:51:15.053387+0000","last_peered":"2024-04-25T05:51:15.053387+0000","last_clean":"2024-04-25T05:51:15.053387+0000","last_became_active":"2024-04-25T05:44:30.947379+0000","last_became_peered":"2024-04-25T05:44:30.947379+0000","last_unstale":"2024-04-25T05:51:15.053387+0000","last_undegraded":"2024-04-25T05:51:15.053387+0000","last_fullsized":"2024-04-25T05:51:15.053387+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:07.888714+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.888714+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:28.057112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":197,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063798+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:51:15.063798+0000","last_peered":"2024-04-25T05:51:15.063798+0000","last_clean":"2024-04-25T05:51:15.063798+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:51:15.063798+0000","last_undegraded":"2024-04-25T05:51:15.063798+0000","last_fullsized":"2024-04-25T05:51:15.063798+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.250098+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"484'6091","reported_seq":6157,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.097320+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:51:19.097320+0000","last_peered":"2024-04-25T05:51:19.097320+0000","last_clean":"2024-04-25T05:51:19.097320+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:51:19.097320+0000","last_undegraded":"2024-04-25T05:51:19.097320+0000","last_fullsized":"2024-04-25T05:51:19.097320+0000","mapping_epoch":456,"log_start":"467'1900","ondisk_log_start":"467'1900","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:37.808327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399179776,"num_objects":6091,"num_object_clones":0,"num_object_copies":18273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":256,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058462+0000","last_change":"2024-04-25T05:50:45.550588+0000","last_active":"2024-04-25T05:51:15.058462+0000","last_peered":"2024-04-25T05:51:15.058462+0000","last_clean":"2024-04-25T05:51:15.058462+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:51:15.058462+0000","last_undegraded":"2024-04-25T05:51:15.058462+0000","last_fullsized":"2024-04-25T05:51:15.058462+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.550474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.550474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.043449+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1037,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063601+0000","last_change":"2024-04-25T05:50:40.134940+0000","last_active":"2024-04-25T05:51:15.063601+0000","last_peered":"2024-04-25T05:51:15.063601+0000","last_clean":"2024-04-25T05:51:15.063601+0000","last_became_active":"2024-04-25T05:46:34.974492+0000","last_became_peered":"2024-04-25T05:46:34.974492+0000","last_unstale":"2024-04-25T05:51:15.063601+0000","last_undegraded":"2024-04-25T05:51:15.063601+0000","last_fullsized":"2024-04-25T05:51:15.063601+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:50:40.134604+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.134604+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:41.529906+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":250,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059161+0000","last_change":"2024-04-25T05:50:14.628452+0000","last_active":"2024-04-25T05:51:15.059161+0000","last_peered":"2024-04-25T05:51:15.059161+0000","last_clean":"2024-04-25T05:51:15.059161+0000","last_became_active":"2024-04-25T05:46:19.997992+0000","last_became_peered":"2024-04-25T05:46:19.997992+0000","last_unstale":"2024-04-25T05:51:15.059161+0000","last_undegraded":"2024-04-25T05:51:15.059161+0000","last_fullsized":"2024-04-25T05:51:15.059161+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.601623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":935,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063594+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:51:15.063594+0000","last_peered":"2024-04-25T05:51:15.063594+0000","last_clean":"2024-04-25T05:51:15.063594+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:51:15.063594+0000","last_undegraded":"2024-04-25T05:51:15.063594+0000","last_fullsized":"2024-04-25T05:51:15.063594+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.323525+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:17.037436+0000","last_change":"2024-04-25T05:51:17.037477+0000","last_active":"2024-04-25T05:51:17.037436+0000","last_peered":"2024-04-25T05:51:17.037436+0000","last_clean":"2024-04-25T05:51:17.037436+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:51:17.037436+0000","last_undegraded":"2024-04-25T05:51:17.037436+0000","last_fullsized":"2024-04-25T05:51:17.037436+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:17.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:51:17.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:26.928401+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1216,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053935+0000","last_change":"2024-04-25T05:49:32.818373+0000","last_active":"2024-04-25T05:51:15.053935+0000","last_peered":"2024-04-25T05:51:15.053935+0000","last_clean":"2024-04-25T05:51:15.053935+0000","last_became_active":"2024-04-25T05:46:23.132479+0000","last_became_peered":"2024-04-25T05:46:23.132479+0000","last_unstale":"2024-04-25T05:51:15.053935+0000","last_undegraded":"2024-04-25T05:51:15.053935+0000","last_fullsized":"2024-04-25T05:51:15.053935+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:32.818274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.818274+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"484'6130","reported_seq":6196,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:19.108757+0000","last_change":"2024-04-25T05:50:12.040668+0000","last_active":"2024-04-25T05:51:19.108757+0000","last_peered":"2024-04-25T05:51:19.108757+0000","last_clean":"2024-04-25T05:51:19.108757+0000","last_became_active":"2024-04-25T05:50:12.040023+0000","last_became_peered":"2024-04-25T05:50:12.040023+0000","last_unstale":"2024-04-25T05:51:19.108757+0000","last_undegraded":"2024-04-25T05:51:19.108757+0000","last_fullsized":"2024-04-25T05:51:19.108757+0000","mapping_epoch":456,"log_start":"466'2000","ondisk_log_start":"466'2000","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:36.039935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":392320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":516,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062633+0000","last_change":"2024-04-25T05:48:29.428977+0000","last_active":"2024-04-25T05:51:15.062633+0000","last_peered":"2024-04-25T05:51:15.062633+0000","last_clean":"2024-04-25T05:51:15.062633+0000","last_became_active":"2024-04-25T05:44:28.184233+0000","last_became_peered":"2024-04-25T05:44:28.184233+0000","last_unstale":"2024-04-25T05:51:15.062633+0000","last_undegraded":"2024-04-25T05:51:15.062633+0000","last_fullsized":"2024-04-25T05:51:15.062633+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:29.428764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:29.428764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1245,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.067699+0000","last_change":"2024-04-25T05:50:07.985223+0000","last_active":"2024-04-25T05:51:15.067699+0000","last_peered":"2024-04-25T05:51:15.067699+0000","last_clean":"2024-04-25T05:51:15.067699+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:51:15.067699+0000","last_undegraded":"2024-04-25T05:51:15.067699+0000","last_fullsized":"2024-04-25T05:51:15.067699+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:50:07.985110+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.985110+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.172799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":460,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064377+0000","last_change":"2024-04-25T05:49:09.560837+0000","last_active":"2024-04-25T05:51:15.064377+0000","last_peered":"2024-04-25T05:51:15.064377+0000","last_clean":"2024-04-25T05:51:15.064377+0000","last_became_active":"2024-04-25T05:49:09.559684+0000","last_became_peered":"2024-04-25T05:49:09.559684+0000","last_unstale":"2024-04-25T05:51:15.064377+0000","last_undegraded":"2024-04-25T05:51:15.064377+0000","last_fullsized":"2024-04-25T05:51:15.064377+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:37.254738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:48:37.254738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":229,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058420+0000","last_change":"2024-04-25T05:50:04.735739+0000","last_active":"2024-04-25T05:51:15.058420+0000","last_peered":"2024-04-25T05:51:15.058420+0000","last_clean":"2024-04-25T05:51:15.058420+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:51:15.058420+0000","last_undegraded":"2024-04-25T05:51:15.058420+0000","last_fullsized":"2024-04-25T05:51:15.058420+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:50:04.735651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.735651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:21.992705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":871,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061498+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:51:15.061498+0000","last_peered":"2024-04-25T05:51:15.061498+0000","last_clean":"2024-04-25T05:51:15.061498+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:51:15.061498+0000","last_undegraded":"2024-04-25T05:51:15.061498+0000","last_fullsized":"2024-04-25T05:51:15.061498+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391237+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":917,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.057939+0000","last_change":"2024-04-25T05:49:00.009223+0000","last_active":"2024-04-25T05:51:15.057939+0000","last_peered":"2024-04-25T05:51:15.057939+0000","last_clean":"2024-04-25T05:51:15.057939+0000","last_became_active":"2024-04-25T05:46:37.982024+0000","last_became_peered":"2024-04-25T05:46:37.982024+0000","last_unstale":"2024-04-25T05:51:15.057939+0000","last_undegraded":"2024-04-25T05:51:15.057939+0000","last_fullsized":"2024-04-25T05:51:15.057939+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:00.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:00.008984+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"484'5979","reported_seq":6044,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.156592+0000","last_change":"2024-04-25T05:50:12.043881+0000","last_active":"2024-04-25T05:51:18.156592+0000","last_peered":"2024-04-25T05:51:18.156592+0000","last_clean":"2024-04-25T05:51:18.156592+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:51:18.156592+0000","last_undegraded":"2024-04-25T05:51:18.156592+0000","last_fullsized":"2024-04-25T05:51:18.156592+0000","mapping_epoch":456,"log_start":"465'1500","ondisk_log_start":"465'1500","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391839744,"num_objects":5979,"num_object_clones":0,"num_object_copies":17937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":382656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":609,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062851+0000","last_change":"2024-04-25T05:50:14.065922+0000","last_active":"2024-04-25T05:51:15.062851+0000","last_peered":"2024-04-25T05:51:15.062851+0000","last_clean":"2024-04-25T05:51:15.062851+0000","last_became_active":"2024-04-25T05:44:28.181412+0000","last_became_peered":"2024-04-25T05:44:28.181412+0000","last_unstale":"2024-04-25T05:51:15.062851+0000","last_undegraded":"2024-04-25T05:51:15.062851+0000","last_fullsized":"2024-04-25T05:51:15.062851+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:22.097234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":217,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058616+0000","last_change":"2024-04-25T05:48:20.054569+0000","last_active":"2024-04-25T05:51:15.058616+0000","last_peered":"2024-04-25T05:51:15.058616+0000","last_clean":"2024-04-25T05:51:15.058616+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:51:15.058616+0000","last_undegraded":"2024-04-25T05:51:15.058616+0000","last_fullsized":"2024-04-25T05:51:15.058616+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:20.054446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:48:20.054446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994524+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":242,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.057969+0000","last_change":"2024-04-25T05:50:11.521033+0000","last_active":"2024-04-25T05:51:15.057969+0000","last_peered":"2024-04-25T05:51:15.057969+0000","last_clean":"2024-04-25T05:51:15.057969+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:51:15.057969+0000","last_undegraded":"2024-04-25T05:51:15.057969+0000","last_fullsized":"2024-04-25T05:51:15.057969+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.520907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.520907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.942600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":516,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054644+0000","last_change":"2024-04-25T05:51:15.026919+0000","last_active":"2024-04-25T05:51:15.054644+0000","last_peered":"2024-04-25T05:51:15.054644+0000","last_clean":"2024-04-25T05:51:15.054644+0000","last_became_active":"2024-04-25T05:46:19.988789+0000","last_became_peered":"2024-04-25T05:46:19.988789+0000","last_unstale":"2024-04-25T05:51:15.054644+0000","last_undegraded":"2024-04-25T05:51:15.054644+0000","last_fullsized":"2024-04-25T05:51:15.054644+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.026628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.026628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:22.404105+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"484'6148","reported_seq":6213,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.162088+0000","last_change":"2024-04-25T05:50:12.042685+0000","last_active":"2024-04-25T05:51:18.162088+0000","last_peered":"2024-04-25T05:51:18.162088+0000","last_clean":"2024-04-25T05:51:18.162088+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:51:18.162088+0000","last_undegraded":"2024-04-25T05:51:18.162088+0000","last_fullsized":"2024-04-25T05:51:18.162088+0000","mapping_epoch":456,"log_start":"466'1700","ondisk_log_start":"466'1700","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:35.841340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":678,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059183+0000","last_change":"2024-04-25T05:50:15.942029+0000","last_active":"2024-04-25T05:51:15.059183+0000","last_peered":"2024-04-25T05:51:15.059183+0000","last_clean":"2024-04-25T05:51:15.059183+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:51:15.059183+0000","last_undegraded":"2024-04-25T05:51:15.059183+0000","last_fullsized":"2024-04-25T05:51:15.059183+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.941896+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.479186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":945,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054297+0000","last_change":"2024-04-25T05:50:47.124683+0000","last_active":"2024-04-25T05:51:15.054297+0000","last_peered":"2024-04-25T05:51:15.054297+0000","last_clean":"2024-04-25T05:51:15.054297+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:51:15.054297+0000","last_undegraded":"2024-04-25T05:51:15.054297+0000","last_fullsized":"2024-04-25T05:51:15.054297+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.124554+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.362413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":248,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058999+0000","last_change":"2024-04-25T05:50:05.416344+0000","last_active":"2024-04-25T05:51:15.058999+0000","last_peered":"2024-04-25T05:51:15.058999+0000","last_clean":"2024-04-25T05:51:15.058999+0000","last_became_active":"2024-04-25T05:46:34.970658+0000","last_became_peered":"2024-04-25T05:46:34.970658+0000","last_unstale":"2024-04-25T05:51:15.058999+0000","last_undegraded":"2024-04-25T05:51:15.058999+0000","last_fullsized":"2024-04-25T05:51:15.058999+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.416227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.416227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.653594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":890,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053270+0000","last_change":"2024-04-25T05:48:03.701098+0000","last_active":"2024-04-25T05:51:15.053270+0000","last_peered":"2024-04-25T05:51:15.053270+0000","last_clean":"2024-04-25T05:51:15.053270+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:51:15.053270+0000","last_undegraded":"2024-04-25T05:51:15.053270+0000","last_fullsized":"2024-04-25T05:51:15.053270+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:03.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:03.700968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"484'5894","reported_seq":5960,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:17.705305+0000","last_change":"2024-04-25T05:50:12.035713+0000","last_active":"2024-04-25T05:51:17.705305+0000","last_peered":"2024-04-25T05:51:17.705305+0000","last_clean":"2024-04-25T05:51:17.705305+0000","last_became_active":"2024-04-25T05:50:12.035376+0000","last_became_peered":"2024-04-25T05:50:12.035376+0000","last_unstale":"2024-04-25T05:51:17.705305+0000","last_undegraded":"2024-04-25T05:51:17.705305+0000","last_fullsized":"2024-04-25T05:51:17.705305+0000","mapping_epoch":456,"log_start":"465'1000","ondisk_log_start":"465'1000","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:31.653834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":377216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":256,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.057979+0000","last_change":"2024-04-25T05:50:39.566845+0000","last_active":"2024-04-25T05:51:15.057979+0000","last_peered":"2024-04-25T05:51:15.057979+0000","last_clean":"2024-04-25T05:51:15.057979+0000","last_became_active":"2024-04-25T05:46:34.970825+0000","last_became_peered":"2024-04-25T05:46:34.970825+0000","last_unstale":"2024-04-25T05:51:15.057979+0000","last_undegraded":"2024-04-25T05:51:15.057979+0000","last_fullsized":"2024-04-25T05:51:15.057979+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.566717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.566717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.356177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":306,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063680+0000","last_change":"2024-04-25T05:49:09.598004+0000","last_active":"2024-04-25T05:51:15.063680+0000","last_peered":"2024-04-25T05:51:15.063680+0000","last_clean":"2024-04-25T05:51:15.063680+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:51:15.063680+0000","last_undegraded":"2024-04-25T05:51:15.063680+0000","last_fullsized":"2024-04-25T05:51:15.063680+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:48:24.506969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.506969+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811467+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":898,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054143+0000","last_change":"2024-04-25T05:50:41.101003+0000","last_active":"2024-04-25T05:51:15.054143+0000","last_peered":"2024-04-25T05:51:15.054143+0000","last_clean":"2024-04-25T05:51:15.054143+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:51:15.054143+0000","last_undegraded":"2024-04-25T05:51:15.054143+0000","last_fullsized":"2024-04-25T05:51:15.054143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.100880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.100880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:43.503491+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":883,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061075+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:51:15.061075+0000","last_peered":"2024-04-25T05:51:15.061075+0000","last_clean":"2024-04-25T05:51:15.061075+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:51:15.061075+0000","last_undegraded":"2024-04-25T05:51:15.061075+0000","last_fullsized":"2024-04-25T05:51:15.061075+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.671987+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"484'5959","reported_seq":6023,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.153602+0000","last_change":"2024-04-25T05:50:12.042051+0000","last_active":"2024-04-25T05:51:18.153602+0000","last_peered":"2024-04-25T05:51:18.153602+0000","last_clean":"2024-04-25T05:51:18.153602+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:51:18.153602+0000","last_undegraded":"2024-04-25T05:51:18.153602+0000","last_fullsized":"2024-04-25T05:51:18.153602+0000","mapping_epoch":456,"log_start":"465'1100","ondisk_log_start":"465'1100","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390529024,"num_objects":5959,"num_object_clones":0,"num_object_copies":17877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5959,"num_write_kb":381376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1133,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053337+0000","last_change":"2024-04-25T05:50:39.051287+0000","last_active":"2024-04-25T05:51:15.053337+0000","last_peered":"2024-04-25T05:51:15.053337+0000","last_clean":"2024-04-25T05:51:15.053337+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:51:15.053337+0000","last_undegraded":"2024-04-25T05:51:15.053337+0000","last_fullsized":"2024-04-25T05:51:15.053337+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:50:39.050944+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.050944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:07.882347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":245,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058310+0000","last_change":"2024-04-25T05:50:11.779506+0000","last_active":"2024-04-25T05:51:15.058310+0000","last_peered":"2024-04-25T05:51:15.058310+0000","last_clean":"2024-04-25T05:51:15.058310+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:51:15.058310+0000","last_undegraded":"2024-04-25T05:51:15.058310+0000","last_fullsized":"2024-04-25T05:51:15.058310+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:50:11.779355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.779355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.341739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":938,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054641+0000","last_change":"2024-04-25T05:48:59.765915+0000","last_active":"2024-04-25T05:51:15.054641+0000","last_peered":"2024-04-25T05:51:15.054641+0000","last_clean":"2024-04-25T05:51:15.054641+0000","last_became_active":"2024-04-25T05:44:28.165906+0000","last_became_peered":"2024-04-25T05:44:28.165906+0000","last_unstale":"2024-04-25T05:51:15.054641+0000","last_undegraded":"2024-04-25T05:51:15.054641+0000","last_fullsized":"2024-04-25T05:51:15.054641+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:48:59.765781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.765781+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":217,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058519+0000","last_change":"2024-04-25T05:49:01.017521+0000","last_active":"2024-04-25T05:51:15.058519+0000","last_peered":"2024-04-25T05:51:15.058519+0000","last_clean":"2024-04-25T05:51:15.058519+0000","last_became_active":"2024-04-25T05:46:34.972304+0000","last_became_peered":"2024-04-25T05:46:34.972304+0000","last_unstale":"2024-04-25T05:51:15.058519+0000","last_undegraded":"2024-04-25T05:51:15.058519+0000","last_fullsized":"2024-04-25T05:51:15.058519+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:01.017406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.017406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994731+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":909,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062155+0000","last_change":"2024-04-25T05:47:32.451583+0000","last_active":"2024-04-25T05:51:15.062155+0000","last_peered":"2024-04-25T05:51:15.062155+0000","last_clean":"2024-04-25T05:51:15.062155+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:51:15.062155+0000","last_undegraded":"2024-04-25T05:51:15.062155+0000","last_fullsized":"2024-04-25T05:51:15.062155+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:47:32.451448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:47:32.451448+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":652,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059508+0000","last_change":"2024-04-25T05:50:43.106046+0000","last_active":"2024-04-25T05:51:15.059508+0000","last_peered":"2024-04-25T05:51:15.059508+0000","last_clean":"2024-04-25T05:51:15.059508+0000","last_became_active":"2024-04-25T05:46:34.966182+0000","last_became_peered":"2024-04-25T05:46:34.966182+0000","last_unstale":"2024-04-25T05:51:15.059508+0000","last_undegraded":"2024-04-25T05:51:15.059508+0000","last_fullsized":"2024-04-25T05:51:15.059508+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.105913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.105913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.313735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":224,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059241+0000","last_change":"2024-04-25T05:50:43.938330+0000","last_active":"2024-04-25T05:51:15.059241+0000","last_peered":"2024-04-25T05:51:15.059241+0000","last_clean":"2024-04-25T05:51:15.059241+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:51:15.059241+0000","last_undegraded":"2024-04-25T05:51:15.059241+0000","last_fullsized":"2024-04-25T05:51:15.059241+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.938246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.938246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:52.848134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":898,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053856+0000","last_change":"2024-04-25T05:48:57.925411+0000","last_active":"2024-04-25T05:51:15.053856+0000","last_peered":"2024-04-25T05:51:15.053856+0000","last_clean":"2024-04-25T05:51:15.053856+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:51:15.053856+0000","last_undegraded":"2024-04-25T05:51:15.053856+0000","last_fullsized":"2024-04-25T05:51:15.053856+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:57.925301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.925301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":597,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061407+0000","last_change":"2024-04-25T05:49:32.097695+0000","last_active":"2024-04-25T05:51:15.061407+0000","last_peered":"2024-04-25T05:51:15.061407+0000","last_clean":"2024-04-25T05:51:15.061407+0000","last_became_active":"2024-04-25T05:46:19.997681+0000","last_became_peered":"2024-04-25T05:46:19.997681+0000","last_unstale":"2024-04-25T05:51:15.061407+0000","last_undegraded":"2024-04-25T05:51:15.061407+0000","last_fullsized":"2024-04-25T05:51:15.061407+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:49:32.097448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.097448+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391212+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":916,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053507+0000","last_change":"2024-04-25T05:50:53.041228+0000","last_active":"2024-04-25T05:51:15.053507+0000","last_peered":"2024-04-25T05:51:15.053507+0000","last_clean":"2024-04-25T05:51:15.053507+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:51:15.053507+0000","last_undegraded":"2024-04-25T05:51:15.053507+0000","last_fullsized":"2024-04-25T05:51:15.053507+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:53.041118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:50:53.041118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.078933+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":867,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061959+0000","last_change":"2024-04-25T05:47:55.512321+0000","last_active":"2024-04-25T05:51:15.061959+0000","last_peered":"2024-04-25T05:51:15.061959+0000","last_clean":"2024-04-25T05:51:15.061959+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:51:15.061959+0000","last_undegraded":"2024-04-25T05:51:15.061959+0000","last_fullsized":"2024-04-25T05:51:15.061959+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:55.512158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:55.512158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390905+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":242,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059432+0000","last_change":"2024-04-25T05:50:13.581116+0000","last_active":"2024-04-25T05:51:15.059432+0000","last_peered":"2024-04-25T05:51:15.059432+0000","last_clean":"2024-04-25T05:51:15.059432+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:51:15.059432+0000","last_undegraded":"2024-04-25T05:51:15.059432+0000","last_fullsized":"2024-04-25T05:51:15.059432+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.581028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.581028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:42.001393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":228,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058788+0000","last_change":"2024-04-25T05:48:26.172633+0000","last_active":"2024-04-25T05:51:15.058788+0000","last_peered":"2024-04-25T05:51:15.058788+0000","last_clean":"2024-04-25T05:51:15.058788+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:51:15.058788+0000","last_undegraded":"2024-04-25T05:51:15.058788+0000","last_fullsized":"2024-04-25T05:51:15.058788+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:48:26.172484+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:48:26.172484+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":501,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053682+0000","last_change":"2024-04-25T05:48:01.779274+0000","last_active":"2024-04-25T05:51:15.053682+0000","last_peered":"2024-04-25T05:51:15.053682+0000","last_clean":"2024-04-25T05:51:15.053682+0000","last_became_active":"2024-04-25T05:44:28.171463+0000","last_became_peered":"2024-04-25T05:44:28.171463+0000","last_unstale":"2024-04-25T05:51:15.053682+0000","last_undegraded":"2024-04-25T05:51:15.053682+0000","last_fullsized":"2024-04-25T05:51:15.053682+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1112,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061043+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:51:15.061043+0000","last_peered":"2024-04-25T05:51:15.061043+0000","last_clean":"2024-04-25T05:51:15.061043+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:51:15.061043+0000","last_undegraded":"2024-04-25T05:51:15.061043+0000","last_fullsized":"2024-04-25T05:51:15.061043+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.466618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":437,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064574+0000","last_change":"2024-04-25T05:50:52.078015+0000","last_active":"2024-04-25T05:51:15.064574+0000","last_peered":"2024-04-25T05:51:15.064574+0000","last_clean":"2024-04-25T05:51:15.064574+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:51:15.064574+0000","last_undegraded":"2024-04-25T05:51:15.064574+0000","last_fullsized":"2024-04-25T05:51:15.064574+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.077914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.221398+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":933,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060574+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:51:15.060574+0000","last_peered":"2024-04-25T05:51:15.060574+0000","last_clean":"2024-04-25T05:51:15.060574+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:51:15.060574+0000","last_undegraded":"2024-04-25T05:51:15.060574+0000","last_fullsized":"2024-04-25T05:51:15.060574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390997+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":902,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054475+0000","last_change":"2024-04-25T05:47:54.761281+0000","last_active":"2024-04-25T05:51:15.054475+0000","last_peered":"2024-04-25T05:51:15.054475+0000","last_clean":"2024-04-25T05:51:15.054475+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:51:15.054475+0000","last_undegraded":"2024-04-25T05:51:15.054475+0000","last_fullsized":"2024-04-25T05:51:15.054475+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:54.761147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:47:54.761147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"388'13","reported_seq":276,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058995+0000","last_change":"2024-04-25T05:49:01.967445+0000","last_active":"2024-04-25T05:51:15.058995+0000","last_peered":"2024-04-25T05:51:15.058995+0000","last_clean":"2024-04-25T05:51:15.058995+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:15.058995+0000","last_undegraded":"2024-04-25T05:51:15.058995+0000","last_fullsized":"2024-04-25T05:51:15.058995+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'13","last_scrub_stamp":"2024-04-25T05:49:01.967348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.967348+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994704+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":501,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053927+0000","last_change":"2024-04-25T05:47:22.804912+0000","last_active":"2024-04-25T05:51:15.053927+0000","last_peered":"2024-04-25T05:51:15.053927+0000","last_clean":"2024-04-25T05:51:15.053927+0000","last_became_active":"2024-04-25T05:44:28.178652+0000","last_became_peered":"2024-04-25T05:44:28.178652+0000","last_unstale":"2024-04-25T05:51:15.053927+0000","last_undegraded":"2024-04-25T05:51:15.053927+0000","last_fullsized":"2024-04-25T05:51:15.053927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:22.804781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:47:22.804781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1029,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064774+0000","last_change":"2024-04-25T05:47:51.961448+0000","last_active":"2024-04-25T05:51:15.064774+0000","last_peered":"2024-04-25T05:51:15.064774+0000","last_clean":"2024-04-25T05:51:15.064774+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:51:15.064774+0000","last_undegraded":"2024-04-25T05:51:15.064774+0000","last_fullsized":"2024-04-25T05:51:15.064774+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:47:51.961301+0000","last_deep_scrub":"291'16","last_deep_scrub_stamp":"2024-04-25T05:42:49.238022+0000","last_clean_scrub_stamp":"2024-04-25T05:47:51.961301+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811278+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":483,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064337+0000","last_change":"2024-04-25T05:49:35.656164+0000","last_active":"2024-04-25T05:51:15.064337+0000","last_peered":"2024-04-25T05:51:15.064337+0000","last_clean":"2024-04-25T05:51:15.064337+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:51:15.064337+0000","last_undegraded":"2024-04-25T05:51:15.064337+0000","last_fullsized":"2024-04-25T05:51:15.064337+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:49:35.656070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.656070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.813062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":897,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062173+0000","last_change":"2024-04-25T05:50:13.065576+0000","last_active":"2024-04-25T05:51:15.062173+0000","last_peered":"2024-04-25T05:51:15.062173+0000","last_clean":"2024-04-25T05:51:15.062173+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:51:15.062173+0000","last_undegraded":"2024-04-25T05:51:15.062173+0000","last_fullsized":"2024-04-25T05:51:15.062173+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.065464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.065464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.492742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":915,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.068808+0000","last_change":"2024-04-25T05:50:06.095785+0000","last_active":"2024-04-25T05:51:15.068808+0000","last_peered":"2024-04-25T05:51:15.068808+0000","last_clean":"2024-04-25T05:51:15.068808+0000","last_became_active":"2024-04-25T05:46:34.970937+0000","last_became_peered":"2024-04-25T05:46:34.970937+0000","last_unstale":"2024-04-25T05:51:15.068808+0000","last_undegraded":"2024-04-25T05:51:15.068808+0000","last_fullsized":"2024-04-25T05:51:15.068808+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.604835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":444,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062962+0000","last_change":"2024-04-25T05:48:57.339166+0000","last_active":"2024-04-25T05:51:15.062962+0000","last_peered":"2024-04-25T05:51:15.062962+0000","last_clean":"2024-04-25T05:51:15.062962+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:51:15.062962+0000","last_undegraded":"2024-04-25T05:51:15.062962+0000","last_fullsized":"2024-04-25T05:51:15.062962+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:48:57.338784+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:48:57.338784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":898,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053960+0000","last_change":"2024-04-25T05:50:45.109485+0000","last_active":"2024-04-25T05:51:15.053960+0000","last_peered":"2024-04-25T05:51:15.053960+0000","last_clean":"2024-04-25T05:51:15.053960+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:51:15.053960+0000","last_undegraded":"2024-04-25T05:51:15.053960+0000","last_fullsized":"2024-04-25T05:51:15.053960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.109246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.109246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.734569+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060903+0000","last_change":"2024-04-25T05:50:49.427823+0000","last_active":"2024-04-25T05:51:15.060903+0000","last_peered":"2024-04-25T05:51:15.060903+0000","last_clean":"2024-04-25T05:51:15.060903+0000","last_became_active":"2024-04-25T05:46:34.970227+0000","last_became_peered":"2024-04-25T05:46:34.970227+0000","last_unstale":"2024-04-25T05:51:15.060903+0000","last_undegraded":"2024-04-25T05:51:15.060903+0000","last_fullsized":"2024-04-25T05:51:15.060903+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.080828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":225,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058132+0000","last_change":"2024-04-25T05:50:02.734274+0000","last_active":"2024-04-25T05:51:15.058132+0000","last_peered":"2024-04-25T05:51:15.058132+0000","last_clean":"2024-04-25T05:51:15.058132+0000","last_became_active":"2024-04-25T05:46:34.974106+0000","last_became_peered":"2024-04-25T05:46:34.974106+0000","last_unstale":"2024-04-25T05:51:15.058132+0000","last_undegraded":"2024-04-25T05:51:15.058132+0000","last_fullsized":"2024-04-25T05:51:15.058132+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:02.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:02.734157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":976,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.070077+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:51:15.070077+0000","last_peered":"2024-04-25T05:51:15.070077+0000","last_clean":"2024-04-25T05:51:15.070077+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:51:15.070077+0000","last_undegraded":"2024-04-25T05:51:15.070077+0000","last_fullsized":"2024-04-25T05:51:15.070077+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:20.599614+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":224,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059103+0000","last_change":"2024-04-25T05:50:05.705768+0000","last_active":"2024-04-25T05:51:15.059103+0000","last_peered":"2024-04-25T05:51:15.059103+0000","last_clean":"2024-04-25T05:51:15.059103+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:51:15.059103+0000","last_undegraded":"2024-04-25T05:51:15.059103+0000","last_fullsized":"2024-04-25T05:51:15.059103+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.705647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.705647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:34.259931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053582+0000","last_change":"2024-04-25T05:47:20.854929+0000","last_active":"2024-04-25T05:51:15.053582+0000","last_peered":"2024-04-25T05:51:15.053582+0000","last_clean":"2024-04-25T05:51:15.053582+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:51:15.053582+0000","last_undegraded":"2024-04-25T05:51:15.053582+0000","last_fullsized":"2024-04-25T05:51:15.053582+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:47:20.854694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:47:20.854694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":907,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061168+0000","last_change":"2024-04-25T05:49:41.202012+0000","last_active":"2024-04-25T05:51:15.061168+0000","last_peered":"2024-04-25T05:51:15.061168+0000","last_clean":"2024-04-25T05:51:15.061168+0000","last_became_active":"2024-04-25T05:46:19.986501+0000","last_became_peered":"2024-04-25T05:46:19.986501+0000","last_unstale":"2024-04-25T05:51:15.061168+0000","last_undegraded":"2024-04-25T05:51:15.061168+0000","last_fullsized":"2024-04-25T05:51:15.061168+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391263+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":915,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062286+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:51:15.062286+0000","last_peered":"2024-04-25T05:51:15.062286+0000","last_clean":"2024-04-25T05:51:15.062286+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:51:15.062286+0000","last_undegraded":"2024-04-25T05:51:15.062286+0000","last_fullsized":"2024-04-25T05:51:15.062286+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391091+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":224,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059832+0000","last_change":"2024-04-25T05:50:07.763957+0000","last_active":"2024-04-25T05:51:15.059832+0000","last_peered":"2024-04-25T05:51:15.059832+0000","last_clean":"2024-04-25T05:51:15.059832+0000","last_became_active":"2024-04-25T05:46:34.978262+0000","last_became_peered":"2024-04-25T05:46:34.978262+0000","last_unstale":"2024-04-25T05:51:15.059832+0000","last_undegraded":"2024-04-25T05:51:15.059832+0000","last_fullsized":"2024-04-25T05:51:15.059832+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:07.763810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.763810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.405363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":614,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060053+0000","last_change":"2024-04-25T05:50:14.959177+0000","last_active":"2024-04-25T05:51:15.060053+0000","last_peered":"2024-04-25T05:51:15.060053+0000","last_clean":"2024-04-25T05:51:15.060053+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:51:15.060053+0000","last_undegraded":"2024-04-25T05:51:15.060053+0000","last_fullsized":"2024-04-25T05:51:15.060053+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.959063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.959063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:44.398322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054037+0000","last_change":"2024-04-25T05:49:31.857568+0000","last_active":"2024-04-25T05:51:15.054037+0000","last_peered":"2024-04-25T05:51:15.054037+0000","last_clean":"2024-04-25T05:51:15.054037+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:51:15.054037+0000","last_undegraded":"2024-04-25T05:51:15.054037+0000","last_fullsized":"2024-04-25T05:51:15.054037+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:49:31.857452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:31.857452+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":734,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060764+0000","last_change":"2024-04-25T05:50:47.087717+0000","last_active":"2024-04-25T05:51:15.060764+0000","last_peered":"2024-04-25T05:51:15.060764+0000","last_clean":"2024-04-25T05:51:15.060764+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:51:15.060764+0000","last_undegraded":"2024-04-25T05:51:15.060764+0000","last_fullsized":"2024-04-25T05:51:15.060764+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:50:47.087575+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.087575+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.480370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":907,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062074+0000","last_change":"2024-04-25T05:48:00.472579+0000","last_active":"2024-04-25T05:51:15.062074+0000","last_peered":"2024-04-25T05:51:15.062074+0000","last_clean":"2024-04-25T05:51:15.062074+0000","last_became_active":"2024-04-25T05:46:19.997149+0000","last_became_peered":"2024-04-25T05:46:19.997149+0000","last_unstale":"2024-04-25T05:51:15.062074+0000","last_undegraded":"2024-04-25T05:51:15.062074+0000","last_fullsized":"2024-04-25T05:51:15.062074+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.390963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":508,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054238+0000","last_change":"2024-04-25T05:49:30.823791+0000","last_active":"2024-04-25T05:51:15.054238+0000","last_peered":"2024-04-25T05:51:15.054238+0000","last_clean":"2024-04-25T05:51:15.054238+0000","last_became_active":"2024-04-25T05:46:19.987367+0000","last_became_peered":"2024-04-25T05:46:19.987367+0000","last_unstale":"2024-04-25T05:51:15.054238+0000","last_undegraded":"2024-04-25T05:51:15.054238+0000","last_fullsized":"2024-04-25T05:51:15.054238+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:30.823661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:30.823661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":895,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062289+0000","last_change":"2024-04-25T05:50:17.140640+0000","last_active":"2024-04-25T05:51:15.062289+0000","last_peered":"2024-04-25T05:51:15.062289+0000","last_clean":"2024-04-25T05:51:15.062289+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:51:15.062289+0000","last_undegraded":"2024-04-25T05:51:15.062289+0000","last_fullsized":"2024-04-25T05:51:15.062289+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:50:17.140516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:32.143695+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":583,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054400+0000","last_change":"2024-04-25T05:50:48.098281+0000","last_active":"2024-04-25T05:51:15.054400+0000","last_peered":"2024-04-25T05:51:15.054400+0000","last_clean":"2024-04-25T05:51:15.054400+0000","last_became_active":"2024-04-25T05:44:28.179203+0000","last_became_peered":"2024-04-25T05:44:28.179203+0000","last_unstale":"2024-04-25T05:51:15.054400+0000","last_undegraded":"2024-04-25T05:51:15.054400+0000","last_fullsized":"2024-04-25T05:51:15.054400+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.351413+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063959+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:51:15.063959+0000","last_peered":"2024-04-25T05:51:15.063959+0000","last_clean":"2024-04-25T05:51:15.063959+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:51:15.063959+0000","last_undegraded":"2024-04-25T05:51:15.063959+0000","last_fullsized":"2024-04-25T05:51:15.063959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.209136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":767,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060516+0000","last_change":"2024-04-25T05:50:41.100007+0000","last_active":"2024-04-25T05:51:15.060516+0000","last_peered":"2024-04-25T05:51:15.060516+0000","last_clean":"2024-04-25T05:51:15.060516+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:51:15.060516+0000","last_undegraded":"2024-04-25T05:51:15.060516+0000","last_fullsized":"2024-04-25T05:51:15.060516+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:50:41.099870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.099870+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.548999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":926,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.054021+0000","last_change":"2024-04-25T05:50:04.837659+0000","last_active":"2024-04-25T05:51:15.054021+0000","last_peered":"2024-04-25T05:51:15.054021+0000","last_clean":"2024-04-25T05:51:15.054021+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:51:15.054021+0000","last_undegraded":"2024-04-25T05:51:15.054021+0000","last_fullsized":"2024-04-25T05:51:15.054021+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:50:04.837536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.837536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:30.301195+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":922,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:18.436463+0000","last_change":"2024-04-25T05:51:18.436527+0000","last_active":"2024-04-25T05:51:18.436463+0000","last_peered":"2024-04-25T05:51:18.436463+0000","last_clean":"2024-04-25T05:51:18.436463+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:51:18.436463+0000","last_undegraded":"2024-04-25T05:51:18.436463+0000","last_fullsized":"2024-04-25T05:51:18.436463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:18.436370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.436370+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:34.484305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":932,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:17.763016+0000","last_change":"2024-04-25T05:51:17.763060+0000","last_active":"2024-04-25T05:51:17.763016+0000","last_peered":"2024-04-25T05:51:17.763016+0000","last_clean":"2024-04-25T05:51:17.763016+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:51:17.763016+0000","last_undegraded":"2024-04-25T05:51:17.763016+0000","last_fullsized":"2024-04-25T05:51:17.763016+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:17.762926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:17.762926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:36.715438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":455,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064639+0000","last_change":"2024-04-25T05:50:09.708374+0000","last_active":"2024-04-25T05:51:15.064639+0000","last_peered":"2024-04-25T05:51:15.064639+0000","last_clean":"2024-04-25T05:51:15.064639+0000","last_became_active":"2024-04-25T05:49:09.550767+0000","last_became_peered":"2024-04-25T05:49:09.550767+0000","last_unstale":"2024-04-25T05:51:15.064639+0000","last_undegraded":"2024-04-25T05:51:15.064639+0000","last_fullsized":"2024-04-25T05:51:15.064639+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.708291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.708291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.813090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":896,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.062043+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:51:15.062043+0000","last_peered":"2024-04-25T05:51:15.062043+0000","last_clean":"2024-04-25T05:51:15.062043+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:51:15.062043+0000","last_undegraded":"2024-04-25T05:51:15.062043+0000","last_fullsized":"2024-04-25T05:51:15.062043+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:26.657262+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":265,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058481+0000","last_change":"2024-04-25T05:50:44.583319+0000","last_active":"2024-04-25T05:51:15.058481+0000","last_peered":"2024-04-25T05:51:15.058481+0000","last_clean":"2024-04-25T05:51:15.058481+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:51:15.058481+0000","last_undegraded":"2024-04-25T05:51:15.058481+0000","last_fullsized":"2024-04-25T05:51:15.058481+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:44.583030+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:50:44.583030+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:45.044248+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":247,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064559+0000","last_change":"2024-04-25T05:49:09.557107+0000","last_active":"2024-04-25T05:51:15.064559+0000","last_peered":"2024-04-25T05:51:15.064559+0000","last_clean":"2024-04-25T05:51:15.064559+0000","last_became_active":"2024-04-25T05:49:09.556671+0000","last_became_peered":"2024-04-25T05:49:09.556671+0000","last_unstale":"2024-04-25T05:51:15.064559+0000","last_undegraded":"2024-04-25T05:51:15.064559+0000","last_fullsized":"2024-04-25T05:51:15.064559+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:05.703233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:05.703233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":913,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.989482+0000","last_change":"2024-04-25T05:51:15.989521+0000","last_active":"2024-04-25T05:51:15.989482+0000","last_peered":"2024-04-25T05:51:15.989482+0000","last_clean":"2024-04-25T05:51:15.989482+0000","last_became_active":"2024-04-25T05:46:34.974521+0000","last_became_peered":"2024-04-25T05:46:34.974521+0000","last_unstale":"2024-04-25T05:51:15.989482+0000","last_undegraded":"2024-04-25T05:51:15.989482+0000","last_fullsized":"2024-04-25T05:51:15.989482+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.989325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.989325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:30.005203+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":920,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064229+0000","last_change":"2024-04-25T05:48:01.898718+0000","last_active":"2024-04-25T05:51:15.064229+0000","last_peered":"2024-04-25T05:51:15.064229+0000","last_clean":"2024-04-25T05:51:15.064229+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:51:15.064229+0000","last_undegraded":"2024-04-25T05:51:15.064229+0000","last_fullsized":"2024-04-25T05:51:15.064229+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.898611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.898611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811330+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":398,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064624+0000","last_change":"2024-04-25T05:49:09.554222+0000","last_active":"2024-04-25T05:51:15.064624+0000","last_peered":"2024-04-25T05:51:15.064624+0000","last_clean":"2024-04-25T05:51:15.064624+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:51:15.064624+0000","last_undegraded":"2024-04-25T05:51:15.064624+0000","last_fullsized":"2024-04-25T05:51:15.064624+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:48:36.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:36.108290+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811496+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":224,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058705+0000","last_change":"2024-04-25T05:50:50.860753+0000","last_active":"2024-04-25T05:51:15.058705+0000","last_peered":"2024-04-25T05:51:15.058705+0000","last_clean":"2024-04-25T05:51:15.058705+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:51:15.058705+0000","last_undegraded":"2024-04-25T05:51:15.058705+0000","last_fullsized":"2024-04-25T05:51:15.058705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_clean_scrub_stamp":"2024-04-25T05:50:50.860626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:18.625619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.053847+0000","last_change":"2024-04-25T05:49:35.840397+0000","last_active":"2024-04-25T05:51:15.053847+0000","last_peered":"2024-04-25T05:51:15.053847+0000","last_clean":"2024-04-25T05:51:15.053847+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:51:15.053847+0000","last_undegraded":"2024-04-25T05:51:15.053847+0000","last_fullsized":"2024-04-25T05:51:15.053847+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.840316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.840316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":216,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.057988+0000","last_change":"2024-04-25T05:48:31.213742+0000","last_active":"2024-04-25T05:51:15.057988+0000","last_peered":"2024-04-25T05:51:15.057988+0000","last_clean":"2024-04-25T05:51:15.057988+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:51:15.057988+0000","last_undegraded":"2024-04-25T05:51:15.057988+0000","last_fullsized":"2024-04-25T05:51:15.057988+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:31.213611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:31.213611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994639+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":692,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064355+0000","last_change":"2024-04-25T05:49:09.555965+0000","last_active":"2024-04-25T05:51:15.064355+0000","last_peered":"2024-04-25T05:51:15.064355+0000","last_clean":"2024-04-25T05:51:15.064355+0000","last_became_active":"2024-04-25T05:49:09.554840+0000","last_became_peered":"2024-04-25T05:49:09.554840+0000","last_unstale":"2024-04-25T05:51:15.064355+0000","last_undegraded":"2024-04-25T05:51:15.064355+0000","last_fullsized":"2024-04-25T05:51:15.064355+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.169828+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.812248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":224,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058891+0000","last_change":"2024-04-25T05:50:45.906872+0000","last_active":"2024-04-25T05:51:15.058891+0000","last_peered":"2024-04-25T05:51:15.058891+0000","last_clean":"2024-04-25T05:51:15.058891+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:51:15.058891+0000","last_undegraded":"2024-04-25T05:51:15.058891+0000","last_fullsized":"2024-04-25T05:51:15.058891+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.906734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.906734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.651537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":216,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.058870+0000","last_change":"2024-04-25T05:48:24.173416+0000","last_active":"2024-04-25T05:51:15.058870+0000","last_peered":"2024-04-25T05:51:15.058870+0000","last_clean":"2024-04-25T05:51:15.058870+0000","last_became_active":"2024-04-25T05:46:34.978586+0000","last_became_peered":"2024-04-25T05:46:34.978586+0000","last_unstale":"2024-04-25T05:51:15.058870+0000","last_undegraded":"2024-04-25T05:51:15.058870+0000","last_fullsized":"2024-04-25T05:51:15.058870+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:24.173135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:24.173135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994579+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":515,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064679+0000","last_change":"2024-04-25T05:50:08.944539+0000","last_active":"2024-04-25T05:51:15.064679+0000","last_peered":"2024-04-25T05:51:15.064679+0000","last_clean":"2024-04-25T05:51:15.064679+0000","last_became_active":"2024-04-25T05:46:19.990577+0000","last_became_peered":"2024-04-25T05:46:19.990577+0000","last_unstale":"2024-04-25T05:51:15.064679+0000","last_undegraded":"2024-04-25T05:51:15.064679+0000","last_fullsized":"2024-04-25T05:51:15.064679+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:25.210582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1180,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.070197+0000","last_change":"2024-04-25T05:50:40.381895+0000","last_active":"2024-04-25T05:51:15.070197+0000","last_peered":"2024-04-25T05:51:15.070197+0000","last_clean":"2024-04-25T05:51:15.070197+0000","last_became_active":"2024-04-25T05:46:34.972040+0000","last_became_peered":"2024-04-25T05:46:34.972040+0000","last_unstale":"2024-04-25T05:51:15.070197+0000","last_undegraded":"2024-04-25T05:51:15.070197+0000","last_fullsized":"2024-04-25T05:51:15.070197+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:02.543415+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":916,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.061204+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:51:15.061204+0000","last_peered":"2024-04-25T05:51:15.061204+0000","last_clean":"2024-04-25T05:51:15.061204+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:51:15.061204+0000","last_undegraded":"2024-04-25T05:51:15.061204+0000","last_fullsized":"2024-04-25T05:51:15.061204+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:15.391153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":453,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063133+0000","last_change":"2024-04-25T05:50:40.775527+0000","last_active":"2024-04-25T05:51:15.063133+0000","last_peered":"2024-04-25T05:51:15.063133+0000","last_clean":"2024-04-25T05:51:15.063133+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:51:15.063133+0000","last_undegraded":"2024-04-25T05:51:15.063133+0000","last_fullsized":"2024-04-25T05:51:15.063133+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:40.775406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.775406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:57.019250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":882,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.060985+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:51:15.060985+0000","last_peered":"2024-04-25T05:51:15.060985+0000","last_clean":"2024-04-25T05:51:15.060985+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:51:15.060985+0000","last_undegraded":"2024-04-25T05:51:15.060985+0000","last_fullsized":"2024-04-25T05:51:15.060985+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:19.291166+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":989,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063006+0000","last_change":"2024-04-25T05:50:43.844399+0000","last_active":"2024-04-25T05:51:15.063006+0000","last_peered":"2024-04-25T05:51:15.063006+0000","last_clean":"2024-04-25T05:51:15.063006+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:51:15.063006+0000","last_undegraded":"2024-04-25T05:51:15.063006+0000","last_fullsized":"2024-04-25T05:51:15.063006+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:43.844256+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.844256+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.303125+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":225,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.059687+0000","last_change":"2024-04-25T05:49:35.890470+0000","last_active":"2024-04-25T05:51:15.059687+0000","last_peered":"2024-04-25T05:51:15.059687+0000","last_clean":"2024-04-25T05:51:15.059687+0000","last_became_active":"2024-04-25T05:46:34.972335+0000","last_became_peered":"2024-04-25T05:46:34.972335+0000","last_unstale":"2024-04-25T05:51:15.059687+0000","last_undegraded":"2024-04-25T05:51:15.059687+0000","last_fullsized":"2024-04-25T05:51:15.059687+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.890392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.890392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:17.994815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":908,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.064054+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:51:15.064054+0000","last_peered":"2024-04-25T05:51:15.064054+0000","last_clean":"2024-04-25T05:51:15.064054+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:15.064054+0000","last_undegraded":"2024-04-25T05:51:15.064054+0000","last_fullsized":"2024-04-25T05:51:15.064054+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:16.929190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":429,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-04-25T05:51:15.063838+0000","last_change":"2024-04-25T05:49:09.559432+0000","last_active":"2024-04-25T05:51:15.063838+0000","last_peered":"2024-04-25T05:51:15.063838+0000","last_clean":"2024-04-25T05:51:15.063838+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:51:15.063838+0000","last_undegraded":"2024-04-25T05:51:15.063838+0000","last_fullsized":"2024-04-25T05:51:15.063838+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:35.227001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:48:35.227001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:16.811437+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6334513152,"num_objects":96657,"num_object_clones":0,"num_object_copies":289971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96657,"num_write_kb":6186048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19003473920,"data_stored":19003473920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71757,"ondisk_log_size":71757,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1021,"num_read_kb":1016,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":10594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070677,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3552596,"kb_used_data":2691212,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":90192556,"statfs":{"total":95995035648,"available":92357177344,"internally_reserved":0,"allocated":2755801088,"data_stored":2755188652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365017,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3821992,"kb_used_data":3082784,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":89923160,"statfs":{"total":95995035648,"available":92081315840,"internally_reserved":0,"allocated":3156770816,"data_stored":3156201852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4499872,"kb_used_data":3499672,"kb_used_omap":1,"kb_used_meta":1000190,"kb_avail":89245280,"statfs":{"total":95995035648,"available":91387166720,"internally_reserved":0,"allocated":3583664128,"data_stored":3583038048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1024195013},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411565,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3321984,"kb_used_data":2681336,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":90423168,"statfs":{"total":95995035648,"available":92593324032,"internally_reserved":0,"allocated":2745688064,"data_stored":2745094570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149927,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3839376,"kb_used_data":2707272,"kb_used_omap":1,"kb_used_meta":1132094,"kb_avail":89905776,"statfs":{"total":95995035648,"available":92063514624,"internally_reserved":0,"allocated":2772246528,"data_stored":2771637109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1159264711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986402,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2987268,"kb_used_data":2288316,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":90757884,"statfs":{"total":95995035648,"available":92936073216,"internally_reserved":0,"allocated":2343235584,"data_stored":2342638130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692062,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2137652,"kb_used_data":1538668,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91607500,"statfs":{"total":95995035648,"available":93806080000,"internally_reserved":0,"allocated":1575596032,"data_stored":1575014910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3590914048,"data_stored":3590914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2750087168,"data_stored":2750087168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2799108096,"data_stored":2799108096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2352676864,"data_stored":2352676864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1578369024,"data_stored":1578369024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3169714176,"data_stored":3169714176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2762604544,"data_stored":2762604544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:51:28.994 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-25T05:51:28.994 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-04-25T05:51:29.071 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:29.112 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 1.7 deep-scrub starts 2024-04-25T05:51:29.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 1.7 deep-scrub ok 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: pgmap v919: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 1.7 deep-scrub starts 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 1.7 deep-scrub ok 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: pgmap v919: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:51:29.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:51:29.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:29.223 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 69 16 109339 109323 98.9803 63.9375 0.0126084 0.0100973 2024-04-25T05:51:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 1.7 deep-scrub starts 2024-04-25T05:51:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 1.7 deep-scrub ok 2024-04-25T05:51:29.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:51:29.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: pgmap v919: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-04-25T05:51:29.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:51:29.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:51:29.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:29 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:51:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:29.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:29.361 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:29.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:29.478 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:29.478 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:29.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:29.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:29.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:29.693 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.699 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:29.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:29.726 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:29.800 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:29.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:29.843 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:29.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:29.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:30.073 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:30.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:30.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:30.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 4.17 scrub starts 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 4.17 scrub ok 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 11.1 scrub starts 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 1.3 deep-scrub starts 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 1.3 deep-scrub ok 2024-04-25T05:51:30.222 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: osdmap e487: 8 total, 8 up, 7 in 2024-04-25T05:51:30.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:51:30.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:51:30.223 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:30 smithi060 ceph-mon[30898]: osdmap e488: 8 total, 8 up, 7 in 2024-04-25T05:51:30.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 70 16 110371 110355 98.4878 64.5 0.0187978 0.010148 2024-04-25T05:51:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.253 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:30.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:30.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 4.17 scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 4.17 scrub ok 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 11.1 scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 1.3 deep-scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 1.3 deep-scrub ok 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: osdmap e487: 8 total, 8 up, 7 in 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:51:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[27010]: osdmap e488: 8 total, 8 up, 7 in 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 4.17 scrub starts 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 4.17 scrub ok 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 11.1 scrub starts 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 1.3 deep-scrub starts 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 1.3 deep-scrub ok 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: osdmap e487: 8 total, 8 up, 7 in 2024-04-25T05:51:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:51:30.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:51:30.394 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:30 smithi043 ceph-mon[37492]: osdmap e488: 8 total, 8 up, 7 in 2024-04-25T05:51:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:30.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:30.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:30.537 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:30.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:30.667 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:30.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:30.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.811 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:30.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:30.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:30.940 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:30.947 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:30.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:30.982 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:31.030 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:31.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:31.074 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:31.086 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:31.113 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:31.149 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pg_num to 21 2024-04-25T05:51:31.152 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:31.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:31.196 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:31.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 71 16 111421 111405 98.0251 65.625 0.0179204 0.0101968 2024-04-25T05:51:31.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:31.236 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: pgmap v921: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1332429114' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.238 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1605334599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: pgmap v921: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1332429114' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.239 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1605334599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:31.244 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:31.244 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:31.244 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:31.244 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:31.245 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:31.245 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:31.245 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:31.245 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:31.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:31.371 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:31.419 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:31.419 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:51:31.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.16413.0:111644 11.5 e489/456 11:ad5759e7:::benchmark_data_smithi043_179003_object111643:head v 489'7121, mlcod=489'7119)", 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:31.413291+0000", 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0046422349999999998, 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046741559999999996, 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413291+0000", 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413291+0000", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413292+0000", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1999999999999999e-06 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413319+0000", 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7095999999999999e-05 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.421 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413334+0000", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.417e-05 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413343+0000", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4970000000000007e-06 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413410+0000", 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7430999999999999e-05 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.413435+0000", 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4736000000000002e-05 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:111651 11.7 11:e10afdd0:::benchmark_data_smithi043_179003_object111650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:31.414812+0000", 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0031209639999999999, 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032534560000000001, 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:31.423 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 111651 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414812+0000", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:31.424 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414812+0000", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414814+0000", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.872e-06 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414846+0000", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1321e-05 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:31.425 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414863+0000", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7203000000000001e-05 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.414871+0000", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3960000000000006e-06 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415049+0000", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000177679 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:31.426 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415435+0000", 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038632400000000001 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.416187+0000", 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00075143499999999995 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.16413.0:111650 11.a e489/456 11:588be9d6:::benchmark_data_smithi043_179003_object111649:head v 489'6937, mlcod=489'6935)", 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:31.415791+0000", 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0021420889999999998, 2024-04-25T05:51:31.427 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00236075, 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415791+0000", 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415791+0000", 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415792+0000", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5700000000000002e-07 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415826+0000", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3676000000000001e-05 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415843+0000", 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7041000000000002e-05 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.429 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415856+0000", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2959000000000001e-05 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415931+0000", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4851999999999994e-05 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.415964+0000", 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3623e-05 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:31.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:111655 11.7 11.550ec447 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:31.417183+0000", 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.00075060600000000002, 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001024297, 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "queued for pg", 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 111655 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.431 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417183+0000", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417183+0000", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417184+0000", 2024-04-25T05:51:31.432 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.576e-06 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417227+0000", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2741999999999998e-05 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417242+0000", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5231999999999999e-05 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:31.417250+0000", 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1929999999999993e-06 2024-04-25T05:51:31.433 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 4 2024-04-25T05:51:31.434 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:31.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:51:31.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:51:31.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: pgmap v921: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-25T05:51:31.434 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1332429114' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-04-25T05:51:31.435 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1605334599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:31.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:31.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:51:31.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:31.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:31.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:31.898 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:31.933 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:32.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:32.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:32.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:32.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.211 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:32.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 72 16 112684 112668 97.76 78.9375 0.00882945 0.0102247 2024-04-25T05:51:32.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:32.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 5.17 scrub starts 2024-04-25T05:51:32.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 5.17 scrub ok 2024-04-25T05:51:32.358 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 11.1 scrub ok 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: osdmap e489: 8 total, 8 up, 7 in 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:32.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 5.17 scrub starts 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 5.17 scrub ok 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 11.1 scrub ok 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: osdmap e489: 8 total, 8 up, 7 in 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:51:32.360 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:32.367 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 5.17 scrub starts 2024-04-25T05:51:32.367 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 5.17 scrub ok 2024-04-25T05:51:32.367 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:51:32.367 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 11.1 scrub ok 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: osdmap e489: 8 total, 8 up, 7 in 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:51:32.368 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:32.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:51:32.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:32.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:32.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:32.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:32.777 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:32.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:32.887 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:32.941 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:32.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:32.945 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:33.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.090 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:33.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:33.191 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:33.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 73 16 114017 114001 97.562 83.3125 0.0145904 0.0102453 2024-04-25T05:51:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:33.329 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:33.363 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: 6.19 deep-scrub starts 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: 6.19 deep-scrub ok 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: pgmap v924: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 983 op/s 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-25T05:51:33.365 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-mon[30898]: osdmap e490: 8 total, 8 up, 7 in 2024-04-25T05:51:33.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: 6.19 deep-scrub starts 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: 6.19 deep-scrub ok 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: pgmap v924: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 983 op/s 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-25T05:51:33.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[27010]: osdmap e490: 8 total, 8 up, 7 in 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: 6.19 deep-scrub starts 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: 6.19 deep-scrub ok 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: pgmap v924: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 983 op/s 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-25T05:51:33.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:33 smithi043 ceph-mon[37492]: osdmap e490: 8 total, 8 up, 7 in 2024-04-25T05:51:33.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:33.574 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:33.574 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:51:33.574 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:51:33.574 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:51:33.574 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:33.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:33.749 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.789 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:33.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:33.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:33.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:34.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:34.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:34.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:34.200 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:34.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 74 16 115219 115203 97.2588 75.125 0.00419552 0.0102774 2024-04-25T05:51:34.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:34.334 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:34.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:51:34.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: 4.2 scrub starts 2024-04-25T05:51:34.347 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: 4.2 scrub ok 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: osdmap e491: 8 total, 8 up, 7 in 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: 11.d scrub starts 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:51:34.348 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:34.436 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:34.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:34.487 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: 4.2 scrub starts 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: 4.2 scrub ok 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: osdmap e491: 8 total, 8 up, 7 in 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:34.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: 11.d scrub starts 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: 4.2 scrub starts 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: 4.2 scrub ok 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: osdmap e491: 8 total, 8 up, 7 in 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: 11.d scrub starts 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:51:34.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:51:34.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:51:34.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-04-25T05:51:34.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.611 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:34.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:34.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:34.758 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:34.802 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:34.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:34.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:51:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:34.898 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:34.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:34.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:35.032 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:35.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:35.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:35.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 75 16 116477 116461 97.0097 78.625 0.00744985 0.0103038 2024-04-25T05:51:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:51:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:51:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.7 deep-scrub starts 2024-04-25T05:51:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.7 deep-scrub ok 2024-04-25T05:51:35.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: pgmap v927: 168 pgs: 1 activating, 4 unknown, 163 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 921 op/s 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.2 deep-scrub starts 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: 1.2 deep-scrub ok 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: osdmap e492: 8 total, 8 up, 7 in 2024-04-25T05:51:35.250 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:35 smithi060 ceph-mon[30898]: osdmap e493: 8 total, 8 up, 7 in 2024-04-25T05:51:35.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:35.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:35.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:51:35.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:51:35.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.7 deep-scrub starts 2024-04-25T05:51:35.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.7 deep-scrub ok 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: pgmap v927: 168 pgs: 1 activating, 4 unknown, 163 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 921 op/s 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.2 deep-scrub starts 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: 1.2 deep-scrub ok 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: osdmap e492: 8 total, 8 up, 7 in 2024-04-25T05:51:35.440 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[27010]: osdmap e493: 8 total, 8 up, 7 in 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.7 deep-scrub starts 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.7 deep-scrub ok 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:51:35.441 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: pgmap v927: 168 pgs: 1 activating, 4 unknown, 163 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 921 op/s 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.2 deep-scrub starts 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: 1.2 deep-scrub ok 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2991467882' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: osdmap e492: 8 total, 8 up, 7 in 2024-04-25T05:51:35.442 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:35 smithi043 ceph-mon[37492]: osdmap e493: 8 total, 8 up, 7 in 2024-04-25T05:51:35.471 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:35.562 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:35.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:35.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:35.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:35.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:35.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:35.891 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:35.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226334 8.8 8:1120fdcc:::benchmark_data_smithi043_104001_object102210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.04511289300001, 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.29533898800000002, 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.912 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226334 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863429+0000", 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863431+0000", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4980000000000001e-06 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863437+0000", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8420000000000001e-06 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863450+0000", 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3424999999999999e-05 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863459+0000", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2129999999999997e-06 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863579+0000", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119181 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863691+0000", 2024-04-25T05:51:35.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011263400000000001 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.863798+0000", 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010700300000000001 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.867039+0000", 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032403499999999999 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.867308+0000", 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00026967000000000001 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158726+0000", 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.29141812900000003 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158768+0000", 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2042999999999997e-05 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226340 8.8 8:1cf7c54f:::benchmark_data_smithi043_104001_object102216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.03739132099997, 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.287807596, 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226340 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871151+0000", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871152+0000", 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3260000000000001e-06 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871159+0000", 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9240000000000001e-06 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871168+0000", 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0510000000000002e-06 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.871175+0000", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3489999999999997e-06 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872018+0000", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00084366499999999995 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872171+0000", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015301600000000001 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.872311+0000", 2024-04-25T05:51:35.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013923900000000001 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.876118+0000", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038069760000000001 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881212+0000", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050944299999999996 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158899+0000", 2024-04-25T05:51:35.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27768725900000002 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158959+0000", 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9361000000000003e-05 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226345 8.4 8:291622b3:::benchmark_data_smithi043_104001_object102221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.03055983399997, 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27591954099999999, 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226345 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877982+0000", 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877983+0000", 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.13e-07 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877986+0000", 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4690000000000002e-06 2024-04-25T05:51:35.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.877994+0000", 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4699999999999996e-06 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878001+0000", 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6719999999999998e-06 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878398+0000", 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039738700000000002 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878515+0000", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117169 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878633+0000", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117488 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881439+0000", 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002806092 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.885451+0000", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040120099999999999 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.153884+0000", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26843287300000002 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.153902+0000", 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8397999999999999e-05 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226346 8.8 8:1dfd0933:::benchmark_data_smithi043_104001_object102222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.030523958, 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.28112287200000002, 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226346 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878018+0000", 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878019+0000", 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6600000000000002e-07 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878022+0000", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0680000000000001e-06 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878028+0000", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6960000000000002e-06 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878033+0000", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2190000000000003e-06 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878127+0000", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4475000000000006e-05 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878237+0000", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109503 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.878344+0000", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107265 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.881455+0000", 2024-04-25T05:51:35.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031110550000000002 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.882740+0000", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001284997 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159099+0000", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27635901899999998 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159141+0000", 2024-04-25T05:51:35.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2109e-05 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226334 8.8 8:1120fdcc:::benchmark_data_smithi043_104001_object102210:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.01472929700003, 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1173450709999999, 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226334 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.935 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893813+0000", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893815+0000", 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8640000000000001e-06 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893824+0000", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1400000000000006e-06 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893845+0000", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0873000000000002e-05 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893856+0000", 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0793e-05 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.937 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897028+0000", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003172663 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op must wait for map", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897034+0000", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0299999999999999e-06 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159254+0000", 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.262219751 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.938 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.159279+0000", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.4542e-05 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011075+0000", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85179589600000005 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.011158+0000", 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3518999999999998e-05 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226340 8.18 8:1cf7c54f:::benchmark_data_smithi043_104001_object102216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.014660628, 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1499044039999999, 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226340 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.940 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893882+0000", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2399999999999999e-07 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893885+0000", 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.83e-06 2024-04-25T05:51:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893899+0000", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4236000000000001e-05 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.893905+0000", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.139e-06 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158402+0000", 2024-04-25T05:51:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26449650899999999 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158430+0000", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8666999999999998e-05 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.039590+0000", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88115992700000001 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.039724+0000", 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013313499999999999 2024-04-25T05:51:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.043720+0000", 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039966180000000004 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.043786+0000", 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5919000000000004e-05 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226345 8.14 8:291622b3:::benchmark_data_smithi043_104001_object102221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.01150735399995, 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1601359410000001, 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226345 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897035+0000", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897036+0000", 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.393e-06 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897044+0000", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5669999999999997e-06 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897056+0000", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1652e-05 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897064+0000", 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.156521+0000", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.25945676899999998 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.156532+0000", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1039999999999999e-05 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051043+0000", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.894510994 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051153+0000", 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109852 2024-04-25T05:51:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.057096+0000", 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0059431110000000001 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.057171+0000", 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4857999999999994e-05 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226346 8.18 8:1dfd0933:::benchmark_data_smithi043_104001_object102222:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 536.01145405399996, 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.147144165, 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226346 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897088+0000", 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897089+0000", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3000000000000002e-07 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897091+0000", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3410000000000001e-06 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897095+0000", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0110000000000002e-06 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:39.897100+0000", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7990000000000001e-06 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158558+0000", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26145807100000001 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.158581+0000", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3422e-05 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.040060+0000", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88147867000000002 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for missing object", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.040209+0000", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000148942 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.044181+0000", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003972382 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.044232+0000", 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1196999999999998e-05 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:226358 8.4 8:2507d7e8:::benchmark_data_smithi043_104001_object102234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 535.72571569499996, 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87045504500000004, 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.955 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226358 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182827+0000", 2024-04-25T05:51:35.956 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182828+0000", 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6029999999999999e-06 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.182834+0000", 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0700000000000003e-06 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183038+0000", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020412100000000001 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183054+0000", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5245999999999999e-05 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183127+0000", 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3009000000000005e-05 2024-04-25T05:51:35.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:40.183135+0000", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5830000000000007e-06 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020562+0000", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.83742649899999999 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.020696+0000", 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013470499999999999 2024-04-25T05:51:35.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.021014+0000", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00031797900000000003 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051256+0000", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.030241160999999999 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.051290+0000", 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4397000000000003e-05 2024-04-25T05:51:35.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053198+0000", 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019084589999999999 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053241+0000", 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2534000000000003e-05 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.961 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.053282+0000", 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0679000000000001e-05 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228015 8.11 8:8aa3395b:::benchmark_data_smithi043_104001_object103891:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 533.93112092199999, 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.126170712, 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.962 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228015 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977421+0000", 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977424+0000", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1619999999999998e-06 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977433+0000", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9469999999999992e-06 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977444+0000", 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1599e-05 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.964 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.977454+0000", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7939999999999997e-06 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.985631+0000", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0081774259999999998 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.985640+0000", 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1710000000000008e-06 2024-04-25T05:51:35.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981642+0000", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99600169299999997 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981656+0000", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3356999999999999e-05 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103494+0000", 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.121838624 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.966 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103592+0000", 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7938999999999999e-05 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228038 8.11 8:8a1cbb27:::benchmark_data_smithi043_104001_object103914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 533.91753735500004, 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1240071120000001, 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:35.967 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228038 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991005+0000", 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991006+0000", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9700000000000005e-07 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991010+0000", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4769999999999997e-06 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991023+0000", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2221999999999999e-05 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991028+0000", 2024-04-25T05:51:35.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1529999999999996e-06 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991086+0000", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8495999999999997e-05 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.991096+0000", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0159e-05 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981686+0000", 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99058917800000001 2024-04-25T05:51:35.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981695+0000", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1989999999999995e-06 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104063+0000", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1223681910000001 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104174+0000", 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000110653 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,1,6,7", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104398+0000", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000224319 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.106906+0000", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0025081190000000001 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.107279+0000", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000372755 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.110458+0000", 2024-04-25T05:51:35.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031791749999999998 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.110501+0000", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2617000000000002e-05 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.114824+0000", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043234880000000003 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.114882+0000", 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7948000000000002e-05 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.115012+0000", 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000130066 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228061 8.19 8:9a04b146:::benchmark_data_smithi043_104001_object103937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 533.89171575600005, 2024-04-25T05:51:35.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.087389859, 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228061 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016827+0000", 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016829+0000", 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9870000000000002e-06 2024-04-25T05:51:35.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016838+0000", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2299999999999997e-06 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016865+0000", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7203e-05 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.016887+0000", 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2500000000000001e-05 2024-04-25T05:51:35.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.017005+0000", 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011742499999999999 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.017012+0000", 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0500000000000003e-06 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981886+0000", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96487387700000005 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981897+0000", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1289e-05 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.074722+0000", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.092824963 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.074853+0000", 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000130925 2024-04-25T05:51:35.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,4,6", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.075106+0000", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025335100000000002 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103590+0000", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.028484112999999998 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103829+0000", 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023812599999999999 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103869+0000", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0166000000000001e-05 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.103969+0000", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010041500000000001 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104099+0000", 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012951 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.981 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:44.104216+0000", 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117729 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:224215 10.9 10:9299bc49:::benchmark_data_smithi043_152558_object95865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 123.769271418, 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.21245112399999999, 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.982 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 224215 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139272+0000", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.248e-06 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139278+0000", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8490000000000002e-06 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139294+0000", 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6143000000000001e-05 2024-04-25T05:51:35.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139302+0000", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2830000000000001e-06 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139363+0000", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0409999999999999e-05 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139498+0000", 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013544100000000001 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139614+0000", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115869 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144295+0000", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046808809999999996 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.145168+0000", 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00087276499999999996 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.351707+0000", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.206538738 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.351722+0000", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5497000000000001e-05 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:224217 10.6 10:6c501e56:::benchmark_data_smithi043_152558_object95867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 123.767117918, 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.22087231800000001, 2024-04-25T05:51:35.987 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 224217 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:51:35.988 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141426+0000", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.139e-06 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141431+0000", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5419999999999996e-06 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141442+0000", 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0774000000000001e-05 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141450+0000", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7030000000000006e-06 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144619+0000", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031695970000000001 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144806+0000", 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018734599999999999 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144937+0000", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013011199999999999 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.160892+0000", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015955882000000001 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.362282+0000", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.201389543 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.362297+0000", 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.468e-05 2024-04-25T05:51:35.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:87349 11.7 11:e68f8836:::benchmark_data_smithi043_179003_object87348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:11.112066+0000", 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 24.796476109, 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.027999512000000001, 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 87349 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.992 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112066+0000", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112066+0000", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112068+0000", 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.567e-06 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112109+0000", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1491e-05 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112121+0000", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1413999999999999e-05 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112130+0000", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0580000000000004e-06 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:35.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112205+0000", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5601999999999999e-05 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112333+0000", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000128038 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.112454+0000", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121076 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.114798+0000", 2024-04-25T05:51:35.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023431289999999998 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.115202+0000", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000404688 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.139950+0000", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.024747906 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.140021+0000", 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0852999999999994e-05 2024-04-25T05:51:35.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:11.140066+0000", 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4690000000000001e-05 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:35.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:89220 11.7 11:e7559d72:::benchmark_data_smithi043_179003_object89219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:12.490484+0000", 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 23.418058330000001, 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.032532621999999997, 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:35.998 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 89220 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490484+0000", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490484+0000", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490485+0000", 2024-04-25T05:51:35.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.203e-06 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490531+0000", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5448000000000003e-05 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490547+0000", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6322999999999999e-05 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490556+0000", 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0669999999999998e-06 2024-04-25T05:51:36.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490609+0000", 2024-04-25T05:51:36.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3313000000000003e-05 2024-04-25T05:51:36.001 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490738+0000", 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012862400000000001 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.490879+0000", 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014113299999999999 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.496231+0000", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053515000000000004 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.496276+0000", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5528999999999997e-05 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.522919+0000", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.026642441999999999 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:36.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.522981+0000", 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2732999999999997e-05 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.523017+0000", 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5306999999999999e-05 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:89221 11.7 11:ebe1bcd6:::benchmark_data_smithi043_179003_object89220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:12.492809+0000", 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 23.415732954999999, 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.033942057999999997, 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:36.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 89221 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492809+0000", 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492809+0000", 2024-04-25T05:51:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492810+0000", 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9699999999999994e-07 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492862+0000", 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1950000000000002e-05 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492872+0000", 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.859e-06 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:36.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.492878+0000", 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2999999999999998e-06 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.493343+0000", 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000464747 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.493496+0000", 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015269799999999999 2024-04-25T05:51:36.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.493682+0000", 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000186102 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.496379+0000", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026969429999999998 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.496391+0000", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1559999999999999e-05 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.526658+0000", 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.030267591999999999 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.526701+0000", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3290000000000001e-05 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:12.526751+0000", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0019999999999999e-05 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:93339 11.7 11:e5f08ea9:::benchmark_data_smithi043_179003_object93338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:15.767629+0000", 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 20.140913472000001, 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.032309929000000001, 2024-04-25T05:51:36.012 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 93339 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767629+0000", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767629+0000", 2024-04-25T05:51:36.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767629+0000", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.58e-07 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767651+0000", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1878e-05 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767658+0000", 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.584e-06 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.767662+0000", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3220000000000003e-06 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.770153+0000", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024907269999999999 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.770296+0000", 2024-04-25T05:51:36.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014274099999999999 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.770702+0000", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00040668099999999999 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.774208+0000", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035061020000000001 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.775020+0000", 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00081200399999999996 2024-04-25T05:51:36.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.799837+0000", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.024816990000000001 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.799894+0000", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7046999999999998e-05 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:15.799939+0000", 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4394999999999998e-05 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.017 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:94956 11.7 11:e9dc5c97:::benchmark_data_smithi043_179003_object94955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:17.150075+0000", 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.758466897000002, 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.035530640000000002, 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:36.018 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 94956 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150075+0000", 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150075+0000", 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:36.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150076+0000", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3499999999999996e-07 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150106+0000", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9810000000000001e-05 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150120+0000", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4375999999999999e-05 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.150130+0000", 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.798e-06 2024-04-25T05:51:36.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.153610+0000", 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003480342 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.153767+0000", 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000156828 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.153927+0000", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016030999999999999 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157037+0000", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031098020000000001 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.161915+0000", 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048773369999999998 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.185514+0000", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.02359909 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.185575+0000", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1118000000000007e-05 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.185606+0000", 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1294000000000002e-05 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.025 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:94968 11.7 11:ed5caa95:::benchmark_data_smithi043_179003_object94967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:17.157273+0000", 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 18.751269313000002, 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.031965777000000001, 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 94968 2024-04-25T05:51:36.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157273+0000", 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157273+0000", 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157274+0000", 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.19e-06 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:36.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157302+0000", 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8011999999999999e-05 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157314+0000", 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1995e-05 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157322+0000", 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3900000000000004e-06 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157487+0000", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016588200000000001 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157621+0000", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013398800000000001 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.157774+0000", 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015211700000000001 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.161690+0000", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039164819999999998 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.162174+0000", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048389599999999999 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.189136+0000", 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.026961855 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.189193+0000", 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7170999999999999e-05 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:17.189239+0000", 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5799e-05 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:36.032 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:36.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:36.170 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:36.201 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:36.224 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 76 16 117768 117752 96.795 80.6875 0.0104222 0.0103265 2024-04-25T05:51:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:36 smithi060 ceph-mon[30898]: 1.14 deep-scrub starts 2024-04-25T05:51:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:36 smithi060 ceph-mon[30898]: 1.14 deep-scrub ok 2024-04-25T05:51:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:36 smithi060 ceph-mon[30898]: 1.f scrub starts 2024-04-25T05:51:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:36 smithi060 ceph-mon[30898]: 1.f scrub ok 2024-04-25T05:51:36.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:36 smithi060 ceph-mon[30898]: osdmap e494: 8 total, 8 up, 7 in 2024-04-25T05:51:36.295 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:36.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:36.336 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:36.337 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.337 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[27010]: 1.14 deep-scrub starts 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[27010]: 1.14 deep-scrub ok 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[27010]: 1.f scrub starts 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[27010]: 1.f scrub ok 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[27010]: osdmap e494: 8 total, 8 up, 7 in 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[37492]: 1.14 deep-scrub starts 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[37492]: 1.14 deep-scrub ok 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[37492]: 1.f scrub starts 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[37492]: 1.f scrub ok 2024-04-25T05:51:36.338 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:36 smithi043 ceph-mon[37492]: osdmap e494: 8 total, 8 up, 7 in 2024-04-25T05:51:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:36.376 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:36.412 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:36.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:36.456 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:36.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:36.496 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:36.540 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:36.581 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:36.589 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:36.598 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:36.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:36.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:36.905 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:36.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:37.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:37.029 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:51:37.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:51:37.029 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-25T05:51:37.029 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd in 0 2024-04-25T05:51:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:37.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:37.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:37.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:51:37.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 77 16 118977 118961 96.5192 75.5625 0.0133275 0.0103559 2024-04-25T05:51:37.297 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:37.370 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[27010]: pgmap v930: 168 pgs: 1 activating, 167 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3021323022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[27010]: osdmap e495: 8 total, 8 up, 7 in 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[37492]: pgmap v930: 168 pgs: 1 activating, 167 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-04-25T05:51:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3021323022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:37.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:51:37.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:37 smithi043 ceph-mon[37492]: osdmap e495: 8 total, 8 up, 7 in 2024-04-25T05:51:37.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:51:37.472 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:37 smithi060 ceph-mon[30898]: pgmap v930: 168 pgs: 1 activating, 167 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-04-25T05:51:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3021323022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:37 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:51:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:37 smithi060 ceph-mon[30898]: osdmap e495: 8 total, 8 up, 7 in 2024-04-25T05:51:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.500 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:37.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:37.601 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:37.611 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:37.631 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.659 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:37.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:37.760 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:37.789 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:37.891 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:37.928 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:37.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:38.029 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.068 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:38.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:38.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:38.204 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:38.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 78 16 120155 120139 96.2257 73.625 0.0120381 0.0103876 2024-04-25T05:51:38.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:38.309 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:38.310 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.357 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:38.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:38.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:51:38.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:51:38.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:38.371 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:38 smithi060 ceph-mon[30898]: osdmap e496: 8 total, 8 up, 7 in 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[27010]: osdmap e496: 8 total, 8 up, 7 in 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:38.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-mon[37492]: osdmap e496: 8 total, 8 up, 7 in 2024-04-25T05:51:38.424 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:38.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.490 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:38.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:38.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:38.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:38.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:38.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:38 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:51:38.738+0000 7f173ec68700 -1 mon.a@0(leader).osd e496 definitely_dead 0 2024-04-25T05:51:38.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:38.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:39.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:39.050 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:39.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:51:39.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:39.200 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:marked in osd.0. 2024-04-25T05:51:39.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 79 16 121417 121401 96.0061 78.875 0.0137414 0.0104092 2024-04-25T05:51:39.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:39.322 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:39.410 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:39.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:39.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:51:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:39 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:39 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:39 smithi060 ceph-mon[30898]: pgmap v933: 168 pgs: 1 peering, 1 activating, 166 active+clean; 7.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s 2024-04-25T05:51:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2404752727' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:39 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:39.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:39.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[27010]: pgmap v933: 168 pgs: 1 peering, 1 activating, 166 active+clean; 7.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s 2024-04-25T05:51:39.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2404752727' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.502 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:39.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:39.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[37492]: pgmap v933: 168 pgs: 1 peering, 1 activating, 166 active+clean; 7.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s 2024-04-25T05:51:39.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2404752727' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.503 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-25T05:51:39.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:39.569 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:39.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:39.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:39.814 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:39.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:39.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:39.936 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:39.936 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:39.936 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:39.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:40.038 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:40.064 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-25T05:51:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:40.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:40.075 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:51:40.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.188 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:40.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:40.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:2024-04-25T05:51:40.226360+0000 min lat: 0.00134748 max lat: 0.2399 avg lat: 0.0104087 2024-04-25T05:51:40.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T05:51:40.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 80 16 121450 121434 94.8321 2.0625 0.00303849 0.0104087 2024-04-25T05:51:40.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.311 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:40.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:51:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: osdmap e497: 8 total, 8 up, 8 in 2024-04-25T05:51:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3919243588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[37492]: osdmap e498: 8 total, 8 up, 8 in 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: osdmap e497: 8 total, 8 up, 8 in 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3919243588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:40 smithi043 ceph-mon[27010]: osdmap e498: 8 total, 8 up, 8 in 2024-04-25T05:51:40.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.434 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:40.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: osdmap e497: 8 total, 8 up, 8 in 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3919243588' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:40 smithi060 ceph-mon[30898]: osdmap e498: 8 total, 8 up, 8 in 2024-04-25T05:51:40.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:40.556 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:40.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:40.670 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:51:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.680 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:40.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:40.782 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.804 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:40.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:40.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:40.928 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:40.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:40.929 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:41.030 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:41.056 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:41.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:41.180 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:41.225 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 81 16 121450 121434 93.6617 0 - 0.0104087 2024-04-25T05:51:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:41.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:41.304 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:41.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:41 smithi043 ceph-mon[27010]: pgmap v936: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 27 KiB/s, 0 objects/s recovering 2024-04-25T05:51:41.373 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:41 smithi043 ceph-mon[27010]: osdmap e499: 8 total, 8 up, 8 in 2024-04-25T05:51:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:41 smithi043 ceph-mon[37492]: pgmap v936: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 27 KiB/s, 0 objects/s recovering 2024-04-25T05:51:41.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:41 smithi043 ceph-mon[37492]: osdmap e499: 8 total, 8 up, 8 in 2024-04-25T05:51:41.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.430 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:41.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:41.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:41 smithi060 ceph-mon[30898]: pgmap v936: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 27 KiB/s, 0 objects/s recovering 2024-04-25T05:51:41.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:41 smithi060 ceph-mon[30898]: osdmap e499: 8 total, 8 up, 8 in 2024-04-25T05:51:41.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:41.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:41.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:41.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:41.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:41.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:41.869 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:41.869 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:41.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:41.977 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:41.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:42.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:42.107 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:42.209 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:42.227 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 82 16 122291 122275 93.1584 26.2812 0.0225932 0.0107284 2024-04-25T05:51:42.257 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:42.359 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:42.371 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[37492]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 496'7597 2024-04-25T05:51:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[37492]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7581 2024-04-25T05:51:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[37492]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7443 2024-04-25T05:51:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[37492]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7694 2024-04-25T05:51:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[37492]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7678 2024-04-25T05:51:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[27010]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 496'7597 2024-04-25T05:51:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[27010]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7581 2024-04-25T05:51:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[27010]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7443 2024-04-25T05:51:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[27010]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7694 2024-04-25T05:51:42.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:42 smithi043 ceph-mon[27010]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7678 2024-04-25T05:51:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:42.412 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:42.454 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:42 smithi060 ceph-mon[30898]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 496'7597 2024-04-25T05:51:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:42 smithi060 ceph-mon[30898]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7581 2024-04-25T05:51:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:42 smithi060 ceph-mon[30898]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7443 2024-04-25T05:51:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:42 smithi060 ceph-mon[30898]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7694 2024-04-25T05:51:42.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:42 smithi060 ceph-mon[30898]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 496'7678 2024-04-25T05:51:42.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:42.504 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:42.536 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:42.546 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:42.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:42.631 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:42.638 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:42.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:51:42.674 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:42.676 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:42.682 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:42.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:42.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:42.947 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:42.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:43.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:43.059 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:43.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:43.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:51:43.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:43.228 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 83 16 123373 123357 92.8493 67.625 0.0106215 0.0107657 2024-04-25T05:51:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:43.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:43.361 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:43.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[37492]: pgmap v939: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:51:43.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[37492]: osdmap e500: 8 total, 8 up, 8 in 2024-04-25T05:51:43.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2052096509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[27010]: pgmap v939: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:51:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[27010]: osdmap e500: 8 total, 8 up, 8 in 2024-04-25T05:51:43.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2052096509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:43.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:43.493 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:43 smithi060 ceph-mon[30898]: pgmap v939: 168 pgs: 1 peering, 167 active+clean; 7.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 80 KiB/s, 0 objects/s recovering 2024-04-25T05:51:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:43 smithi060 ceph-mon[30898]: osdmap e500: 8 total, 8 up, 8 in 2024-04-25T05:51:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2052096509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:43.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:43.623 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:43.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:43.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:43.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:43.907 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:43.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:43.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:44.008 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:44.066 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:44.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.204 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:44.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:44.228 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 84 16 124365 124349 92.4821 62 0.0113686 0.0108079 2024-04-25T05:51:44.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:44.333 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:44.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:44.481 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:44 smithi060 ceph-mon[30898]: osdmap e501: 8 total, 8 up, 8 in 2024-04-25T05:51:44.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:44.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:44.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:44 smithi043 ceph-mon[37492]: osdmap e501: 8 total, 8 up, 8 in 2024-04-25T05:51:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:44 smithi043 ceph-mon[27010]: osdmap e501: 8 total, 8 up, 8 in 2024-04-25T05:51:44.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:44.749 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:44.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:44.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:44.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:44.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:44.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:45.064 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:51:45.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:51:45.064 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-25T05:51:45.064 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T05:51:45.065 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T05:51:45.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-25T05:51:45.077 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:45.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.358 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:45.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:45.359 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 85 16 125380 125364 92.1405 63.4375 0.0129822 0.0108484 2024-04-25T05:51:45.359 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:125498 11.4 11:294e40f2:::benchmark_data_smithi043_179003_object125497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:45.353296+0000", 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.001362002, 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0014011749999999999, 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:45.360 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 125498 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353296+0000", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353296+0000", 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:45.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353297+0000", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2299999999999999e-07 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353347+0000", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9574999999999999e-05 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353356+0000", 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4029999999999996e-06 2024-04-25T05:51:45.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353364+0000", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7680000000000008e-06 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353422+0000", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7707999999999997e-05 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353655+0000", 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023272599999999999 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.363 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.353833+0000", 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017859899999999999 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(client.16413.0:125492 11.0 e503/456 11:0d6a3ed8:::benchmark_data_smithi043_179003_object125491:head v 503'7796, mlcod=502'7792)", 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:51:45.354048+0000", 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.00061039000000000004, 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000757842, 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:51:45.364 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354048+0000", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354048+0000", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354048+0000", 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4200000000000001e-07 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354078+0000", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9825999999999999e-05 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354087+0000", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7809999999999994e-06 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354093+0000", 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9819999999999996e-06 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.366 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354160+0000", 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6710000000000003e-05 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:51:45.354179+0000", 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9236e-05 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 2 2024-04-25T05:51:45.367 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:45.460 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.496 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:45.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:45.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:45.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:51:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.635 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:45.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: pgmap v942: 168 pgs: 2 active+remapped+backfilling, 1 peering, 165 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 452 op/s; 15012/367716 objects misplaced (4.082%); 7.2 MiB/s, 118 objects/s recovering 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: osdmap e502: 8 total, 8 up, 8 in 2024-04-25T05:51:45.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: pgmap v942: 168 pgs: 2 active+remapped+backfilling, 1 peering, 165 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 452 op/s; 15012/367716 objects misplaced (4.082%); 7.2 MiB/s, 118 objects/s recovering 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: osdmap e502: 8 total, 8 up, 8 in 2024-04-25T05:51:45.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:45.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: pgmap v942: 168 pgs: 2 active+remapped+backfilling, 1 peering, 165 active+clean; 7.5 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 452 op/s; 15012/367716 objects misplaced (4.082%); 7.2 MiB/s, 118 objects/s recovering 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: osdmap e502: 8 total, 8 up, 8 in 2024-04-25T05:51:45.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:51:45.766 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:45.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:45.908 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:45.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:46.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:46.071 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:46.173 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:46.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:46.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 86 16 126407 126391 91.8155 64.1875 0.0103161 0.0108865 2024-04-25T05:51:46.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:51:46.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:46.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:46.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:46.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:46.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:46 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:46.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:46 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:46.494 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:46 smithi060 ceph-mon[30898]: osdmap e503: 8 total, 8 up, 8 in 2024-04-25T05:51:46.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:46.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:46.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[27010]: osdmap e503: 8 total, 8 up, 8 in 2024-04-25T05:51:46.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:51:46.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2446798120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:51:46.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:46 smithi043 ceph-mon[37492]: osdmap e503: 8 total, 8 up, 8 in 2024-04-25T05:51:46.721 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:46.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:46.886 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:46.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:46.987 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:47.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:47.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.165 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:47.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:47.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 87 16 127348 127332 91.4362 58.8125 0.00970037 0.0109314 2024-04-25T05:51:47.266 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:47.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:47.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:47.448 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:47.457 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:47.458 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "" 2024-04-25T05:51:47.458 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.463 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:47.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:47 smithi060 ceph-mon[30898]: pgmap v945: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s; 37965/371946 objects misplaced (10.207%); 12 MiB/s, 203 objects/s recovering 2024-04-25T05:51:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:47 smithi060 ceph-mon[30898]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:51:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:47 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:51:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:47 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:51:47.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:47.588 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:47.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[37492]: pgmap v945: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s; 37965/371946 objects misplaced (10.207%); 12 MiB/s, 203 objects/s recovering 2024-04-25T05:51:47.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[37492]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:51:47.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:51:47.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:51:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[27010]: pgmap v945: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s; 37965/371946 objects misplaced (10.207%); 12 MiB/s, 203 objects/s recovering 2024-04-25T05:51:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[27010]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-25T05:51:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:51:47.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:47 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:51:47.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:47.730 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:47.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:47.867 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:47.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:47.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:48.005 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:48.107 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:48.139 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:48.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 88 15 128310 128295 91.0811 60.1875 0.0222224 0.0109742 2024-04-25T05:51:48.241 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:48.273 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:48.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:48.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:48 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:51:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:48 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:51:48.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:48.545 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:48 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:51:48.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:48 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:51:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:48 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:51:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:48 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:51:48.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:48.677 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:48.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:48.820 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:48.890 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:48.890 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:51:48.890 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:51:48.891 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:51:48.891 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:48.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:48.959 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:48.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:48.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:49.060 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:49.102 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:49.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:49.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout: 89 16 129364 129348 90.7972 65.8125 0.00762781 0.0110089 2024-04-25T05:51:49.236 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:49.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:49.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:49.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:49 smithi060 ceph-mon[30898]: pgmap v946: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 703 B/s rd, 41 MiB/s wr, 662 op/s; 37664/373863 objects misplaced (10.074%); 15 MiB/s, 246 objects/s recovering 2024-04-25T05:51:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:49 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:51:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:49 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:51:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:49.510 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:49.612 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[27010]: pgmap v946: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 703 B/s rd, 41 MiB/s wr, 662 op/s; 37664/373863 objects misplaced (10.074%); 15 MiB/s, 246 objects/s recovering 2024-04-25T05:51:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:51:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:51:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:49.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[37492]: pgmap v946: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 162 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 703 B/s rd, 41 MiB/s wr, 662 op/s; 37664/373863 objects misplaced (10.074%); 15 MiB/s, 246 objects/s recovering 2024-04-25T05:51:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:51:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:51:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:51:49.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:51:49.648 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:49.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:51:49.750 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.783 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:49.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:49.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:49.905 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:49.918 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:49.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:50.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:50.053 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:51:50.067 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:50.100 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:51:50.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:50.195 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:50.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total time run: 90.0078 2024-04-25T05:51:50.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Total writes made: 130286 2024-04-25T05:51:50.229 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Write size: 65536 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Object size: 65536 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Bandwidth (MB/sec): 90.4686 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Bandwidth: 34.1275 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max bandwidth (MB/sec): 202.5 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min bandwidth (MB/sec): 0 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average IOPS: 1447 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev IOPS: 546.05 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max IOPS: 3240 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min IOPS: 0 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Average Latency(s): 0.0110493 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Stddev Latency(s): 0.0258693 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Max latency(s): 2.27621 2024-04-25T05:51:50.230 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Min latency(s): 0.00134748 2024-04-25T05:51:50.233 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Cleaning up (deleting benchmark objects) 2024-04-25T05:51:50.298 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:50.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[27010]: 1.10 scrub starts 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[27010]: 1.10 scrub ok 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[27010]: 1.e scrub starts 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[27010]: 1.e scrub ok 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[27010]: osdmap e504: 8 total, 8 up, 8 in 2024-04-25T05:51:50.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[37492]: 1.10 scrub starts 2024-04-25T05:51:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[37492]: 1.10 scrub ok 2024-04-25T05:51:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[37492]: 1.e scrub starts 2024-04-25T05:51:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[37492]: 1.e scrub ok 2024-04-25T05:51:50.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:50 smithi043 ceph-mon[37492]: osdmap e504: 8 total, 8 up, 8 in 2024-04-25T05:51:50.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:50.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:50 smithi060 ceph-mon[30898]: 1.10 scrub starts 2024-04-25T05:51:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:50 smithi060 ceph-mon[30898]: 1.10 scrub ok 2024-04-25T05:51:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:50 smithi060 ceph-mon[30898]: 1.e scrub starts 2024-04-25T05:51:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:50 smithi060 ceph-mon[30898]: 1.e scrub ok 2024-04-25T05:51:50.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:50 smithi060 ceph-mon[30898]: osdmap e504: 8 total, 8 up, 8 in 2024-04-25T05:51:50.577 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:50.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:50.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:50.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228018 8.17 8.878e0737 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 548.83199282099997, 2024-04-25T05:51:50.811 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0050960120000001, 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228018 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975922+0000", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975924+0000", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.711e-06 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975930+0000", 2024-04-25T05:51:50.813 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1469999999999998e-06 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975947+0000", 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7601999999999999e-05 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.975956+0000", 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9400000000000008e-06 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.814 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976889+0000", 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093282800000000002 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.976898+0000", 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8769999999999999e-06 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.980999+0000", 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.00410116 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.815 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981018+0000", 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8746999999999999e-05 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228040 8.17 8.3c35da57 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 548.81453853100004, 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98798208499999995, 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:50.816 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228040 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993376+0000", 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993377+0000", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.012e-06 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993383+0000", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9769999999999997e-06 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993394+0000", 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0370999999999999e-05 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993402+0000", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1729999999999995e-06 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993458+0000", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6025e-05 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993465+0000", 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2339999999999997e-06 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981341+0000", 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98787640899999996 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981358+0000", 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6884e-05 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25115.0:228041 8.17 8.7bebe17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-04-25T05:51:50.820 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 548.81438322300005, 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98792126800000002, 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25115", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4286252813", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228041 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:51:50.821 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993531+0000", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993532+0000", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8400000000000005e-07 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993535+0000", 2024-04-25T05:51:50.822 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.6070000000000002e-06 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993542+0000", 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9619999999999996e-06 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993548+0000", 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8190000000000003e-06 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.823 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993585+0000", 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6328000000000002e-05 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:41.993588+0000", 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4319999999999999e-06 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981443+0000", 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98785475499999997 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:42:42.981453+0000", 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8810000000000008e-06 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:6 9.7 9:e9b4c7c6:::benchmark_data_smithi043_129820_object5:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 515.60736534600005, 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.006128809, 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 6 2024-04-25T05:51:50.826 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200549+0000", 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.827 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200550+0000", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3300000000000002e-07 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200583+0000", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2659e-05 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200590+0000", 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9890000000000003e-06 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200595+0000", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7899999999999996e-06 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200646+0000", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0822000000000002e-05 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200656+0000", 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5109999999999992e-06 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.829 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192142+0000", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99148672699999996 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192281+0000", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139018 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192427+0000", 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014579999999999999 2024-04-25T05:51:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.194168+0000", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0017404429999999999 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205031+0000", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010862966 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206577+0000", 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015465780000000001 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206635+0000", 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7998000000000003e-05 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206678+0000", 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2975000000000001e-05 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:12 9.d 9:b543e130:::benchmark_data_smithi043_129820_object11:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:51:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 515.60721159599996, 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0180756520000001, 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 12 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:51:50.833 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200703+0000", 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.200704+0000", 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2099999999999997e-07 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.834 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212291+0000", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011587573 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212312+0000", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0627000000000001e-05 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212324+0000", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.185e-05 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.835 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212430+0000", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010610199999999999 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.212444+0000", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4188999999999999e-05 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205046+0000", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99260219400000005 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205178+0000", 2024-04-25T05:51:50.836 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013207299999999999 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205330+0000", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000151438 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.209820+0000", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044903390000000003 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210175+0000", 2024-04-25T05:51:50.837 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00035483800000000001 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218678+0000", 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0085029509999999999 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218737+0000", 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9036000000000002e-05 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.838 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218779+0000", 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1921000000000001e-05 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:20 9.d 9:b555abe5:::benchmark_data_smithi043_129820_object19:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 515.59317396200004, 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0041661829999999, 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.839 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 20 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.840 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214741+0000", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2600000000000002e-07 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214807+0000", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5287e-05 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214816+0000", 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7630000000000006e-06 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.841 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214821+0000", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214856+0000", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5147000000000001e-05 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.214861+0000", 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.205852+0000", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99099027500000003 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206034+0000", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000182628 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206177+0000", 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014306599999999999 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210096+0000", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0039190930000000002 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.210182+0000", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6057000000000006e-05 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218819+0000", 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0086365039999999997 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218868+0000", 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9302000000000003e-05 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.218907+0000", 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8674999999999998e-05 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.845 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:34 9.7 9:e321d650:::benchmark_data_smithi043_129820_object33:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 515.58907396200004, 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.988632811, 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 34 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.846 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218841+0000", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4099999999999998e-07 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.847 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218902+0000", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0786999999999999e-05 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218912+0000", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4930000000000004e-06 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218920+0000", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1890000000000007e-06 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218967+0000", 2024-04-25T05:51:50.848 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6733000000000001e-05 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:15.218984+0000", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7153999999999999e-05 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192758+0000", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97377451299999995 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.192893+0000", 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013458900000000001 2024-04-25T05:51:50.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.193067+0000", 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017409899999999999 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.195263+0000", 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021959700000000002 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.850 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.206451+0000", 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011187704 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207381+0000", 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00093041 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.851 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207433+0000", 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2145000000000003e-05 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:43:16.207474+0000", 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0284000000000003e-05 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122647 9.9 9:946f8a80:::benchmark_data_smithi043_129820_object1333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 424.53960826500003, 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98571332099999998, 2024-04-25T05:51:50.852 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122647 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268306+0000", 2024-04-25T05:51:50.853 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268307+0000", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5600000000000004e-07 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268313+0000", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9560000000000002e-06 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268323+0000", 2024-04-25T05:51:50.854 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3479999999999993e-06 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268330+0000", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.861e-06 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268387+0000", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7014e-05 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.268396+0000", 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.842e-06 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249234+0000", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98083744799999995 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249334+0000", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9856000000000001e-05 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.249425+0000", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1697e-05 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.856 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253800+0000", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043740819999999996 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253847+0000", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7231999999999998e-05 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253904+0000", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7262000000000002e-05 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.253938+0000", 2024-04-25T05:51:50.857 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4394000000000003e-05 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254020+0000", 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1372999999999995e-05 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122655 9.9 9:91969b48:::benchmark_data_smithi043_129820_object1341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 424.53528236199998, 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98173263399999999, 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122655 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272632+0000", 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272633+0000", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.39e-07 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272636+0000", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4369999999999999e-06 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272641+0000", 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0069999999999997e-06 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.860 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272644+0000", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1300000000000001e-06 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272681+0000", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7063000000000001e-05 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.272687+0000", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2059999999999998e-06 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251540+0000", 2024-04-25T05:51:50.861 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97885334999999996 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251625+0000", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5487000000000003e-05 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.251692+0000", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6076999999999998e-05 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254167+0000", 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002475929 2024-04-25T05:51:50.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254218+0000", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1023999999999998e-05 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254276+0000", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7348e-05 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254326+0000", 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0139999999999998e-05 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.863 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254365+0000", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8996999999999999e-05 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:122657 9.9 9:97065b42:::benchmark_data_smithi043_129820_object1343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 424.53481359800003, 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98447252200000002, 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.864 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 122657 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273101+0000", 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273102+0000", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1399999999999995e-07 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273106+0000", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4240000000000003e-06 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273116+0000", 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1109999999999997e-06 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.866 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273123+0000", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2089999999999999e-06 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273170+0000", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7475999999999997e-05 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:46.273178+0000", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5789999999999998e-06 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.867 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252726+0000", 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97954808900000001 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252818+0000", 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2471000000000003e-05 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.252938+0000", 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119312 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.868 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254376+0000", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001438745 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.254418+0000", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1127000000000001e-05 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257471+0000", 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003053596 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.869 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257528+0000", 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7151000000000002e-05 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:44:47.257574+0000", 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5318000000000002e-05 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234262 9.13 9:c8a64bff:::benchmark_data_smithi043_129820_object112948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:51:50.870 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 337.38839200699999, 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044775727, 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234262 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:51:50.871 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419524+0000", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.114e-06 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419529+0000", 2024-04-25T05:51:50.872 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1320000000000002e-06 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419541+0000", 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2517000000000001e-05 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419551+0000", 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.55e-06 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.873 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419626+0000", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5401999999999994e-05 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419639+0000", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3045e-05 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458481+0000", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0388411980000001 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458596+0000", 2024-04-25T05:51:50.874 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114869 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458847+0000", 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000251838 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.875 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.875 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463718+0000", 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048708349999999996 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.876 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463779+0000", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0671000000000002e-05 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464065+0000", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028592400000000001 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464124+0000", 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8913000000000003e-05 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.877 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464180+0000", 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6740000000000002e-05 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464298+0000", 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117979 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.878 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234266 9.13 9:cde636a1:::benchmark_data_smithi043_129820_object112952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 337.38463081499998, 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.043099716, 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234266 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.879 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423285+0000", 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.47e-07 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.880 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423290+0000", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-06 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423301+0000", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1249e-05 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423309+0000", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.881 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423370+0000", 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1496999999999998e-05 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423380+0000", 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.459982+0000", 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.036602386 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.882 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460104+0000", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121814 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460347+0000", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024315799999999999 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464108+0000", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003761347 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.883 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464306+0000", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019793999999999999 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466211+0000", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001904836 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466254+0000", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2517e-05 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466302+0000", 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8791000000000001e-05 2024-04-25T05:51:50.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466384+0000", 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1110000000000001e-05 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237263 9.10 9:09fb57ed:::benchmark_data_smithi043_129820_object115949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.71172343299997, 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0207323559999999, 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.885 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237263 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:51:50.886 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096192+0000", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8400000000000004e-07 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096195+0000", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1750000000000001e-06 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096202+0000", 2024-04-25T05:51:50.887 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8600000000000004e-06 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096206+0000", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096257+0000", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0695000000000002e-05 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096262+0000", 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5300000000000004e-06 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.888 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113813+0000", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0175506910000001 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114018+0000", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000205124 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114193+0000", 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000174981 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.889 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116636+0000", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024427059999999998 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116671+0000", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5380999999999999e-05 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116825+0000", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000153993 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.890 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116873+0000", 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7308000000000002e-05 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116924+0000", 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1087000000000001e-05 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237265 9.13 9:ced551e7:::benchmark_data_smithi043_129820_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.710499986, 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.029123689, 2024-04-25T05:51:50.891 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237265 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.892 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3300000000000001e-07 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097420+0000", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0969999999999999e-06 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097429+0000", 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0380000000000005e-06 2024-04-25T05:51:50.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097436+0000", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7449999999999995e-06 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097483+0000", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6343e-05 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097489+0000", 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6830000000000002e-06 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115159+0000", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0176693729999999 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115269+0000", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109979 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115548+0000", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027962100000000003 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.895 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118288+0000", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027397879999999999 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120356+0000", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020682280000000001 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126451+0000", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060945569999999996 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126495+0000", 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3942e-05 2024-04-25T05:51:50.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126538+0000", 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3562000000000001e-05 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237270 9.13 9:ccb5a786:::benchmark_data_smithi043_129820_object115956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.70713440600002, 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.025924235, 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.897 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237270 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100781+0000", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1200000000000002e-07 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100789+0000", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7179999999999996e-06 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100798+0000", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8599999999999999e-06 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.899 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100804+0000", 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0009999999999998e-06 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100867+0000", 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3851999999999998e-05 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100877+0000", 2024-04-25T05:51:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1940000000000004e-06 2024-04-25T05:51:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.901 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117090+0000", 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016213622 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117202+0000", 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111869 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117405+0000", 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020259799999999999 2024-04-25T05:51:50.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126115+0000", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0087104629999999999 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126211+0000", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5789000000000001e-05 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126619+0000", 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00040798499999999998 2024-04-25T05:51:50.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126663+0000", 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3543e-05 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126705+0000", 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2029e-05 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237273 9.10 9:0de19dcd:::benchmark_data_smithi043_129820_object115959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.904 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.70536312299998, 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0209831819999999, 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237273 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102553+0000", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.034e-06 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102559+0000", 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9089999999999997e-06 2024-04-25T05:51:50.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102570+0000", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.112e-05 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102578+0000", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2709999999999992e-06 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102617+0000", 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8695999999999997e-05 2024-04-25T05:51:50.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102625+0000", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9929999999999995e-06 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115511+0000", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0128864559999999 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115643+0000", 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131746 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115827+0000", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018383200000000001 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118880+0000", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030529400000000001 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119838+0000", 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00095799800000000005 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.909 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123433+0000", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035950230000000001 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123491+0000", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8242000000000002e-05 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123535+0000", 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3921999999999997e-05 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237293 9.13 9:cf509854:::benchmark_data_smithi043_129820_object115979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.69724430000002, 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016106186, 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237293 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.911 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110671+0000", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110676+0000", 2024-04-25T05:51:50.912 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6029999999999998e-06 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110685+0000", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6149999999999997e-06 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110691+0000", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8569999999999997e-06 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110748+0000", 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6978999999999998e-05 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110757+0000", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7949999999999996e-06 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118323+0000", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007566237 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118419+0000", 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5356e-05 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118567+0000", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014814099999999999 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126167+0000", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.007599522 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126302+0000", 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013507600000000001 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126712+0000", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041020800000000001 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126745+0000", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3250999999999997e-05 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126777+0000", 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1547000000000001e-05 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237310 9.13 9:cc1e0a4c:::benchmark_data_smithi043_129820_object115996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.68801624999998, 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007006198, 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237310 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119899+0000", 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9100000000000003e-07 2024-04-25T05:51:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119903+0000", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5090000000000002e-06 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119912+0000", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0440000000000001e-06 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119917+0000", 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2959999999999998e-06 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.919 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119983+0000", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5779999999999997e-05 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119993+0000", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8179999999999999e-06 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119292+0000", 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99929933400000004 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119417+0000", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125449 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119591+0000", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017395799999999999 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126261+0000", 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066693810000000003 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.921 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126390+0000", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000129164 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126784+0000", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039376400000000001 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126863+0000", 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8963999999999998e-05 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126905+0000", 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2045999999999997e-05 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237325 9.13 9:cbb3ea78:::benchmark_data_smithi043_129820_object116011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.67841943799999, 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99763348100000004, 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237325 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:51:50.924 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129496+0000", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4600000000000004e-07 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129500+0000", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1790000000000001e-06 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129511+0000", 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0611000000000001e-05 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.925 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129518+0000", 2024-04-25T05:51:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6020000000000002e-06 2024-04-25T05:51:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129565+0000", 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6158000000000001e-05 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129573+0000", 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6240000000000008e-06 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120395+0000", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99082194099999998 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120502+0000", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010669 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120678+0000", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175976 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126345+0000", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005667025 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126911+0000", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056641199999999997 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127019+0000", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107938 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127082+0000", 2024-04-25T05:51:50.929 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2848999999999994e-05 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127129+0000", 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6730000000000002e-05 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237346 9.10 9:0cf8b6b3:::benchmark_data_smithi043_129820_object116032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 334.666630489, 2024-04-25T05:51:50.930 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98237561100000004, 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237346 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.931 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141285+0000", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3699999999999999e-07 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141290+0000", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6820000000000002e-06 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:51:50.932 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141300+0000", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9890000000000004e-06 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141309+0000", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9840000000000007e-06 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141364+0000", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4886999999999998e-05 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141373+0000", 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2369999999999998e-06 2024-04-25T05:51:50.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116981+0000", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97560839600000004 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117101+0000", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119911 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117270+0000", 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016895300000000001 2024-04-25T05:51:50.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119880+0000", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026103020000000001 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.121458+0000", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001577422 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123574+0000", 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021161970000000002 2024-04-25T05:51:50.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123625+0000", 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0562000000000001e-05 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123660+0000", 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5151999999999998e-05 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:51:50.936 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:51:50.937 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:51:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:50.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:50.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:51.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:51.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:51.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:51.347 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 1.12 scrub starts 2024-04-25T05:51:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 1.12 scrub ok 2024-04-25T05:51:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 37632/386961 objects misplaced (9.725%); 23 MiB/s, 366 objects/s recovering 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 1.d scrub starts 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 1.d scrub ok 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 1.12 scrub starts 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 1.12 scrub ok 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 37632/386961 objects misplaced (9.725%); 23 MiB/s, 366 objects/s recovering 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 1.d scrub starts 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 1.d scrub ok 2024-04-25T05:51:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:51:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:51 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:51:51.394 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:51:51.441 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:51:51.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:51.489 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:51.489 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:51:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:51.535 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:51:51.579 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:51:51.591 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:51.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:51.633 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:51:51.684 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:51:51.723 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:51:51.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:51.734 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:51:51.734 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:51:51.734 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:51:51.734 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:51:51.735 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:51:51.735 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:51:51.735 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:51:51.735 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:51:51.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 1.12 scrub starts 2024-04-25T05:51:51.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 1.12 scrub ok 2024-04-25T05:51:51.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: pgmap v947: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 37632/386961 objects misplaced (9.725%); 23 MiB/s, 366 objects/s recovering 2024-04-25T05:51:51.737 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 1.d scrub starts 2024-04-25T05:51:51.738 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 1.d scrub ok 2024-04-25T05:51:51.738 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:51:51.738 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:51 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:51:51.760 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:51.862 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:51.863 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:51:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:51.914 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:51.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:52.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:52.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:52.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:52.218 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:52.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:52.360 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:52.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:52.500 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:52.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:52.617 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:52 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:51:52.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:52 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:51:52.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/153877850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:52.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:52 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:51:52.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:52 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:51:52.634 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/153877850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:51:52.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:51:52.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/153877850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:51:52.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:51:52.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:52 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:51:52.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:51:52.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:52.767 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:52.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:52.900 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:52.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:53.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:53.030 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:53.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:53.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:53.235 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:51:53.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:51:53.235 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-04-25T05:51:53.235 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-04-25T05:51:53.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:51:53.278 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:53.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.348 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:53.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[27010]: pgmap v949: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 657 B/s rd, 40 MiB/s wr, 643 op/s; 37632/386961 objects misplaced (9.725%); 15 MiB/s, 240 objects/s recovering 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[37492]: pgmap v949: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 657 B/s rd, 40 MiB/s wr, 643 op/s; 37632/386961 objects misplaced (9.725%); 15 MiB/s, 240 objects/s recovering 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:51:53.434 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:53 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:51:53.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:53.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:53.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:53.610 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:53.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:53.743 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:53 smithi060 ceph-mon[30898]: pgmap v949: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 162 active+clean; 7.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 657 B/s rd, 40 MiB/s wr, 643 op/s; 37632/386961 objects misplaced (9.725%); 15 MiB/s, 240 objects/s recovering 2024-04-25T05:51:53.743 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:53 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:51:53.743 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:53 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:51:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:53.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:53.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:53.889 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:53.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:53.964 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:53.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:53.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:51:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:51:54.017 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:54.049 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:51:54.049 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:51:54.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:54.152 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:54.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:54.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:54.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:54.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:54.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:54.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:54.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:54.785 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:54.823 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:54.926 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:54.960 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:54.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:55.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.093 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:55.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:55.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:51:55.182 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:51:55.194 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:55.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:55.229 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:51:55.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:55.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:55.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:51:55.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:51:55.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: pgmap v950: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 899 op/s; 36170/384864 objects misplaced (9.398%); 23 MiB/s, 365 objects/s recovering 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: pgmap v950: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 899 op/s; 36170/384864 objects misplaced (9.398%); 23 MiB/s, 365 objects/s recovering 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:51:55.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:55 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:51:55.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:55.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:55.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: pgmap v950: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.3 KiB/s rd, 43 MiB/s wr, 899 op/s; 36170/384864 objects misplaced (9.398%); 23 MiB/s, 365 objects/s recovering 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:51:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:55 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:51:55.798 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:55.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:55.977 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:55.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:56.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.111 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:56.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:56.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:56.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:56.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:56.344 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:51:56.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:56.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:56.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:56.580 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2458862111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.581 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.581 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2458862111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.581 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.635 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:56.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:56.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:51:56.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2458862111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:51:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:51:56.790 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:56.892 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:56.926 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:56.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:57.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:57.060 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:57.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:57.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:57.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:57.335 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:57.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:57.467 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:57.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:57.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: pgmap v951: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 41 MiB/s wr, 1.01k op/s; 35066/380808 objects misplaced (9.208%); 23 MiB/s, 371 objects/s recovering 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: osdmap e505: 8 total, 8 up, 8 in 2024-04-25T05:51:57.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: pgmap v951: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 41 MiB/s wr, 1.01k op/s; 35066/380808 objects misplaced (9.208%); 23 MiB/s, 371 objects/s recovering 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: osdmap e505: 8 total, 8 up, 8 in 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:51:57.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:57 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:51:57.649 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:51:57.649 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: pgmap v951: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 41 MiB/s wr, 1.01k op/s; 35066/380808 objects misplaced (9.208%); 23 MiB/s, 371 objects/s recovering 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: osdmap e505: 8 total, 8 up, 8 in 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:51:57.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:57 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:51:57.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:51:57.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:51:57.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:57.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:57.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:57.983 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:58.016 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:58.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:58.153 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:58.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:58.395 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:58.429 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:58.531 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:58.566 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:58.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: from='client.26441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 4.d deep-scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[27010]: 4.d deep-scrub ok 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: from='client.26441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 4.d deep-scrub starts 2024-04-25T05:51:58.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:58 smithi043 ceph-mon[37492]: 4.d deep-scrub ok 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: from='client.26441 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 4.d deep-scrub starts 2024-04-25T05:51:58.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:58 smithi060 ceph-mon[30898]: 4.d deep-scrub ok 2024-04-25T05:51:58.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:58.713 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:58.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:51:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:51:58.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:51:58.988 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:58.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:51:59.090 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:51:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.122 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:51:59.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:51:59.264 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:51:59.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:59.394 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:59.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:51:59.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:51:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:51:59.634 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:51:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[27010]: pgmap v953: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 767 B/s rd, 12 MiB/s wr, 719 op/s; 34255/378882 objects misplaced (9.041%); 19 MiB/s, 302 objects/s recovering 2024-04-25T05:51:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[37492]: pgmap v953: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 767 B/s rd, 12 MiB/s wr, 719 op/s; 34255/378882 objects misplaced (9.041%); 19 MiB/s, 302 objects/s recovering 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:51:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:51:59.666 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:51:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:59 smithi060 ceph-mon[30898]: pgmap v953: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 767 B/s rd, 12 MiB/s wr, 719 op/s; 34255/378882 objects misplaced (9.041%); 19 MiB/s, 302 objects/s recovering 2024-04-25T05:51:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:59 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:51:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:59 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:51:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:59 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:51:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:51:59 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:51:59.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:59.803 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:51:59.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:51:59.937 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:51:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:51:59.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:00.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:00.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:51:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:51:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:00.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:00.211 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:00.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:00.347 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:00.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:00.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:00.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:00.619 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:00 smithi043 ceph-mon[27010]: osdmap e506: 8 total, 8 up, 8 in 2024-04-25T05:52:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:00 smithi043 ceph-mon[37492]: osdmap e506: 8 total, 8 up, 8 in 2024-04-25T05:52:00.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:00 smithi060 ceph-mon[30898]: osdmap e506: 8 total, 8 up, 8 in 2024-04-25T05:52:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:00.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:00.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:01.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:01.057 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:01.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:01.196 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:01.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.330 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:01.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:01.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:01.463 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:01.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:01 smithi043 ceph-mon[27010]: 11.7 scrub starts 2024-04-25T05:52:01.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:01 smithi043 ceph-mon[27010]: pgmap v955: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.25k op/s; 32007/366165 objects misplaced (8.741%); 28 MiB/s, 444 objects/s recovering 2024-04-25T05:52:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:01 smithi043 ceph-mon[37492]: 11.7 scrub starts 2024-04-25T05:52:01.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:01 smithi043 ceph-mon[37492]: pgmap v955: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.25k op/s; 32007/366165 objects misplaced (8.741%); 28 MiB/s, 444 objects/s recovering 2024-04-25T05:52:01.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:01 smithi060 ceph-mon[30898]: 11.7 scrub starts 2024-04-25T05:52:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:01 smithi060 ceph-mon[30898]: pgmap v955: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 12 MiB/s wr, 1.25k op/s; 32007/366165 objects misplaced (8.741%); 28 MiB/s, 444 objects/s recovering 2024-04-25T05:52:01.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:01.878 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:01.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:01.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:02.009 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":952,"stamp":"2024-04-25T05:51:56.335235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8304116958,"num_objects":126936,"num_object_clones":0,"num_object_copies":380808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35066,"num_objects_unfound":0,"num_objects_dirty":126936,"num_whiteouts":0,"num_read":2469,"num_read_kb":2189,"num_write":134782,"num_write_kb":8338350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5159,"num_bytes_recovered":295609255,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77127,"ondisk_log_size":77127,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":509,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":31223848,"kb_used_data":24736616,"kb_used_omap":12,"kb_used_meta":6486963,"kb_avail":718737368,"statfs":{"total":767960285184,"available":735987064832,"internally_reserved":0,"allocated":25330294784,"data_stored":25324792497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":6642650672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":151650328,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2598,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":9477,"num_write_kb":377281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3096,"num_bytes_recovered":202899456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.283902"},"pg_stats":[{"pgid":"11.3","version":"504'8386","reported_seq":8567,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.570080+0000","last_change":"2024-04-25T05:51:25.992776+0000","last_active":"2024-04-25T05:51:53.570080+0000","last_peered":"2024-04-25T05:51:53.570080+0000","last_clean":"2024-04-25T05:51:53.570080+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:51:53.570080+0000","last_undegraded":"2024-04-25T05:51:53.570080+0000","last_fullsized":"2024-04-25T05:51:53.570080+0000","mapping_epoch":456,"log_start":"473'3465","ondisk_log_start":"473'3465","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'6608","last_scrub_stamp":"2024-04-25T05:51:25.992415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:51:25.992415+0000","objects_scrubbed":6498,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:36.479074+0000","scrub_duration":2601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522321920,"num_objects":7970,"num_object_clones":0,"num_object_copies":23910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":424,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.092381+0000","last_change":"2024-04-25T05:50:08.705811+0000","last_active":"2024-04-25T05:51:50.092381+0000","last_peered":"2024-04-25T05:51:50.092381+0000","last_clean":"2024-04-25T05:51:50.092381+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:51:50.092381+0000","last_undegraded":"2024-04-25T05:51:50.092381+0000","last_fullsized":"2024-04-25T05:51:50.092381+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:50:08.705677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.705677+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":947,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074241+0000","last_change":"2024-04-25T05:51:40.101355+0000","last_active":"2024-04-25T05:51:50.074241+0000","last_peered":"2024-04-25T05:51:50.074241+0000","last_clean":"2024-04-25T05:51:50.074241+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:51:50.074241+0000","last_undegraded":"2024-04-25T05:51:50.074241+0000","last_fullsized":"2024-04-25T05:51:50.074241+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":362,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.079020+0000","last_change":"2024-04-25T05:50:12.546520+0000","last_active":"2024-04-25T05:51:50.079020+0000","last_peered":"2024-04-25T05:51:50.079020+0000","last_clean":"2024-04-25T05:51:50.079020+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:51:50.079020+0000","last_undegraded":"2024-04-25T05:51:50.079020+0000","last_fullsized":"2024-04-25T05:51:50.079020+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:50:12.546342+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.546342+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":501,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091907+0000","last_change":"2024-04-25T05:50:13.709776+0000","last_active":"2024-04-25T05:51:50.091907+0000","last_peered":"2024-04-25T05:51:50.091907+0000","last_clean":"2024-04-25T05:51:50.091907+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:51:50.091907+0000","last_undegraded":"2024-04-25T05:51:50.091907+0000","last_fullsized":"2024-04-25T05:51:50.091907+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.709675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.709675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1259,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080048+0000","last_change":"2024-04-25T05:49:33.878907+0000","last_active":"2024-04-25T05:51:50.080048+0000","last_peered":"2024-04-25T05:51:50.080048+0000","last_clean":"2024-04-25T05:51:50.080048+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:51:50.080048+0000","last_undegraded":"2024-04-25T05:51:50.080048+0000","last_fullsized":"2024-04-25T05:51:50.080048+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:33.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:33.878785+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"504'8397","reported_seq":11469,"reported_epoch":504,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:51:52.797927+0000","last_change":"2024-04-25T05:51:41.456911+0000","last_active":"2024-04-25T05:51:52.797927+0000","last_peered":"2024-04-25T05:51:52.797927+0000","last_clean":"2024-04-25T05:51:39.184844+0000","last_became_active":"2024-04-25T05:51:41.441650+0000","last_became_peered":"2024-04-25T05:51:41.441650+0000","last_unstale":"2024-04-25T05:51:52.797927+0000","last_undegraded":"2024-04-25T05:51:52.797927+0000","last_fullsized":"2024-04-25T05:51:52.797927+0000","mapping_epoch":498,"log_start":"471'3063","ondisk_log_start":"471'3063","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":5334,"log_dups_size":0,"ondisk_log_size":5334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529072128,"num_objects":8073,"num_object_clones":0,"num_object_copies":24219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6585,"num_objects_unfound":0,"num_objects_dirty":8073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1481,"num_bytes_recovered":97058816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,7],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":8073}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":425,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091585+0000","last_change":"2024-04-25T05:49:09.556929+0000","last_active":"2024-04-25T05:51:50.091585+0000","last_peered":"2024-04-25T05:51:50.091585+0000","last_clean":"2024-04-25T05:51:50.091585+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:51:50.091585+0000","last_undegraded":"2024-04-25T05:51:50.091585+0000","last_fullsized":"2024-04-25T05:51:50.091585+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:48:59.073429+0000","last_deep_scrub":"277'25","last_deep_scrub_stamp":"2024-04-25T05:42:24.143971+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.073429+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":662,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.077404+0000","last_change":"2024-04-25T05:51:22.353776+0000","last_active":"2024-04-25T05:51:50.077404+0000","last_peered":"2024-04-25T05:51:50.077404+0000","last_clean":"2024-04-25T05:51:50.077404+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:51:50.077404+0000","last_undegraded":"2024-04-25T05:51:50.077404+0000","last_fullsized":"2024-04-25T05:51:50.077404+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.353535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.353535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:50.430842+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":960,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075610+0000","last_change":"2024-04-25T05:51:40.109308+0000","last_active":"2024-04-25T05:51:50.075610+0000","last_peered":"2024-04-25T05:51:50.075610+0000","last_clean":"2024-04-25T05:51:50.075610+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:51:50.075610+0000","last_undegraded":"2024-04-25T05:51:50.075610+0000","last_fullsized":"2024-04-25T05:51:50.075610+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.345574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":290,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078670+0000","last_change":"2024-04-25T05:50:46.862960+0000","last_active":"2024-04-25T05:51:50.078670+0000","last_peered":"2024-04-25T05:51:50.078670+0000","last_clean":"2024-04-25T05:51:50.078670+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:51:50.078670+0000","last_undegraded":"2024-04-25T05:51:50.078670+0000","last_fullsized":"2024-04-25T05:51:50.078670+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:46.862820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.862820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:55.777488+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":857,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074625+0000","last_change":"2024-04-25T05:50:06.715616+0000","last_active":"2024-04-25T05:51:50.074625+0000","last_peered":"2024-04-25T05:51:50.074625+0000","last_clean":"2024-04-25T05:51:50.074625+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:51:50.074625+0000","last_undegraded":"2024-04-25T05:51:50.074625+0000","last_fullsized":"2024-04-25T05:51:50.074625+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:50:06.715476+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.715476+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"79'128","reported_seq":1146,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074949+0000","last_change":"2024-04-25T05:51:43.261405+0000","last_active":"2024-04-25T05:51:50.074949+0000","last_peered":"2024-04-25T05:51:50.074949+0000","last_clean":"2024-04-25T05:51:50.074949+0000","last_became_active":"2024-04-25T05:51:43.257069+0000","last_became_peered":"2024-04-25T05:51:43.257069+0000","last_unstale":"2024-04-25T05:51:50.074949+0000","last_undegraded":"2024-04-25T05:51:50.074949+0000","last_fullsized":"2024-04-25T05:51:50.074949+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.990556+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.361400+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"504'8380","reported_seq":11782,"reported_epoch":504,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:51:53.626240+0000","last_change":"2024-04-25T05:51:41.463174+0000","last_active":"2024-04-25T05:51:53.626240+0000","last_peered":"2024-04-25T05:51:53.626240+0000","last_clean":"2024-04-25T05:51:39.182821+0000","last_became_active":"2024-04-25T05:51:41.447463+0000","last_became_peered":"2024-04-25T05:51:41.447463+0000","last_unstale":"2024-04-25T05:51:53.626240+0000","last_undegraded":"2024-04-25T05:51:53.626240+0000","last_fullsized":"2024-04-25T05:51:53.626240+0000","mapping_epoch":498,"log_start":"472'3453","ondisk_log_start":"472'3453","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'6932","last_scrub_stamp":"2024-04-25T05:51:30.948125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.948125+0000","objects_scrubbed":6847,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517341184,"num_objects":7894,"num_object_clones":0,"num_object_copies":23682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6280,"num_objects_unfound":0,"num_objects_dirty":7894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8380,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1607,"num_bytes_recovered":105316352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":7894}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":733,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091617+0000","last_change":"2024-04-25T05:50:11.713907+0000","last_active":"2024-04-25T05:51:50.091617+0000","last_peered":"2024-04-25T05:51:50.091617+0000","last_clean":"2024-04-25T05:51:50.091617+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:51:50.091617+0000","last_undegraded":"2024-04-25T05:51:50.091617+0000","last_fullsized":"2024-04-25T05:51:50.091617+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:50:11.713794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.713794+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":968,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088213+0000","last_change":"2024-04-25T05:51:25.729280+0000","last_active":"2024-04-25T05:51:50.088213+0000","last_peered":"2024-04-25T05:51:50.088213+0000","last_clean":"2024-04-25T05:51:50.088213+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:51:50.088213+0000","last_undegraded":"2024-04-25T05:51:50.088213+0000","last_fullsized":"2024-04-25T05:51:50.088213+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:25.729152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:25.729152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:48.412579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":257,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074670+0000","last_change":"2024-04-25T05:49:29.951087+0000","last_active":"2024-04-25T05:51:50.074670+0000","last_peered":"2024-04-25T05:51:50.074670+0000","last_clean":"2024-04-25T05:51:50.074670+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:51:50.074670+0000","last_undegraded":"2024-04-25T05:51:50.074670+0000","last_fullsized":"2024-04-25T05:51:50.074670+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:29.950964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:49:29.950964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":935,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074518+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:51:50.074518+0000","last_peered":"2024-04-25T05:51:50.074518+0000","last_clean":"2024-04-25T05:51:50.074518+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:51:50.074518+0000","last_undegraded":"2024-04-25T05:51:50.074518+0000","last_fullsized":"2024-04-25T05:51:50.074518+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"504'8344","reported_seq":8449,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:54.158695+0000","last_change":"2024-04-25T05:50:12.041469+0000","last_active":"2024-04-25T05:51:54.158695+0000","last_peered":"2024-04-25T05:51:54.158695+0000","last_clean":"2024-04-25T05:51:54.158695+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:51:54.158695+0000","last_undegraded":"2024-04-25T05:51:54.158695+0000","last_fullsized":"2024-04-25T05:51:54.158695+0000","mapping_epoch":456,"log_start":"475'3913","ondisk_log_start":"475'3913","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514457600,"num_objects":7850,"num_object_clones":0,"num_object_copies":23550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":518208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":976,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075559+0000","last_change":"2024-04-25T05:50:10.992094+0000","last_active":"2024-04-25T05:51:50.075559+0000","last_peered":"2024-04-25T05:51:50.075559+0000","last_clean":"2024-04-25T05:51:50.075559+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:51:50.075559+0000","last_undegraded":"2024-04-25T05:51:50.075559+0000","last_fullsized":"2024-04-25T05:51:50.075559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:10.991997+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:50:10.991997+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":932,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081809+0000","last_change":"2024-04-25T05:51:20.429881+0000","last_active":"2024-04-25T05:51:50.081809+0000","last_peered":"2024-04-25T05:51:50.081809+0000","last_clean":"2024-04-25T05:51:50.081809+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:51:50.081809+0000","last_undegraded":"2024-04-25T05:51:50.081809+0000","last_fullsized":"2024-04-25T05:51:50.081809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:20.429760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:20.429760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:28.031802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":618,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091807+0000","last_change":"2024-04-25T05:51:46.715282+0000","last_active":"2024-04-25T05:51:50.091807+0000","last_peered":"2024-04-25T05:51:50.091807+0000","last_clean":"2024-04-25T05:51:50.091807+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:51:50.091807+0000","last_undegraded":"2024-04-25T05:51:50.091807+0000","last_fullsized":"2024-04-25T05:51:50.091807+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:51:46.715159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:46.715159+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:15.436561+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":271,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074101+0000","last_change":"2024-04-25T05:51:40.105267+0000","last_active":"2024-04-25T05:51:50.074101+0000","last_peered":"2024-04-25T05:51:50.074101+0000","last_clean":"2024-04-25T05:51:50.074101+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:51:50.074101+0000","last_undegraded":"2024-04-25T05:51:50.074101+0000","last_fullsized":"2024-04-25T05:51:50.074101+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.069502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.069502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:38.754009+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":821,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074048+0000","last_change":"2024-04-25T05:50:47.941283+0000","last_active":"2024-04-25T05:51:50.074048+0000","last_peered":"2024-04-25T05:51:50.074048+0000","last_clean":"2024-04-25T05:51:50.074048+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:51:50.074048+0000","last_undegraded":"2024-04-25T05:51:50.074048+0000","last_fullsized":"2024-04-25T05:51:50.074048+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:50:47.940906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.940906+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.099117+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"504'8473","reported_seq":8578,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.360627+0000","last_change":"2024-04-25T05:50:12.042142+0000","last_active":"2024-04-25T05:51:53.360627+0000","last_peered":"2024-04-25T05:51:53.360627+0000","last_clean":"2024-04-25T05:51:53.360627+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:51:53.360627+0000","last_undegraded":"2024-04-25T05:51:53.360627+0000","last_fullsized":"2024-04-25T05:51:53.360627+0000","mapping_epoch":456,"log_start":"475'4048","ondisk_log_start":"475'4048","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527761408,"num_objects":8053,"num_object_clones":0,"num_object_copies":24159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":528832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":940,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088127+0000","last_change":"2024-04-25T05:49:03.437727+0000","last_active":"2024-04-25T05:51:50.088127+0000","last_peered":"2024-04-25T05:51:50.088127+0000","last_clean":"2024-04-25T05:51:50.088127+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:51:50.088127+0000","last_undegraded":"2024-04-25T05:51:50.088127+0000","last_fullsized":"2024-04-25T05:51:50.088127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:03.437604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:03.437604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1114,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081923+0000","last_change":"2024-04-25T05:51:40.149035+0000","last_active":"2024-04-25T05:51:50.081923+0000","last_peered":"2024-04-25T05:51:50.081923+0000","last_clean":"2024-04-25T05:51:50.081923+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:51:50.081923+0000","last_undegraded":"2024-04-25T05:51:50.081923+0000","last_fullsized":"2024-04-25T05:51:50.081923+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:51:29.363198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.363198+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:37.123655+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":962,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075528+0000","last_change":"2024-04-25T05:51:40.084198+0000","last_active":"2024-04-25T05:51:50.075528+0000","last_peered":"2024-04-25T05:51:50.075528+0000","last_clean":"2024-04-25T05:51:50.075528+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:51:50.075528+0000","last_undegraded":"2024-04-25T05:51:50.075528+0000","last_fullsized":"2024-04-25T05:51:50.075528+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.460712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.460712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":495,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088097+0000","last_change":"2024-04-25T05:50:41.801622+0000","last_active":"2024-04-25T05:51:50.088097+0000","last_peered":"2024-04-25T05:51:50.088097+0000","last_clean":"2024-04-25T05:51:50.088097+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:51:50.088097+0000","last_undegraded":"2024-04-25T05:51:50.088097+0000","last_fullsized":"2024-04-25T05:51:50.088097+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.801508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.801508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1185,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091089+0000","last_change":"2024-04-25T05:51:33.761115+0000","last_active":"2024-04-25T05:51:50.091089+0000","last_peered":"2024-04-25T05:51:50.091089+0000","last_clean":"2024-04-25T05:51:50.091089+0000","last_became_active":"2024-04-25T05:51:33.171322+0000","last_became_peered":"2024-04-25T05:51:33.171322+0000","last_unstale":"2024-04-25T05:51:50.091089+0000","last_undegraded":"2024-04-25T05:51:50.091089+0000","last_fullsized":"2024-04-25T05:51:50.091089+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.760985+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.760985+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:53.057905+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"504'8111","reported_seq":8217,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.212786+0000","last_change":"2024-04-25T05:50:12.042923+0000","last_active":"2024-04-25T05:51:53.212786+0000","last_peered":"2024-04-25T05:51:53.212786+0000","last_clean":"2024-04-25T05:51:53.212786+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:51:53.212786+0000","last_undegraded":"2024-04-25T05:51:53.212786+0000","last_fullsized":"2024-04-25T05:51:53.212786+0000","mapping_epoch":456,"log_start":"474'3789","ondisk_log_start":"474'3789","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506265600,"num_objects":7725,"num_object_clones":0,"num_object_copies":23175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1150,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.082054+0000","last_change":"2024-04-25T05:48:25.100906+0000","last_active":"2024-04-25T05:51:50.082054+0000","last_peered":"2024-04-25T05:51:50.082054+0000","last_clean":"2024-04-25T05:51:50.082054+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:51:50.082054+0000","last_undegraded":"2024-04-25T05:51:50.082054+0000","last_fullsized":"2024-04-25T05:51:50.082054+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:48:25.100752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:48:25.100752+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":625,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088399+0000","last_change":"2024-04-25T05:50:46.875806+0000","last_active":"2024-04-25T05:51:50.088399+0000","last_peered":"2024-04-25T05:51:50.088399+0000","last_clean":"2024-04-25T05:51:50.088399+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:51:50.088399+0000","last_undegraded":"2024-04-25T05:51:50.088399+0000","last_fullsized":"2024-04-25T05:51:50.088399+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:46.875662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:46.875662+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":27,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089583+0000","last_change":"2024-04-25T05:51:40.129777+0000","last_active":"2024-04-25T05:51:50.089583+0000","last_peered":"2024-04-25T05:51:50.089583+0000","last_clean":"2024-04-25T05:51:50.089583+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:51:50.089583+0000","last_undegraded":"2024-04-25T05:51:50.089583+0000","last_fullsized":"2024-04-25T05:51:50.089583+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":313,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075426+0000","last_change":"2024-04-25T05:50:12.172126+0000","last_active":"2024-04-25T05:51:50.075426+0000","last_peered":"2024-04-25T05:51:50.075426+0000","last_clean":"2024-04-25T05:51:50.075426+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:51:50.075426+0000","last_undegraded":"2024-04-25T05:51:50.075426+0000","last_fullsized":"2024-04-25T05:51:50.075426+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:12.172001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.172001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1141,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074368+0000","last_change":"2024-04-25T05:51:33.289789+0000","last_active":"2024-04-25T05:51:50.074368+0000","last_peered":"2024-04-25T05:51:50.074368+0000","last_clean":"2024-04-25T05:51:50.074368+0000","last_became_active":"2024-04-25T05:51:33.168828+0000","last_became_peered":"2024-04-25T05:51:33.168828+0000","last_unstale":"2024-04-25T05:51:50.074368+0000","last_undegraded":"2024-04-25T05:51:50.074368+0000","last_fullsized":"2024-04-25T05:51:50.074368+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.289669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.289669+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:57.938092+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"504'8263","reported_seq":8403,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:52.767679+0000","last_change":"2024-04-25T05:51:34.510387+0000","last_active":"2024-04-25T05:51:52.767679+0000","last_peered":"2024-04-25T05:51:52.767679+0000","last_clean":"2024-04-25T05:51:52.767679+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:51:52.767679+0000","last_undegraded":"2024-04-25T05:51:52.767679+0000","last_fullsized":"2024-04-25T05:51:52.767679+0000","mapping_epoch":456,"log_start":"471'2955","ondisk_log_start":"471'2955","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":2999,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521404440,"num_objects":7957,"num_object_clones":0,"num_object_copies":23871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7957,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8263,"num_write_kb":518977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":29,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090203+0000","last_change":"2024-04-25T05:51:40.124005+0000","last_active":"2024-04-25T05:51:50.090203+0000","last_peered":"2024-04-25T05:51:50.090203+0000","last_clean":"2024-04-25T05:51:50.090203+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:51:50.090203+0000","last_undegraded":"2024-04-25T05:51:50.090203+0000","last_fullsized":"2024-04-25T05:51:50.090203+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.678604+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":681,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072052+0000","last_change":"2024-04-25T05:51:32.949273+0000","last_active":"2024-04-25T05:51:50.072052+0000","last_peered":"2024-04-25T05:51:50.072052+0000","last_clean":"2024-04-25T05:51:50.072052+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:51:50.072052+0000","last_undegraded":"2024-04-25T05:51:50.072052+0000","last_fullsized":"2024-04-25T05:51:50.072052+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:51:32.949135+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:51:32.949135+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:43.062052+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074197+0000","last_change":"2024-04-25T05:50:08.761599+0000","last_active":"2024-04-25T05:51:50.074197+0000","last_peered":"2024-04-25T05:51:50.074197+0000","last_clean":"2024-04-25T05:51:50.074197+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:51:50.074197+0000","last_undegraded":"2024-04-25T05:51:50.074197+0000","last_fullsized":"2024-04-25T05:51:50.074197+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.761515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.761515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.086244+0000","last_change":"2024-04-25T05:51:40.094093+0000","last_active":"2024-04-25T05:51:50.086244+0000","last_peered":"2024-04-25T05:51:50.086244+0000","last_clean":"2024-04-25T05:51:50.086244+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:51:50.086244+0000","last_undegraded":"2024-04-25T05:51:50.086244+0000","last_fullsized":"2024-04-25T05:51:50.086244+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.086479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.086479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1141,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080931+0000","last_change":"2024-04-25T05:51:33.347884+0000","last_active":"2024-04-25T05:51:50.080931+0000","last_peered":"2024-04-25T05:51:50.080931+0000","last_clean":"2024-04-25T05:51:50.080931+0000","last_became_active":"2024-04-25T05:51:33.176805+0000","last_became_peered":"2024-04-25T05:51:33.176805+0000","last_unstale":"2024-04-25T05:51:50.080931+0000","last_undegraded":"2024-04-25T05:51:50.080931+0000","last_fullsized":"2024-04-25T05:51:50.080931+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.347743+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:35.643409+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"504'8482","reported_seq":8587,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:54.177648+0000","last_change":"2024-04-25T05:50:12.038131+0000","last_active":"2024-04-25T05:51:54.177648+0000","last_peered":"2024-04-25T05:51:54.177648+0000","last_clean":"2024-04-25T05:51:54.177648+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:51:54.177648+0000","last_undegraded":"2024-04-25T05:51:54.177648+0000","last_fullsized":"2024-04-25T05:51:54.177648+0000","mapping_epoch":456,"log_start":"474'4066","ondisk_log_start":"474'4066","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522059776,"num_objects":7966,"num_object_clones":0,"num_object_copies":23898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8482,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":698,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.077502+0000","last_change":"2024-04-25T05:51:40.105086+0000","last_active":"2024-04-25T05:51:50.077502+0000","last_peered":"2024-04-25T05:51:50.077502+0000","last_clean":"2024-04-25T05:51:50.077502+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:51:50.077502+0000","last_undegraded":"2024-04-25T05:51:50.077502+0000","last_fullsized":"2024-04-25T05:51:50.077502+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:42.148429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:50:42.148429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.140725+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":43,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090097+0000","last_change":"2024-04-25T05:51:40.143489+0000","last_active":"2024-04-25T05:51:50.090097+0000","last_peered":"2024-04-25T05:51:50.090097+0000","last_clean":"2024-04-25T05:51:50.090097+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:51:50.090097+0000","last_undegraded":"2024-04-25T05:51:50.090097+0000","last_fullsized":"2024-04-25T05:51:50.090097+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:48:59.056469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.056469+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":969,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075351+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:51:50.075351+0000","last_peered":"2024-04-25T05:51:50.075351+0000","last_clean":"2024-04-25T05:51:50.075351+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:51:50.075351+0000","last_undegraded":"2024-04-25T05:51:50.075351+0000","last_fullsized":"2024-04-25T05:51:50.075351+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.818280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":943,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074694+0000","last_change":"2024-04-25T05:49:32.024307+0000","last_active":"2024-04-25T05:51:50.074694+0000","last_peered":"2024-04-25T05:51:50.074694+0000","last_clean":"2024-04-25T05:51:50.074694+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:51:50.074694+0000","last_undegraded":"2024-04-25T05:51:50.074694+0000","last_fullsized":"2024-04-25T05:51:50.074694+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:32.024156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.024156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1160,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078482+0000","last_change":"2024-04-25T05:51:43.248181+0000","last_active":"2024-04-25T05:51:50.078482+0000","last_peered":"2024-04-25T05:51:50.078482+0000","last_clean":"2024-04-25T05:51:50.078482+0000","last_became_active":"2024-04-25T05:51:43.247802+0000","last_became_peered":"2024-04-25T05:51:43.247802+0000","last_unstale":"2024-04-25T05:51:50.078482+0000","last_undegraded":"2024-04-25T05:51:50.078482+0000","last_fullsized":"2024-04-25T05:51:50.078482+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:24.379318+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:51:24.379318+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.454385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"504'8551","reported_seq":8662,"reported_epoch":504,"state":"active+remapped+backfill_wait","last_fresh":"2024-04-25T05:51:54.198087+0000","last_change":"2024-04-25T05:51:41.458169+0000","last_active":"2024-04-25T05:51:54.198087+0000","last_peered":"2024-04-25T05:51:54.198087+0000","last_clean":"2024-04-25T05:51:39.187155+0000","last_became_active":"2024-04-25T05:51:41.446699+0000","last_became_peered":"2024-04-25T05:51:41.446699+0000","last_unstale":"2024-04-25T05:51:54.198087+0000","last_undegraded":"2024-04-25T05:51:54.198087+0000","last_fullsized":"2024-04-25T05:51:54.198087+0000","mapping_epoch":498,"log_start":"476'4243","ondisk_log_start":"476'4243","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523042816,"num_objects":7981,"num_object_clones":0,"num_object_copies":23943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7981,"num_objects_unfound":0,"num_objects_dirty":7981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8551,"num_write_kb":529024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":7981}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":271,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074765+0000","last_change":"2024-04-25T05:51:40.105730+0000","last_active":"2024-04-25T05:51:50.074765+0000","last_peered":"2024-04-25T05:51:50.074765+0000","last_clean":"2024-04-25T05:51:50.074765+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:51:50.074765+0000","last_undegraded":"2024-04-25T05:51:50.074765+0000","last_fullsized":"2024-04-25T05:51:50.074765+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:18.888403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.888403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:36.188492+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":279,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073703+0000","last_change":"2024-04-25T05:50:12.786445+0000","last_active":"2024-04-25T05:51:50.073703+0000","last_peered":"2024-04-25T05:51:50.073703+0000","last_clean":"2024-04-25T05:51:50.073703+0000","last_became_active":"2024-04-25T05:46:34.986336+0000","last_became_peered":"2024-04-25T05:46:34.986336+0000","last_unstale":"2024-04-25T05:51:50.073703+0000","last_undegraded":"2024-04-25T05:51:50.073703+0000","last_fullsized":"2024-04-25T05:51:50.073703+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:50:12.786333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:12.786333+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2056,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:52.097353+0000","last_change":"2024-04-25T05:51:40.120019+0000","last_active":"2024-04-25T05:51:52.097353+0000","last_peered":"2024-04-25T05:51:52.097353+0000","last_clean":"2024-04-25T05:51:52.097353+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:51:52.097353+0000","last_undegraded":"2024-04-25T05:51:52.097353+0000","last_fullsized":"2024-04-25T05:51:52.097353+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:51:29.129304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.129304+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:57.335001+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1055,"num_read_kb":1050,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089809+0000","last_change":"2024-04-25T05:51:40.097878+0000","last_active":"2024-04-25T05:51:50.089809+0000","last_peered":"2024-04-25T05:51:50.089809+0000","last_clean":"2024-04-25T05:51:50.089809+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:51:50.089809+0000","last_undegraded":"2024-04-25T05:51:50.089809+0000","last_fullsized":"2024-04-25T05:51:50.089809+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.488514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.488514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":939,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075023+0000","last_change":"2024-04-25T05:51:22.949696+0000","last_active":"2024-04-25T05:51:50.075023+0000","last_peered":"2024-04-25T05:51:50.075023+0000","last_clean":"2024-04-25T05:51:50.075023+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:51:50.075023+0000","last_undegraded":"2024-04-25T05:51:50.075023+0000","last_fullsized":"2024-04-25T05:51:50.075023+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.949432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.949432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:33.813758+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1137,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075320+0000","last_change":"2024-04-25T05:51:33.980181+0000","last_active":"2024-04-25T05:51:50.075320+0000","last_peered":"2024-04-25T05:51:50.075320+0000","last_clean":"2024-04-25T05:51:50.075320+0000","last_became_active":"2024-04-25T05:51:33.176526+0000","last_became_peered":"2024-04-25T05:51:33.176526+0000","last_unstale":"2024-04-25T05:51:50.075320+0000","last_undegraded":"2024-04-25T05:51:50.075320+0000","last_fullsized":"2024-04-25T05:51:50.075320+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.979836+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.979836+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:38.405079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"504'8225","reported_seq":8331,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:54.166548+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:51:54.166548+0000","last_peered":"2024-04-25T05:51:54.166548+0000","last_clean":"2024-04-25T05:51:54.166548+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:51:54.166548+0000","last_undegraded":"2024-04-25T05:51:54.166548+0000","last_fullsized":"2024-04-25T05:51:54.166548+0000","mapping_epoch":456,"log_start":"475'3954","ondisk_log_start":"475'3954","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508100608,"num_objects":7753,"num_object_clones":0,"num_object_copies":23259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8225,"num_write_kb":511296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":961,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.082216+0000","last_change":"2024-04-25T05:51:24.036146+0000","last_active":"2024-04-25T05:51:50.082216+0000","last_peered":"2024-04-25T05:51:50.082216+0000","last_clean":"2024-04-25T05:51:50.082216+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:51:50.082216+0000","last_undegraded":"2024-04-25T05:51:50.082216+0000","last_fullsized":"2024-04-25T05:51:50.082216+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:24.035990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:51:24.035990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:45.482411+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":846,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080757+0000","last_change":"2024-04-25T05:50:13.028112+0000","last_active":"2024-04-25T05:51:50.080757+0000","last_peered":"2024-04-25T05:51:50.080757+0000","last_clean":"2024-04-25T05:51:50.080757+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:51:50.080757+0000","last_undegraded":"2024-04-25T05:51:50.080757+0000","last_fullsized":"2024-04-25T05:51:50.080757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.027901+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":948,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081492+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:51:50.081492+0000","last_peered":"2024-04-25T05:51:50.081492+0000","last_clean":"2024-04-25T05:51:50.081492+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:51:50.081492+0000","last_undegraded":"2024-04-25T05:51:50.081492+0000","last_fullsized":"2024-04-25T05:51:50.081492+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1273,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:51.224042+0000","last_change":"2024-04-25T05:51:51.224083+0000","last_active":"2024-04-25T05:51:51.224042+0000","last_peered":"2024-04-25T05:51:51.224042+0000","last_clean":"2024-04-25T05:51:51.224042+0000","last_became_active":"2024-04-25T05:51:43.247617+0000","last_became_peered":"2024-04-25T05:51:43.247617+0000","last_unstale":"2024-04-25T05:51:51.224042+0000","last_undegraded":"2024-04-25T05:51:51.224042+0000","last_fullsized":"2024-04-25T05:51:51.224042+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:51.223847+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:51:51.223847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:19.894983+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075810+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:51:50.075810+0000","last_peered":"2024-04-25T05:51:50.075810+0000","last_clean":"2024-04-25T05:51:50.075810+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:51:50.075810+0000","last_undegraded":"2024-04-25T05:51:50.075810+0000","last_fullsized":"2024-04-25T05:51:50.075810+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.250098+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"504'8398","reported_seq":8504,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:54.170238+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:51:54.170238+0000","last_peered":"2024-04-25T05:51:54.170238+0000","last_clean":"2024-04-25T05:51:54.170238+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:51:54.170238+0000","last_undegraded":"2024-04-25T05:51:54.170238+0000","last_fullsized":"2024-04-25T05:51:54.170238+0000","mapping_epoch":456,"log_start":"476'4128","ondisk_log_start":"476'4128","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519176192,"num_objects":7922,"num_object_clones":0,"num_object_copies":23766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":296,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.083090+0000","last_change":"2024-04-25T05:50:45.550588+0000","last_active":"2024-04-25T05:51:50.083090+0000","last_peered":"2024-04-25T05:51:50.083090+0000","last_clean":"2024-04-25T05:51:50.083090+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:51:50.083090+0000","last_undegraded":"2024-04-25T05:51:50.083090+0000","last_fullsized":"2024-04-25T05:51:50.083090+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.550474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.550474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.043449+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1098,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075452+0000","last_change":"2024-04-25T05:51:40.157376+0000","last_active":"2024-04-25T05:51:50.075452+0000","last_peered":"2024-04-25T05:51:50.075452+0000","last_clean":"2024-04-25T05:51:50.075452+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:51:50.075452+0000","last_undegraded":"2024-04-25T05:51:50.075452+0000","last_fullsized":"2024-04-25T05:51:50.075452+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:50:40.134604+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.134604+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:58.655315+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":296,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.079272+0000","last_change":"2024-04-25T05:51:40.107136+0000","last_active":"2024-04-25T05:51:50.079272+0000","last_peered":"2024-04-25T05:51:50.079272+0000","last_clean":"2024-04-25T05:51:50.079272+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:51:50.079272+0000","last_undegraded":"2024-04-25T05:51:50.079272+0000","last_fullsized":"2024-04-25T05:51:50.079272+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.628364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.628364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":975,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.069542+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:51:50.069542+0000","last_peered":"2024-04-25T05:51:50.069542+0000","last_clean":"2024-04-25T05:51:50.069542+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:51:50.069542+0000","last_undegraded":"2024-04-25T05:51:50.069542+0000","last_fullsized":"2024-04-25T05:51:50.069542+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.323525+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":939,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.083040+0000","last_change":"2024-04-25T05:51:17.037477+0000","last_active":"2024-04-25T05:51:50.083040+0000","last_peered":"2024-04-25T05:51:50.083040+0000","last_clean":"2024-04-25T05:51:50.083040+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:51:50.083040+0000","last_undegraded":"2024-04-25T05:51:50.083040+0000","last_fullsized":"2024-04-25T05:51:50.083040+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:17.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:43:30.152081+0000","last_clean_scrub_stamp":"2024-04-25T05:51:17.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:26.928401+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1281,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081296+0000","last_change":"2024-04-25T05:51:43.250000+0000","last_active":"2024-04-25T05:51:50.081296+0000","last_peered":"2024-04-25T05:51:50.081296+0000","last_clean":"2024-04-25T05:51:50.081296+0000","last_became_active":"2024-04-25T05:51:43.249544+0000","last_became_peered":"2024-04-25T05:51:43.249544+0000","last_unstale":"2024-04-25T05:51:50.081296+0000","last_undegraded":"2024-04-25T05:51:50.081296+0000","last_fullsized":"2024-04-25T05:51:50.081296+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:51:34.121865+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:59.466953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"504'8406","reported_seq":8518,"reported_epoch":504,"state":"active+remapped+backfill_toofull","last_fresh":"2024-04-25T05:51:54.197303+0000","last_change":"2024-04-25T05:51:41.430912+0000","last_active":"2024-04-25T05:51:54.197303+0000","last_peered":"2024-04-25T05:51:54.197303+0000","last_clean":"2024-04-25T05:51:39.176979+0000","last_became_active":"2024-04-25T05:51:41.418878+0000","last_became_peered":"2024-04-25T05:51:41.418878+0000","last_unstale":"2024-04-25T05:51:54.197303+0000","last_undegraded":"2024-04-25T05:51:54.197303+0000","last_fullsized":"2024-04-25T05:51:54.197303+0000","mapping_epoch":498,"log_start":"475'4172","ondisk_log_start":"475'4172","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516816896,"num_objects":7886,"num_object_clones":0,"num_object_copies":23658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7886,"num_objects_unfound":0,"num_objects_dirty":7886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8406,"num_write_kb":521344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":7886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089499+0000","last_change":"2024-04-25T05:51:40.098006+0000","last_active":"2024-04-25T05:51:50.089499+0000","last_peered":"2024-04-25T05:51:50.089499+0000","last_clean":"2024-04-25T05:51:50.089499+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:51:50.089499+0000","last_undegraded":"2024-04-25T05:51:50.089499+0000","last_fullsized":"2024-04-25T05:51:50.089499+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:21.946445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:21.946445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:24.637301+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1292,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075478+0000","last_change":"2024-04-25T05:50:07.985223+0000","last_active":"2024-04-25T05:51:50.075478+0000","last_peered":"2024-04-25T05:51:50.075478+0000","last_clean":"2024-04-25T05:51:50.075478+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:51:50.075478+0000","last_undegraded":"2024-04-25T05:51:50.075478+0000","last_fullsized":"2024-04-25T05:51:50.075478+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:50:07.985110+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.985110+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":514,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.680940+0000","last_change":"2024-04-25T05:51:50.680986+0000","last_active":"2024-04-25T05:51:50.680940+0000","last_peered":"2024-04-25T05:51:50.680940+0000","last_clean":"2024-04-25T05:51:50.680940+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:51:50.680940+0000","last_undegraded":"2024-04-25T05:51:50.680940+0000","last_fullsized":"2024-04-25T05:51:50.680940+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:50.680590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:51:50.680590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:19.648503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":269,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072966+0000","last_change":"2024-04-25T05:50:04.735739+0000","last_active":"2024-04-25T05:51:50.072966+0000","last_peered":"2024-04-25T05:51:50.072966+0000","last_clean":"2024-04-25T05:51:50.072966+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:51:50.072966+0000","last_undegraded":"2024-04-25T05:51:50.072966+0000","last_fullsized":"2024-04-25T05:51:50.072966+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T05:50:04.735651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.735651+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":912,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080623+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:51:50.080623+0000","last_peered":"2024-04-25T05:51:50.080623+0000","last_clean":"2024-04-25T05:51:50.080623+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:51:50.080623+0000","last_undegraded":"2024-04-25T05:51:50.080623+0000","last_fullsized":"2024-04-25T05:51:50.080623+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":982,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072303+0000","last_change":"2024-04-25T05:51:43.239239+0000","last_active":"2024-04-25T05:51:50.072303+0000","last_peered":"2024-04-25T05:51:50.072303+0000","last_clean":"2024-04-25T05:51:50.072303+0000","last_became_active":"2024-04-25T05:51:43.238847+0000","last_became_peered":"2024-04-25T05:51:43.238847+0000","last_unstale":"2024-04-25T05:51:50.072303+0000","last_undegraded":"2024-04-25T05:51:50.072303+0000","last_fullsized":"2024-04-25T05:51:50.072303+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.881703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.881703+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:57.942392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"79'128","reported_seq":793,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091167+0000","last_change":"2024-04-25T05:51:49.705797+0000","last_active":"2024-04-25T05:51:50.091167+0000","last_peered":"2024-04-25T05:51:50.091167+0000","last_clean":"2024-04-25T05:51:50.091167+0000","last_became_active":"2024-04-25T05:51:38.084985+0000","last_became_peered":"2024-04-25T05:51:38.084985+0000","last_unstale":"2024-04-25T05:51:50.091167+0000","last_undegraded":"2024-04-25T05:51:50.091167+0000","last_fullsized":"2024-04-25T05:51:50.091167+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:49.705539+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:51:49.705539+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:06.530351+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"504'8333","reported_seq":8438,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.288019+0000","last_change":"2024-04-25T05:50:12.043881+0000","last_active":"2024-04-25T05:51:53.288019+0000","last_peered":"2024-04-25T05:51:53.288019+0000","last_clean":"2024-04-25T05:51:53.288019+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:51:53.288019+0000","last_undegraded":"2024-04-25T05:51:53.288019+0000","last_fullsized":"2024-04-25T05:51:53.288019+0000","mapping_epoch":456,"log_start":"475'3886","ondisk_log_start":"475'3886","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4447,"log_dups_size":0,"ondisk_log_size":4447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521076736,"num_objects":7951,"num_object_clones":0,"num_object_copies":23853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8333,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":35,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089890+0000","last_change":"2024-04-25T05:51:40.140728+0000","last_active":"2024-04-25T05:51:50.089890+0000","last_peered":"2024-04-25T05:51:50.089890+0000","last_clean":"2024-04-25T05:51:50.089890+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:51:50.089890+0000","last_undegraded":"2024-04-25T05:51:50.089890+0000","last_fullsized":"2024-04-25T05:51:50.089890+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073792+0000","last_change":"2024-04-25T05:51:23.930630+0000","last_active":"2024-04-25T05:51:50.073792+0000","last_peered":"2024-04-25T05:51:50.073792+0000","last_clean":"2024-04-25T05:51:50.073792+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:51:50.073792+0000","last_undegraded":"2024-04-25T05:51:50.073792+0000","last_fullsized":"2024-04-25T05:51:50.073792+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:23.930498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.930498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:30.691099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":290,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.084429+0000","last_change":"2024-04-25T05:51:28.554329+0000","last_active":"2024-04-25T05:51:50.084429+0000","last_peered":"2024-04-25T05:51:50.084429+0000","last_clean":"2024-04-25T05:51:50.084429+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:51:50.084429+0000","last_undegraded":"2024-04-25T05:51:50.084429+0000","last_fullsized":"2024-04-25T05:51:50.084429+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.554087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.554087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:40.690982+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089507+0000","last_change":"2024-04-25T05:51:40.099599+0000","last_active":"2024-04-25T05:51:50.089507+0000","last_peered":"2024-04-25T05:51:50.089507+0000","last_clean":"2024-04-25T05:51:50.089507+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:51:50.089507+0000","last_undegraded":"2024-04-25T05:51:50.089507+0000","last_fullsized":"2024-04-25T05:51:50.089507+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.026628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.026628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:19.748509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"79'128","reported_seq":1149,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078808+0000","last_change":"2024-04-25T05:51:38.083307+0000","last_active":"2024-04-25T05:51:50.078808+0000","last_peered":"2024-04-25T05:51:50.078808+0000","last_clean":"2024-04-25T05:51:50.078808+0000","last_became_active":"2024-04-25T05:51:38.082878+0000","last_became_peered":"2024-04-25T05:51:38.082878+0000","last_unstale":"2024-04-25T05:51:50.078808+0000","last_undegraded":"2024-04-25T05:51:50.078808+0000","last_fullsized":"2024-04-25T05:51:50.078808+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:34.402045+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:27.431762+0000","last_clean_scrub_stamp":"2024-04-25T05:51:34.402045+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:39.361404+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"504'8538","reported_seq":8643,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.371055+0000","last_change":"2024-04-25T05:50:12.042685+0000","last_active":"2024-04-25T05:51:53.371055+0000","last_peered":"2024-04-25T05:51:53.371055+0000","last_clean":"2024-04-25T05:51:53.371055+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:51:53.371055+0000","last_undegraded":"2024-04-25T05:51:53.371055+0000","last_fullsized":"2024-04-25T05:51:53.371055+0000","mapping_epoch":456,"log_start":"475'4109","ondisk_log_start":"475'4109","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531103744,"num_objects":8104,"num_object_clones":0,"num_object_copies":24312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":718,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.077289+0000","last_change":"2024-04-25T05:50:15.942029+0000","last_active":"2024-04-25T05:51:50.077289+0000","last_peered":"2024-04-25T05:51:50.077289+0000","last_clean":"2024-04-25T05:51:50.077289+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:51:50.077289+0000","last_undegraded":"2024-04-25T05:51:50.077289+0000","last_fullsized":"2024-04-25T05:51:50.077289+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.941896+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":985,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.082891+0000","last_change":"2024-04-25T05:50:47.124683+0000","last_active":"2024-04-25T05:51:50.082891+0000","last_peered":"2024-04-25T05:51:50.082891+0000","last_clean":"2024-04-25T05:51:50.082891+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:51:50.082891+0000","last_undegraded":"2024-04-25T05:51:50.082891+0000","last_fullsized":"2024-04-25T05:51:50.082891+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.124554+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:04.362413+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":302,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078666+0000","last_change":"2024-04-25T05:51:40.105669+0000","last_active":"2024-04-25T05:51:50.078666+0000","last_peered":"2024-04-25T05:51:50.078666+0000","last_clean":"2024-04-25T05:51:50.078666+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:51:50.078666+0000","last_undegraded":"2024-04-25T05:51:50.078666+0000","last_fullsized":"2024-04-25T05:51:50.078666+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:31.543423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:31.543423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:00.374852+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":939,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080291+0000","last_change":"2024-04-25T05:51:32.113005+0000","last_active":"2024-04-25T05:51:50.080291+0000","last_peered":"2024-04-25T05:51:50.080291+0000","last_clean":"2024-04-25T05:51:50.080291+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:51:50.080291+0000","last_undegraded":"2024-04-25T05:51:50.080291+0000","last_fullsized":"2024-04-25T05:51:50.080291+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:32.112759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:32.112759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:38.214717+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"79'128","reported_seq":1160,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074269+0000","last_change":"2024-04-25T05:51:43.253589+0000","last_active":"2024-04-25T05:51:50.074269+0000","last_peered":"2024-04-25T05:51:50.074269+0000","last_clean":"2024-04-25T05:51:50.074269+0000","last_became_active":"2024-04-25T05:51:43.253153+0000","last_became_peered":"2024-04-25T05:51:43.253153+0000","last_unstale":"2024-04-25T05:51:50.074269+0000","last_undegraded":"2024-04-25T05:51:50.074269+0000","last_fullsized":"2024-04-25T05:51:50.074269+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:48:27.160639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:48:27.160639+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.286809+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":873814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"504'8285","reported_seq":11196,"reported_epoch":504,"state":"active+remapped+backfilling","last_fresh":"2024-04-25T05:51:54.184084+0000","last_change":"2024-04-25T05:51:41.456905+0000","last_active":"2024-04-25T05:51:54.184084+0000","last_peered":"2024-04-25T05:51:54.184084+0000","last_clean":"2024-04-25T05:51:39.162148+0000","last_became_active":"2024-04-25T05:51:41.438180+0000","last_became_peered":"2024-04-25T05:51:41.438180+0000","last_unstale":"2024-04-25T05:51:54.184084+0000","last_undegraded":"2024-04-25T05:51:54.184084+0000","last_fullsized":"2024-04-25T05:51:54.184084+0000","mapping_epoch":498,"log_start":"475'3858","ondisk_log_start":"475'3858","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4427,"log_dups_size":0,"ondisk_log_size":4427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507445248,"num_objects":7743,"num_object_clones":0,"num_object_copies":23229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6334,"num_objects_unfound":0,"num_objects_dirty":7743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8285,"num_write_kb":512896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1401,"num_bytes_recovered":91815936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2,4],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":7743}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":302,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078272+0000","last_change":"2024-04-25T05:51:40.106790+0000","last_active":"2024-04-25T05:51:50.078272+0000","last_peered":"2024-04-25T05:51:50.078272+0000","last_clean":"2024-04-25T05:51:50.078272+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:51:50.078272+0000","last_undegraded":"2024-04-25T05:51:50.078272+0000","last_fullsized":"2024-04-25T05:51:50.078272+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.566717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.566717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":355,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:52.684448+0000","last_change":"2024-04-25T05:51:52.684484+0000","last_active":"2024-04-25T05:51:52.684448+0000","last_peered":"2024-04-25T05:51:52.684448+0000","last_clean":"2024-04-25T05:51:52.684448+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:51:52.684448+0000","last_undegraded":"2024-04-25T05:51:52.684448+0000","last_fullsized":"2024-04-25T05:51:52.684448+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:51:52.684385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:52.684385+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:02.417234+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":938,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.083674+0000","last_change":"2024-04-25T05:50:41.101003+0000","last_active":"2024-04-25T05:51:50.083674+0000","last_peered":"2024-04-25T05:51:50.083674+0000","last_clean":"2024-04-25T05:51:50.083674+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:51:50.083674+0000","last_undegraded":"2024-04-25T05:51:50.083674+0000","last_fullsized":"2024-04-25T05:51:50.083674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:41.100880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.100880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":923,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074520+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:51:50.074520+0000","last_peered":"2024-04-25T05:51:50.074520+0000","last_clean":"2024-04-25T05:51:50.074520+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:51:50.074520+0000","last_undegraded":"2024-04-25T05:51:50.074520+0000","last_fullsized":"2024-04-25T05:51:50.074520+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:03.671987+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"79'128","reported_seq":924,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.195259+0000","last_change":"2024-04-25T05:51:50.195303+0000","last_active":"2024-04-25T05:51:50.195259+0000","last_peered":"2024-04-25T05:51:50.195259+0000","last_clean":"2024-04-25T05:51:50.195259+0000","last_became_active":"2024-04-25T05:51:38.089237+0000","last_became_peered":"2024-04-25T05:51:38.089237+0000","last_unstale":"2024-04-25T05:51:50.195259+0000","last_undegraded":"2024-04-25T05:51:50.195259+0000","last_fullsized":"2024-04-25T05:51:50.195259+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:50.194941+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:51:50.194941+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:51.721072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"504'8296","reported_seq":8401,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:53.107564+0000","last_change":"2024-04-25T05:50:12.042051+0000","last_active":"2024-04-25T05:51:53.107564+0000","last_peered":"2024-04-25T05:51:53.107564+0000","last_clean":"2024-04-25T05:51:53.107564+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:51:53.107564+0000","last_undegraded":"2024-04-25T05:51:53.107564+0000","last_fullsized":"2024-04-25T05:51:53.107564+0000","mapping_epoch":456,"log_start":"472'3435","ondisk_log_start":"472'3435","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":516292608,"num_objects":7878,"num_object_clones":0,"num_object_copies":23634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1181,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.079584+0000","last_change":"2024-04-25T05:50:39.051287+0000","last_active":"2024-04-25T05:51:50.079584+0000","last_peered":"2024-04-25T05:51:50.079584+0000","last_clean":"2024-04-25T05:51:50.079584+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:51:50.079584+0000","last_undegraded":"2024-04-25T05:51:50.079584+0000","last_fullsized":"2024-04-25T05:51:50.079584+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:50:39.050944+0000","last_deep_scrub":"118'9","last_deep_scrub_stamp":"2024-04-25T05:36:29.348920+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.050944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:07.882347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":285,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074756+0000","last_change":"2024-04-25T05:50:11.779506+0000","last_active":"2024-04-25T05:51:50.074756+0000","last_peered":"2024-04-25T05:51:50.074756+0000","last_clean":"2024-04-25T05:51:50.074756+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:51:50.074756+0000","last_undegraded":"2024-04-25T05:51:50.074756+0000","last_fullsized":"2024-04-25T05:51:50.074756+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:50:11.779355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.779355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":998,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.083325+0000","last_change":"2024-04-25T05:51:47.204028+0000","last_active":"2024-04-25T05:51:50.083325+0000","last_peered":"2024-04-25T05:51:50.083325+0000","last_clean":"2024-04-25T05:51:50.083325+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:51:50.083325+0000","last_undegraded":"2024-04-25T05:51:50.083325+0000","last_fullsized":"2024-04-25T05:51:50.083325+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:51:47.203851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:47.203851+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:54.358855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":263,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074898+0000","last_change":"2024-04-25T05:51:40.084319+0000","last_active":"2024-04-25T05:51:50.074898+0000","last_peered":"2024-04-25T05:51:50.074898+0000","last_clean":"2024-04-25T05:51:50.074898+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:51:50.074898+0000","last_undegraded":"2024-04-25T05:51:50.074898+0000","last_fullsized":"2024-04-25T05:51:50.074898+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:01.017406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.017406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"79'128","reported_seq":1020,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072554+0000","last_change":"2024-04-25T05:51:49.918140+0000","last_active":"2024-04-25T05:51:50.072554+0000","last_peered":"2024-04-25T05:51:50.072554+0000","last_clean":"2024-04-25T05:51:50.072554+0000","last_became_active":"2024-04-25T05:51:38.084725+0000","last_became_peered":"2024-04-25T05:51:38.084725+0000","last_unstale":"2024-04-25T05:51:50.072554+0000","last_undegraded":"2024-04-25T05:51:50.072554+0000","last_fullsized":"2024-04-25T05:51:50.072554+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:49.917843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:49.917843+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:59.352607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":957,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080909+0000","last_change":"2024-04-25T05:51:28.373915+0000","last_active":"2024-04-25T05:51:50.080909+0000","last_peered":"2024-04-25T05:51:50.080909+0000","last_clean":"2024-04-25T05:51:50.080909+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:51:50.080909+0000","last_undegraded":"2024-04-25T05:51:50.080909+0000","last_fullsized":"2024-04-25T05:51:50.080909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:51:28.373733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.373733+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:32.873549+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":698,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.077048+0000","last_change":"2024-04-25T05:51:40.105548+0000","last_active":"2024-04-25T05:51:50.077048+0000","last_peered":"2024-04-25T05:51:50.077048+0000","last_clean":"2024-04-25T05:51:50.077048+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:51:50.077048+0000","last_undegraded":"2024-04-25T05:51:50.077048+0000","last_fullsized":"2024-04-25T05:51:50.077048+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.105913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.105913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:51.847737+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073872+0000","last_change":"2024-04-25T05:50:43.938330+0000","last_active":"2024-04-25T05:51:50.073872+0000","last_peered":"2024-04-25T05:51:50.073872+0000","last_clean":"2024-04-25T05:51:50.073872+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:51:50.073872+0000","last_undegraded":"2024-04-25T05:51:50.073872+0000","last_fullsized":"2024-04-25T05:51:50.073872+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:43.938246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.938246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:52.848134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":947,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081703+0000","last_change":"2024-04-25T05:51:48.376273+0000","last_active":"2024-04-25T05:51:50.081703+0000","last_peered":"2024-04-25T05:51:50.081703+0000","last_clean":"2024-04-25T05:51:50.081703+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:51:50.081703+0000","last_undegraded":"2024-04-25T05:51:50.081703+0000","last_fullsized":"2024-04-25T05:51:50.081703+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:48.375877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:48.375877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:54.578657+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"79'128","reported_seq":1137,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075687+0000","last_change":"2024-04-25T05:51:38.097014+0000","last_active":"2024-04-25T05:51:50.075687+0000","last_peered":"2024-04-25T05:51:50.075687+0000","last_clean":"2024-04-25T05:51:50.075687+0000","last_became_active":"2024-04-25T05:51:38.096570+0000","last_became_peered":"2024-04-25T05:51:38.096570+0000","last_unstale":"2024-04-25T05:51:50.075687+0000","last_undegraded":"2024-04-25T05:51:50.075687+0000","last_fullsized":"2024-04-25T05:51:50.075687+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.990556+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.361374+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":37,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089833+0000","last_change":"2024-04-25T05:51:40.154295+0000","last_active":"2024-04-25T05:51:50.089833+0000","last_peered":"2024-04-25T05:51:50.089833+0000","last_clean":"2024-04-25T05:51:50.089833+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:51:50.089833+0000","last_undegraded":"2024-04-25T05:51:50.089833+0000","last_fullsized":"2024-04-25T05:51:50.089833+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:49:32.097448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:49:32.097448+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":956,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.079786+0000","last_change":"2024-04-25T05:50:53.041228+0000","last_active":"2024-04-25T05:51:50.079786+0000","last_peered":"2024-04-25T05:51:50.079786+0000","last_clean":"2024-04-25T05:51:50.079786+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:51:50.079786+0000","last_undegraded":"2024-04-25T05:51:50.079786+0000","last_fullsized":"2024-04-25T05:51:50.079786+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:53.041118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:50:53.041118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.078933+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":915,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075646+0000","last_change":"2024-04-25T05:51:30.313765+0000","last_active":"2024-04-25T05:51:50.075646+0000","last_peered":"2024-04-25T05:51:50.075646+0000","last_clean":"2024-04-25T05:51:50.075646+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:51:50.075646+0000","last_undegraded":"2024-04-25T05:51:50.075646+0000","last_fullsized":"2024-04-25T05:51:50.075646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.313655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.313655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:56.653238+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":282,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081079+0000","last_change":"2024-04-25T05:50:13.581116+0000","last_active":"2024-04-25T05:51:50.081079+0000","last_peered":"2024-04-25T05:51:50.081079+0000","last_clean":"2024-04-25T05:51:50.081079+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:51:50.081079+0000","last_undegraded":"2024-04-25T05:51:50.081079+0000","last_fullsized":"2024-04-25T05:51:50.081079+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.581028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.581028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"79'128","reported_seq":900,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074410+0000","last_change":"2024-04-25T05:51:48.880550+0000","last_active":"2024-04-25T05:51:50.074410+0000","last_peered":"2024-04-25T05:51:50.074410+0000","last_clean":"2024-04-25T05:51:50.074410+0000","last_became_active":"2024-04-25T05:51:38.083334+0000","last_became_peered":"2024-04-25T05:51:38.083334+0000","last_unstale":"2024-04-25T05:51:50.074410+0000","last_undegraded":"2024-04-25T05:51:50.074410+0000","last_fullsized":"2024-04-25T05:51:50.074410+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:48.880238+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:51:48.880238+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:53.132384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":277,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073804+0000","last_change":"2024-04-25T05:51:30.900773+0000","last_active":"2024-04-25T05:51:50.073804+0000","last_peered":"2024-04-25T05:51:50.073804+0000","last_clean":"2024-04-25T05:51:50.073804+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:51:50.073804+0000","last_undegraded":"2024-04-25T05:51:50.073804+0000","last_fullsized":"2024-04-25T05:51:50.073804+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:51:30.900430+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.900430+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:40.456977+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.092266+0000","last_change":"2024-04-25T05:51:40.099353+0000","last_active":"2024-04-25T05:51:50.092266+0000","last_peered":"2024-04-25T05:51:50.092266+0000","last_clean":"2024-04-25T05:51:50.092266+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:51:50.092266+0000","last_undegraded":"2024-04-25T05:51:50.092266+0000","last_fullsized":"2024-04-25T05:51:50.092266+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:01.779110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:48:01.779110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1159,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081232+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:51:50.081232+0000","last_peered":"2024-04-25T05:51:50.081232+0000","last_clean":"2024-04-25T05:51:50.081232+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:51:50.081232+0000","last_undegraded":"2024-04-25T05:51:50.081232+0000","last_fullsized":"2024-04-25T05:51:50.081232+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:09.466618+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":478,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090726+0000","last_change":"2024-04-25T05:50:52.078015+0000","last_active":"2024-04-25T05:51:50.090726+0000","last_peered":"2024-04-25T05:51:50.090726+0000","last_clean":"2024-04-25T05:51:50.090726+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:51:50.090726+0000","last_undegraded":"2024-04-25T05:51:50.090726+0000","last_fullsized":"2024-04-25T05:51:50.090726+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.077914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.221398+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"79'128","reported_seq":961,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089621+0000","last_change":"2024-04-25T05:51:43.253285+0000","last_active":"2024-04-25T05:51:50.089621+0000","last_peered":"2024-04-25T05:51:50.089621+0000","last_clean":"2024-04-25T05:51:50.089621+0000","last_became_active":"2024-04-25T05:51:43.252912+0000","last_became_peered":"2024-04-25T05:51:43.252912+0000","last_unstale":"2024-04-25T05:51:50.089621+0000","last_undegraded":"2024-04-25T05:51:50.089621+0000","last_fullsized":"2024-04-25T05:51:50.089621+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:49:00.008984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:49:00.008984+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:54.610567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":973,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074625+0000","last_change":"2024-04-25T05:48:07.299633+0000","last_active":"2024-04-25T05:51:50.074625+0000","last_peered":"2024-04-25T05:51:50.074625+0000","last_clean":"2024-04-25T05:51:50.074625+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:51:50.074625+0000","last_undegraded":"2024-04-25T05:51:50.074625+0000","last_fullsized":"2024-04-25T05:51:50.074625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:48:07.299486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:07.299486+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":950,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080180+0000","last_change":"2024-04-25T05:51:31.094777+0000","last_active":"2024-04-25T05:51:50.080180+0000","last_peered":"2024-04-25T05:51:50.080180+0000","last_clean":"2024-04-25T05:51:50.080180+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:51:50.080180+0000","last_undegraded":"2024-04-25T05:51:50.080180+0000","last_fullsized":"2024-04-25T05:51:50.080180+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:31.094706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:31.094706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:50.500538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":334,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073998+0000","last_change":"2024-04-25T05:51:40.149640+0000","last_active":"2024-04-25T05:51:50.073998+0000","last_peered":"2024-04-25T05:51:50.073998+0000","last_clean":"2024-04-25T05:51:50.073998+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:51:50.073998+0000","last_undegraded":"2024-04-25T05:51:50.073998+0000","last_fullsized":"2024-04-25T05:51:50.073998+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'13","last_scrub_stamp":"2024-04-25T05:49:01.967348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:01.967348+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090043+0000","last_change":"2024-04-25T05:51:40.110100+0000","last_active":"2024-04-25T05:51:50.090043+0000","last_peered":"2024-04-25T05:51:50.090043+0000","last_clean":"2024-04-25T05:51:50.090043+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:51:50.090043+0000","last_undegraded":"2024-04-25T05:51:50.090043+0000","last_fullsized":"2024-04-25T05:51:50.090043+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.972698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.972698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:32.873419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1077,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091146+0000","last_change":"2024-04-25T05:51:23.777966+0000","last_active":"2024-04-25T05:51:50.091146+0000","last_peered":"2024-04-25T05:51:50.091146+0000","last_clean":"2024-04-25T05:51:50.091146+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:51:50.091146+0000","last_undegraded":"2024-04-25T05:51:50.091146+0000","last_fullsized":"2024-04-25T05:51:50.091146+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.777815+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:29.449156+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":524,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.092225+0000","last_change":"2024-04-25T05:49:35.656164+0000","last_active":"2024-04-25T05:51:50.092225+0000","last_peered":"2024-04-25T05:51:50.092225+0000","last_clean":"2024-04-25T05:51:50.092225+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:51:50.092225+0000","last_undegraded":"2024-04-25T05:51:50.092225+0000","last_fullsized":"2024-04-25T05:51:50.092225+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:49:35.656070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.656070+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":937,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.071957+0000","last_change":"2024-04-25T05:50:13.065576+0000","last_active":"2024-04-25T05:51:50.071957+0000","last_peered":"2024-04-25T05:51:50.071957+0000","last_clean":"2024-04-25T05:51:50.071957+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:51:50.071957+0000","last_undegraded":"2024-04-25T05:51:50.071957+0000","last_fullsized":"2024-04-25T05:51:50.071957+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.065464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.065464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":961,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073260+0000","last_change":"2024-04-25T05:51:40.083901+0000","last_active":"2024-04-25T05:51:50.073260+0000","last_peered":"2024-04-25T05:51:50.073260+0000","last_clean":"2024-04-25T05:51:50.073260+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:51:50.073260+0000","last_undegraded":"2024-04-25T05:51:50.073260+0000","last_fullsized":"2024-04-25T05:51:50.073260+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":492,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072494+0000","last_change":"2024-04-25T05:51:23.925876+0000","last_active":"2024-04-25T05:51:50.072494+0000","last_peered":"2024-04-25T05:51:50.072494+0000","last_clean":"2024-04-25T05:51:50.072494+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:51:50.072494+0000","last_undegraded":"2024-04-25T05:51:50.072494+0000","last_fullsized":"2024-04-25T05:51:50.072494+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:51:23.925746+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.925746+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:24.331789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":938,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081793+0000","last_change":"2024-04-25T05:50:45.109485+0000","last_active":"2024-04-25T05:51:50.081793+0000","last_peered":"2024-04-25T05:51:50.081793+0000","last_clean":"2024-04-25T05:51:50.081793+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:51:50.081793+0000","last_undegraded":"2024-04-25T05:51:50.081793+0000","last_fullsized":"2024-04-25T05:51:50.081793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.109246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.109246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.734569+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":945,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074335+0000","last_change":"2024-04-25T05:51:40.102243+0000","last_active":"2024-04-25T05:51:50.074335+0000","last_peered":"2024-04-25T05:51:50.074335+0000","last_clean":"2024-04-25T05:51:50.074335+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:51:50.074335+0000","last_undegraded":"2024-04-25T05:51:50.074335+0000","last_fullsized":"2024-04-25T05:51:50.074335+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:55.457769+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.086161+0000","last_change":"2024-04-25T05:51:40.094829+0000","last_active":"2024-04-25T05:51:50.086161+0000","last_peered":"2024-04-25T05:51:50.086161+0000","last_clean":"2024-04-25T05:51:50.086161+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:51:50.086161+0000","last_undegraded":"2024-04-25T05:51:50.086161+0000","last_fullsized":"2024-04-25T05:51:50.086161+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:02.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:02.734157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"79'128","reported_seq":1144,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080468+0000","last_change":"2024-04-25T05:51:38.092647+0000","last_active":"2024-04-25T05:51:50.080468+0000","last_peered":"2024-04-25T05:51:50.080468+0000","last_clean":"2024-04-25T05:51:50.080468+0000","last_became_active":"2024-04-25T05:51:38.088632+0000","last_became_peered":"2024-04-25T05:51:38.088632+0000","last_unstale":"2024-04-25T05:51:50.080468+0000","last_undegraded":"2024-04-25T05:51:50.080468+0000","last_fullsized":"2024-04-25T05:51:50.080468+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":490,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:34.289261+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.289261+0000","last_clean_scrub_stamp":"2024-04-25T05:51:34.289261+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:03.253427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":1016,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073912+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:51:50.073912+0000","last_peered":"2024-04-25T05:51:50.073912+0000","last_clean":"2024-04-25T05:51:50.073912+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:51:50.073912+0000","last_undegraded":"2024-04-25T05:51:50.073912+0000","last_fullsized":"2024-04-25T05:51:50.073912+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:20.599614+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073597+0000","last_change":"2024-04-25T05:50:05.705768+0000","last_active":"2024-04-25T05:51:50.073597+0000","last_peered":"2024-04-25T05:51:50.073597+0000","last_clean":"2024-04-25T05:51:50.073597+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:51:50.073597+0000","last_undegraded":"2024-04-25T05:51:50.073597+0000","last_fullsized":"2024-04-25T05:51:50.073597+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.705647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.705647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":948,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080793+0000","last_change":"2024-04-25T05:51:21.891165+0000","last_active":"2024-04-25T05:51:50.080793+0000","last_peered":"2024-04-25T05:51:50.080793+0000","last_clean":"2024-04-25T05:51:50.080793+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:51:50.080793+0000","last_undegraded":"2024-04-25T05:51:50.080793+0000","last_fullsized":"2024-04-25T05:51:50.080793+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:21.891058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:21.891058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:33.882400+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":953,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074607+0000","last_change":"2024-04-25T05:51:40.104333+0000","last_active":"2024-04-25T05:51:50.074607+0000","last_peered":"2024-04-25T05:51:50.074607+0000","last_clean":"2024-04-25T05:51:50.074607+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:51:50.074607+0000","last_undegraded":"2024-04-25T05:51:50.074607+0000","last_fullsized":"2024-04-25T05:51:50.074607+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":955,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081497+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:51:50.081497+0000","last_peered":"2024-04-25T05:51:50.081497+0000","last_clean":"2024-04-25T05:51:50.081497+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:51:50.081497+0000","last_undegraded":"2024-04-25T05:51:50.081497+0000","last_fullsized":"2024-04-25T05:51:50.081497+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.094181+0000","last_change":"2024-04-25T05:51:40.098114+0000","last_active":"2024-04-25T05:51:50.094181+0000","last_peered":"2024-04-25T05:51:50.094181+0000","last_clean":"2024-04-25T05:51:50.094181+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:51:50.094181+0000","last_undegraded":"2024-04-25T05:51:50.094181+0000","last_fullsized":"2024-04-25T05:51:50.094181+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:07.763810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.763810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":654,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.079123+0000","last_change":"2024-04-25T05:50:14.959177+0000","last_active":"2024-04-25T05:51:50.079123+0000","last_peered":"2024-04-25T05:51:50.079123+0000","last_clean":"2024-04-25T05:51:50.079123+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:51:50.079123+0000","last_undegraded":"2024-04-25T05:51:50.079123+0000","last_fullsized":"2024-04-25T05:51:50.079123+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.959063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.959063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":939,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081396+0000","last_change":"2024-04-25T05:49:31.857568+0000","last_active":"2024-04-25T05:51:50.081396+0000","last_peered":"2024-04-25T05:51:50.081396+0000","last_clean":"2024-04-25T05:51:50.081396+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:51:50.081396+0000","last_undegraded":"2024-04-25T05:51:50.081396+0000","last_fullsized":"2024-04-25T05:51:50.081396+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:49:31.857452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:31.857452+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":774,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081109+0000","last_change":"2024-04-25T05:50:47.087717+0000","last_active":"2024-04-25T05:51:50.081109+0000","last_peered":"2024-04-25T05:51:50.081109+0000","last_clean":"2024-04-25T05:51:50.081109+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:51:50.081109+0000","last_undegraded":"2024-04-25T05:51:50.081109+0000","last_fullsized":"2024-04-25T05:51:50.081109+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:50:47.087575+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:50:47.087575+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:10.480370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":953,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081109+0000","last_change":"2024-04-25T05:51:40.085579+0000","last_active":"2024-04-25T05:51:50.081109+0000","last_peered":"2024-04-25T05:51:50.081109+0000","last_clean":"2024-04-25T05:51:50.081109+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:51:50.081109+0000","last_undegraded":"2024-04-25T05:51:50.081109+0000","last_fullsized":"2024-04-25T05:51:50.081109+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:00.472467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:48:00.472467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089439+0000","last_change":"2024-04-25T05:51:40.103589+0000","last_active":"2024-04-25T05:51:50.089439+0000","last_peered":"2024-04-25T05:51:50.089439+0000","last_clean":"2024-04-25T05:51:50.089439+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:51:50.089439+0000","last_undegraded":"2024-04-25T05:51:50.089439+0000","last_fullsized":"2024-04-25T05:51:50.089439+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:30.823661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:30.823661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":936,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072737+0000","last_change":"2024-04-25T05:50:17.140640+0000","last_active":"2024-04-25T05:51:50.072737+0000","last_peered":"2024-04-25T05:51:50.072737+0000","last_clean":"2024-04-25T05:51:50.072737+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:51:50.072737+0000","last_undegraded":"2024-04-25T05:51:50.072737+0000","last_fullsized":"2024-04-25T05:51:50.072737+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:50:17.140516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":37,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.089944+0000","last_change":"2024-04-25T05:51:40.147090+0000","last_active":"2024-04-25T05:51:50.089944+0000","last_peered":"2024-04-25T05:51:50.089944+0000","last_clean":"2024-04-25T05:51:50.089944+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:51:50.089944+0000","last_undegraded":"2024-04-25T05:51:50.089944+0000","last_fullsized":"2024-04-25T05:51:50.089944+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:05.442680+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":939,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074501+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:51:50.074501+0000","last_peered":"2024-04-25T05:51:50.074501+0000","last_clean":"2024-04-25T05:51:50.074501+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:51:50.074501+0000","last_undegraded":"2024-04-25T05:51:50.074501+0000","last_fullsized":"2024-04-25T05:51:50.074501+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:00.209136+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":814,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078962+0000","last_change":"2024-04-25T05:50:41.100007+0000","last_active":"2024-04-25T05:51:50.078962+0000","last_peered":"2024-04-25T05:51:50.078962+0000","last_clean":"2024-04-25T05:51:50.078962+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:51:50.078962+0000","last_undegraded":"2024-04-25T05:51:50.078962+0000","last_fullsized":"2024-04-25T05:51:50.078962+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:50:41.099870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:41.099870+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.548999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":967,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.082119+0000","last_change":"2024-04-25T05:50:04.837659+0000","last_active":"2024-04-25T05:51:50.082119+0000","last_peered":"2024-04-25T05:51:50.082119+0000","last_clean":"2024-04-25T05:51:50.082119+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:51:50.082119+0000","last_undegraded":"2024-04-25T05:51:50.082119+0000","last_fullsized":"2024-04-25T05:51:50.082119+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:50:04.837536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:04.837536+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":963,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074953+0000","last_change":"2024-04-25T05:51:18.436527+0000","last_active":"2024-04-25T05:51:50.074953+0000","last_peered":"2024-04-25T05:51:50.074953+0000","last_clean":"2024-04-25T05:51:50.074953+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:51:50.074953+0000","last_undegraded":"2024-04-25T05:51:50.074953+0000","last_fullsized":"2024-04-25T05:51:50.074953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:18.436370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.436370+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:34.484305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":973,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088360+0000","last_change":"2024-04-25T05:51:17.763060+0000","last_active":"2024-04-25T05:51:50.088360+0000","last_peered":"2024-04-25T05:51:50.088360+0000","last_clean":"2024-04-25T05:51:50.088360+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:51:50.088360+0000","last_undegraded":"2024-04-25T05:51:50.088360+0000","last_fullsized":"2024-04-25T05:51:50.088360+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:17.762926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:17.762926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:36.715438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.078333+0000","last_change":"2024-04-25T05:51:40.105423+0000","last_active":"2024-04-25T05:51:50.078333+0000","last_peered":"2024-04-25T05:51:50.078333+0000","last_clean":"2024-04-25T05:51:50.078333+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:51:50.078333+0000","last_undegraded":"2024-04-25T05:51:50.078333+0000","last_fullsized":"2024-04-25T05:51:50.078333+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.708291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.708291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":936,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075790+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:51:50.075790+0000","last_peered":"2024-04-25T05:51:50.075790+0000","last_clean":"2024-04-25T05:51:50.075790+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:51:50.075790+0000","last_undegraded":"2024-04-25T05:51:50.075790+0000","last_fullsized":"2024-04-25T05:51:50.075790+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":306,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.080936+0000","last_change":"2024-04-25T05:50:44.583319+0000","last_active":"2024-04-25T05:51:50.080936+0000","last_peered":"2024-04-25T05:51:50.080936+0000","last_clean":"2024-04-25T05:51:50.080936+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:51:50.080936+0000","last_undegraded":"2024-04-25T05:51:50.080936+0000","last_fullsized":"2024-04-25T05:51:50.080936+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:50:44.583030+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:50:44.583030+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090000+0000","last_change":"2024-04-25T05:51:40.101375+0000","last_active":"2024-04-25T05:51:50.090000+0000","last_peered":"2024-04-25T05:51:50.090000+0000","last_clean":"2024-04-25T05:51:50.090000+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:51:50.090000+0000","last_undegraded":"2024-04-25T05:51:50.090000+0000","last_fullsized":"2024-04-25T05:51:50.090000+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.835919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.835919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:33.707196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":960,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072964+0000","last_change":"2024-04-25T05:51:40.109523+0000","last_active":"2024-04-25T05:51:50.072964+0000","last_peered":"2024-04-25T05:51:50.072964+0000","last_clean":"2024-04-25T05:51:50.072964+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:51:50.072964+0000","last_undegraded":"2024-04-25T05:51:50.072964+0000","last_fullsized":"2024-04-25T05:51:50.072964+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.989325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.989325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:32.817542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":968,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.091708+0000","last_change":"2024-04-25T05:51:26.851300+0000","last_active":"2024-04-25T05:51:50.091708+0000","last_peered":"2024-04-25T05:51:50.091708+0000","last_clean":"2024-04-25T05:51:50.091708+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:51:50.091708+0000","last_undegraded":"2024-04-25T05:51:50.091708+0000","last_fullsized":"2024-04-25T05:51:50.091708+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:26.851169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:51:26.851169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:32.404070+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":439,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.090984+0000","last_change":"2024-04-25T05:49:09.554222+0000","last_active":"2024-04-25T05:51:50.090984+0000","last_peered":"2024-04-25T05:51:50.090984+0000","last_clean":"2024-04-25T05:51:50.090984+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:51:50.090984+0000","last_undegraded":"2024-04-25T05:51:50.090984+0000","last_fullsized":"2024-04-25T05:51:50.090984+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:48:36.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:36.108290+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073705+0000","last_change":"2024-04-25T05:50:50.860753+0000","last_active":"2024-04-25T05:51:50.073705+0000","last_peered":"2024-04-25T05:51:50.073705+0000","last_clean":"2024-04-25T05:51:50.073705+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:51:50.073705+0000","last_undegraded":"2024-04-25T05:51:50.073705+0000","last_fullsized":"2024-04-25T05:51:50.073705+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:50.860626+0000","last_clean_scrub_stamp":"2024-04-25T05:50:50.860626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:18.625619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":928,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.081202+0000","last_change":"2024-04-25T05:49:35.840397+0000","last_active":"2024-04-25T05:51:50.081202+0000","last_peered":"2024-04-25T05:51:50.081202+0000","last_clean":"2024-04-25T05:51:50.081202+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:51:50.081202+0000","last_undegraded":"2024-04-25T05:51:50.081202+0000","last_fullsized":"2024-04-25T05:51:50.081202+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.840316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.840316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072405+0000","last_change":"2024-04-25T05:51:31.994370+0000","last_active":"2024-04-25T05:51:50.072405+0000","last_peered":"2024-04-25T05:51:50.072405+0000","last_clean":"2024-04-25T05:51:50.072405+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:51:50.072405+0000","last_undegraded":"2024-04-25T05:51:50.072405+0000","last_fullsized":"2024-04-25T05:51:50.072405+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:51:31.994196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:55.920950+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":753,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.088259+0000","last_change":"2024-04-25T05:51:40.142768+0000","last_active":"2024-04-25T05:51:50.088259+0000","last_peered":"2024-04-25T05:51:50.088259+0000","last_clean":"2024-04-25T05:51:50.088259+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:51:50.088259+0000","last_undegraded":"2024-04-25T05:51:50.088259+0000","last_fullsized":"2024-04-25T05:51:50.088259+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:49:06.169828+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":265,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.073901+0000","last_change":"2024-04-25T05:50:45.906872+0000","last_active":"2024-04-25T05:51:50.073901+0000","last_peered":"2024-04-25T05:51:50.073901+0000","last_clean":"2024-04-25T05:51:50.073901+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:51:50.073901+0000","last_undegraded":"2024-04-25T05:51:50.073901+0000","last_fullsized":"2024-04-25T05:51:50.073901+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.906734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.906734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:12.651537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":271,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.072679+0000","last_change":"2024-04-25T05:51:40.084285+0000","last_active":"2024-04-25T05:51:50.072679+0000","last_peered":"2024-04-25T05:51:50.072679+0000","last_clean":"2024-04-25T05:51:50.072679+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:51:50.072679+0000","last_undegraded":"2024-04-25T05:51:50.072679+0000","last_fullsized":"2024-04-25T05:51:50.072679+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.941496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.941496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:57.898003+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.086046+0000","last_change":"2024-04-25T05:51:40.110384+0000","last_active":"2024-04-25T05:51:50.086046+0000","last_peered":"2024-04-25T05:51:50.086046+0000","last_clean":"2024-04-25T05:51:50.086046+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:51:50.086046+0000","last_undegraded":"2024-04-25T05:51:50.086046+0000","last_fullsized":"2024-04-25T05:51:50.086046+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1250,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074026+0000","last_change":"2024-04-25T05:51:40.151139+0000","last_active":"2024-04-25T05:51:50.074026+0000","last_peered":"2024-04-25T05:51:50.074026+0000","last_clean":"2024-04-25T05:51:50.074026+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:51:50.074026+0000","last_undegraded":"2024-04-25T05:51:50.074026+0000","last_fullsized":"2024-04-25T05:51:50.074026+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:53.979247+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":956,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.075274+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:51:50.075274+0000","last_peered":"2024-04-25T05:51:50.075274+0000","last_clean":"2024-04-25T05:51:50.075274+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:51:50.075274+0000","last_undegraded":"2024-04-25T05:51:50.075274+0000","last_fullsized":"2024-04-25T05:51:50.075274+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":494,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.087909+0000","last_change":"2024-04-25T05:50:40.775527+0000","last_active":"2024-04-25T05:51:50.087909+0000","last_peered":"2024-04-25T05:51:50.087909+0000","last_clean":"2024-04-25T05:51:50.087909+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:51:50.087909+0000","last_undegraded":"2024-04-25T05:51:50.087909+0000","last_fullsized":"2024-04-25T05:51:50.087909+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:40.775406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.775406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:57.019250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":923,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074424+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:51:50.074424+0000","last_peered":"2024-04-25T05:51:50.074424+0000","last_clean":"2024-04-25T05:51:50.074424+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:51:50.074424+0000","last_undegraded":"2024-04-25T05:51:50.074424+0000","last_fullsized":"2024-04-25T05:51:50.074424+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":1029,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.087637+0000","last_change":"2024-04-25T05:50:43.844399+0000","last_active":"2024-04-25T05:51:50.087637+0000","last_peered":"2024-04-25T05:51:50.087637+0000","last_clean":"2024-04-25T05:51:50.087637+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:51:50.087637+0000","last_undegraded":"2024-04-25T05:51:50.087637+0000","last_fullsized":"2024-04-25T05:51:50.087637+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:43.844256+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:50:43.844256+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:51:59.303125+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":23,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.094287+0000","last_change":"2024-04-25T05:51:40.091558+0000","last_active":"2024-04-25T05:51:50.094287+0000","last_peered":"2024-04-25T05:51:50.094287+0000","last_clean":"2024-04-25T05:51:50.094287+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:51:50.094287+0000","last_undegraded":"2024-04-25T05:51:50.094287+0000","last_fullsized":"2024-04-25T05:51:50.094287+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.890392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.890392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":948,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:50.074604+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:51:50.074604+0000","last_peered":"2024-04-25T05:51:50.074604+0000","last_clean":"2024-04-25T05:51:50.074604+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:51:50.074604+0000","last_undegraded":"2024-04-25T05:51:50.074604+0000","last_fullsized":"2024-04-25T05:51:50.074604+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:52:16.929190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":478,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-04-25T05:51:51.705582+0000","last_change":"2024-04-25T05:51:51.705620+0000","last_active":"2024-04-25T05:51:51.705582+0000","last_peered":"2024-04-25T05:51:51.705582+0000","last_clean":"2024-04-25T05:51:51.705582+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:51:51.705582+0000","last_undegraded":"2024-04-25T05:51:51.705582+0000","last_fullsized":"2024-04-25T05:51:51.705582+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:51.705348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:51.705348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:19.365818+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":8303738904,"num_objects":126706,"num_object_clones":0,"num_object_copies":380118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35066,"num_objects_unfound":0,"num_objects_dirty":126706,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133868,"num_write_kb":8338305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4489,"num_bytes_recovered":294191104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25222656000,"data_stored":25222643784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73331,"ondisk_log_size":73331,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1055,"num_read_kb":1050,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1399190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070684,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4466612,"kb_used_data":3574700,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":89278540,"statfs":{"total":95995035648,"available":91421224960,"internally_reserved":0,"allocated":3660492800,"data_stored":3659799021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":913308106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365024,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4887516,"kb_used_data":4076116,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":88857636,"statfs":{"total":95995035648,"available":90990219264,"internally_reserved":0,"allocated":4173942784,"data_stored":4173272397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":830863814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5603584,"kb_used_data":4561656,"kb_used_omap":1,"kb_used_meta":1041854,"kb_avail":88141568,"statfs":{"total":95995035648,"available":90256965632,"internally_reserved":0,"allocated":4671135744,"data_stored":4670428385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1066858949},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849128,"kb_used_data":290400,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92896024,"statfs":{"total":95995035648,"available":95125528576,"internally_reserved":0,"allocated":297369600,"data_stored":296686266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411572,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4273440,"kb_used_data":3588376,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":89471712,"statfs":{"total":95995035648,"available":91619033088,"internally_reserved":0,"allocated":3674497024,"data_stored":3673822037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149934,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4717176,"kb_used_data":3562736,"kb_used_omap":1,"kb_used_meta":1154302,"kb_avail":89027976,"statfs":{"total":95995035648,"available":91164647424,"internally_reserved":0,"allocated":3648241664,"data_stored":3647542587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1182005703},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986409,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3799096,"kb_used_data":3041840,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":89946056,"statfs":{"total":95995035648,"available":92104761344,"internally_reserved":0,"allocated":3114844160,"data_stored":3114145283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692069,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2627296,"kb_used_data":2040792,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91117856,"statfs":{"total":95995035648,"available":93304684544,"internally_reserved":0,"allocated":2089771008,"data_stored":2089096521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295567360,"data_stored":295567360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4641390592,"data_stored":4641390592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3664449536,"data_stored":3664445464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3619749888,"data_stored":3619749888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3098611712,"data_stored":3098607640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2084110336,"data_stored":2084110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4165144576,"data_stored":4165140504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3653632000,"data_stored":3653632000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:52:02.015 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-04-25T05:52:02.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-25T05:52:02.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:02.082 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:02.134 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:02.215 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:02.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:02.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:02.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:02.474 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:02.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[27010]: 2.2 deep-scrub starts 2024-04-25T05:52:02.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[27010]: 2.2 deep-scrub ok 2024-04-25T05:52:02.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[27010]: 11.4 scrub starts 2024-04-25T05:52:02.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[37492]: 2.2 deep-scrub starts 2024-04-25T05:52:02.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[37492]: 2.2 deep-scrub ok 2024-04-25T05:52:02.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:02 smithi043 ceph-mon[37492]: 11.4 scrub starts 2024-04-25T05:52:02.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:02.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:02 smithi060 ceph-mon[30898]: 2.2 deep-scrub starts 2024-04-25T05:52:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:02 smithi060 ceph-mon[30898]: 2.2 deep-scrub ok 2024-04-25T05:52:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:02 smithi060 ceph-mon[30898]: 11.4 scrub starts 2024-04-25T05:52:02.753 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:02.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:02.885 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:02.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:02.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:52:02.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:03.019 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:03.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:03.166 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:03.223 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:03.266 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:03.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:03.313 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:03.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:03.359 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:03.407 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:03.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:03.459 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:03.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[27010]: 11.7 scrub ok 2024-04-25T05:52:03.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[27010]: pgmap v956: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 889 op/s; 32007/366165 objects misplaced (8.741%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:03.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[27010]: 4.1a scrub starts 2024-04-25T05:52:03.459 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[27010]: 4.1a scrub ok 2024-04-25T05:52:03.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2380526513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:03.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[37492]: 11.7 scrub ok 2024-04-25T05:52:03.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[37492]: pgmap v956: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 889 op/s; 32007/366165 objects misplaced (8.741%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:03.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[37492]: 4.1a scrub starts 2024-04-25T05:52:03.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[37492]: 4.1a scrub ok 2024-04-25T05:52:03.461 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2380526513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.470 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:03.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:03.517 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:03.563 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:03.569 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:03.569 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:03.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:03.593 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:03.603 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:52:03.604 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:52:03.604 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:52:03.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:52:03.604 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:52:03.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:03.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-mon[30898]: 11.7 scrub ok 2024-04-25T05:52:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-mon[30898]: pgmap v956: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.4 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 889 op/s; 32007/366165 objects misplaced (8.741%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:03.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-mon[30898]: 4.1a scrub starts 2024-04-25T05:52:03.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-mon[30898]: 4.1a scrub ok 2024-04-25T05:52:03.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2380526513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:03.778 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:03.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:03.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:03.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:04.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:04.044 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:04.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:04.182 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:04.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:04.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:04.315 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:04.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:04.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:04.579 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[37492]: 11.4 scrub ok 2024-04-25T05:52:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:04.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[27010]: 11.4 scrub ok 2024-04-25T05:52:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:04.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:04.680 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:04.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:04 smithi060 ceph-mon[30898]: 11.4 scrub ok 2024-04-25T05:52:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:04.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:04.853 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:04.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:04.988 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:04.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:05.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.122 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:05.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:05.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.259 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:05.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:05.352 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:05.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:05.411 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:05.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:05.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:05.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:05.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:05.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:05.815 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[27010]: pgmap v957: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 30437/358239 objects misplaced (8.496%); 23 MiB/s, 363 objects/s recovering 2024-04-25T05:52:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:52:05.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:52:05.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:05.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[37492]: pgmap v957: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 30437/358239 objects misplaced (8.496%); 23 MiB/s, 363 objects/s recovering 2024-04-25T05:52:05.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:52:05.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:05 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:52:05.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:05.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:05.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:05 smithi060 ceph-mon[30898]: pgmap v957: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 30437/358239 objects misplaced (8.496%); 23 MiB/s, 363 objects/s recovering 2024-04-25T05:52:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:05 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:52:05.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:05 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:52:06.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.100 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:06.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:06.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:52:06.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:06.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:06.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:06.443 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:06.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:52:06.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:06.607 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:06.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:52:06.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:52:06.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:52:06.633 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:52:06.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:52:06.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:52:06.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:52:06.634 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:06 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:52:06.710 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:06.731 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:06.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:06.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:06.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:06.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:06 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:52:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:06 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:52:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:06 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:52:06.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:06 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:52:07.004 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:07.106 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:07.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:07.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:07.273 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:07.375 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:07.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:07.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:07.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: pgmap v958: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 992 op/s; 29122/351885 objects misplaced (8.276%); 22 MiB/s, 351 objects/s recovering 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: 11.f scrub starts 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2642739832' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: pgmap v958: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 992 op/s; 29122/351885 objects misplaced (8.276%); 22 MiB/s, 351 objects/s recovering 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: 11.f scrub starts 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2642739832' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:07.575 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:07 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:07.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:07.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:07.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:07.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:07.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:07.983 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:07.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: pgmap v958: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 992 op/s; 29122/351885 objects misplaced (8.276%); 22 MiB/s, 351 objects/s recovering 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: 11.f scrub starts 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2642739832' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:07.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:07 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-25T05:52:08.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.124 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:08.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:08.225 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:08.256 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:08.358 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:08.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:08.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.527 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:08.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:08.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:08.660 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:08.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:08.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:52:08.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: osdmap e507: 8 total, 8 up, 8 in 2024-04-25T05:52:08.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: pgmap v960: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 616 op/s; 28446/350019 objects misplaced (8.127%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:52:08.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:08.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: 6.1c deep-scrub starts 2024-04-25T05:52:08.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[37492]: 6.1c deep-scrub ok 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: osdmap e507: 8 total, 8 up, 8 in 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: pgmap v960: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 616 op/s; 28446/350019 objects misplaced (8.127%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: 6.1c deep-scrub starts 2024-04-25T05:52:08.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:08 smithi043 ceph-mon[27010]: 6.1c deep-scrub ok 2024-04-25T05:52:08.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:08.939 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:08.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: osdmap e507: 8 total, 8 up, 8 in 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: pgmap v960: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 163 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 616 op/s; 28446/350019 objects misplaced (8.127%); 16 MiB/s, 250 objects/s recovering 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: 6.1c deep-scrub starts 2024-04-25T05:52:08.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:08 smithi060 ceph-mon[30898]: 6.1c deep-scrub ok 2024-04-25T05:52:09.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:09.163 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:09.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:09.341 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:09.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:09.476 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:09.567 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:52:09.578 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:09.623 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:09.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:09.855 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pg_num to 11 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: osdmap e508: 8 total, 8 up, 8 in 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:09.880 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: osdmap e508: 8 total, 8 up, 8 in 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:09.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: osdmap e508: 8 total, 8 up, 8 in 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:52:09.888 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:09.889 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3648112652' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:09.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:09.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:10.023 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:10.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:10.154 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:10.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:10.284 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:10.386 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:10.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:10.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.561 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:10.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:10.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:10.697 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:10.799 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:10.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[37492]: 11.f scrub ok 2024-04-25T05:52:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[37492]: osdmap e509: 8 total, 8 up, 8 in 2024-04-25T05:52:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[37492]: pgmap v963: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 26272/336492 objects misplaced (7.808%); 22 MiB/s, 346 objects/s recovering 2024-04-25T05:52:10.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[37492]: osdmap e510: 8 total, 8 up, 8 in 2024-04-25T05:52:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[27010]: 11.f scrub ok 2024-04-25T05:52:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[27010]: osdmap e509: 8 total, 8 up, 8 in 2024-04-25T05:52:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[27010]: pgmap v963: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 26272/336492 objects misplaced (7.808%); 22 MiB/s, 346 objects/s recovering 2024-04-25T05:52:10.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:10 smithi043 ceph-mon[27010]: osdmap e510: 8 total, 8 up, 8 in 2024-04-25T05:52:10.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:10.973 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:10.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:10 smithi060 ceph-mon[30898]: 11.f scrub ok 2024-04-25T05:52:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:10 smithi060 ceph-mon[30898]: osdmap e509: 8 total, 8 up, 8 in 2024-04-25T05:52:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:10 smithi060 ceph-mon[30898]: pgmap v963: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 26272/336492 objects misplaced (7.808%); 22 MiB/s, 346 objects/s recovering 2024-04-25T05:52:10.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:10 smithi060 ceph-mon[30898]: osdmap e510: 8 total, 8 up, 8 in 2024-04-25T05:52:11.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:11.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:11.251 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:11.353 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.386 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:11.487 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:11.527 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:11.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.662 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:11.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:11.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:11.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:11.825 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:11.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:11.956 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:11.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:12.058 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:12.098 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:12.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.240 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:12.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:12.341 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:12.355 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:12 smithi043 ceph-mon[27010]: osdmap e511: 8 total, 8 up, 8 in 2024-04-25T05:52:12.355 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:12 smithi043 ceph-mon[37492]: osdmap e511: 8 total, 8 up, 8 in 2024-04-25T05:52:12.386 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:12.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:12.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:12 smithi060 ceph-mon[30898]: osdmap e511: 8 total, 8 up, 8 in 2024-04-25T05:52:12.525 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:12.541 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:12.621 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:12.628 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:12.656 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:12.659 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:12.723 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:12.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:12.764 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:12.790 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:12.808 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:12.848 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:12.890 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:12.892 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:12.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:12.933 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:12.945 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:12.946 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:13.021 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[27010]: pgmap v966: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 26272/336492 objects misplaced (7.808%); 17 MiB/s, 273 objects/s recovering 2024-04-25T05:52:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[27010]: osdmap e512: 8 total, 8 up, 8 in 2024-04-25T05:52:13.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1892113597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:13.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[37492]: pgmap v966: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 26272/336492 objects misplaced (7.808%); 17 MiB/s, 273 objects/s recovering 2024-04-25T05:52:13.141 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[37492]: osdmap e512: 8 total, 8 up, 8 in 2024-04-25T05:52:13.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1892113597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:13.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:13.191 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:13.288 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:52:13.289 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:52:13.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:13.329 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:13.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:13 smithi060 ceph-mon[30898]: pgmap v966: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 2 unknown, 156 active+clean, 3 active+remapped+backfilling, 1 active+clean+scrubbing, 4 peering; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 26272/336492 objects misplaced (7.808%); 17 MiB/s, 273 objects/s recovering 2024-04-25T05:52:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:13 smithi060 ceph-mon[30898]: osdmap e512: 8 total, 8 up, 8 in 2024-04-25T05:52:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1892113597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:13.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:13.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:13.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:13.735 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:13.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:52:13.837 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:13.871 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:13.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:13.972 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:14.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.006 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:14.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:14.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:14.134 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:14.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:14.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:14.284 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:14.385 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:14.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:14.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:14.568 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:14.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:14.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:14.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:52:14.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:14.834 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:14.860 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:14.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:14.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:14.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:15.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:15.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.291 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:15.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:15.345 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:15 smithi043 ceph-mon[27010]: pgmap v968: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 unknown, 159 active+clean, 3 active+remapped+backfilling, 3 peering; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 652 op/s; 24789/327729 objects misplaced (7.564%); 16 MiB/s, 253 objects/s recovering 2024-04-25T05:52:15.345 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:15.346 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:15 smithi043 ceph-mon[37492]: pgmap v968: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 unknown, 159 active+clean, 3 active+remapped+backfilling, 3 peering; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 652 op/s; 24789/327729 objects misplaced (7.564%); 16 MiB/s, 253 objects/s recovering 2024-04-25T05:52:15.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:15.437 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:15 smithi060 ceph-mon[30898]: pgmap v968: 168 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 unknown, 159 active+clean, 3 active+remapped+backfilling, 3 peering; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 652 op/s; 24789/327729 objects misplaced (7.564%); 16 MiB/s, 253 objects/s recovering 2024-04-25T05:52:15.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:15.562 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:15.564 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:15.666 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:15.714 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:15.815 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:15.860 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:15.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:15.999 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:15.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:16.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:52:16.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:16.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:16.133 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:52:16.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:16.298 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:52:16.298 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:52:16.298 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:52:16.299 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:52:16.299 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T05:52:16.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-25T05:52:16.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/490068672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.317 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-25T05:52:16.318 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/490068672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.318 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.318 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:16 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-25T05:52:16.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:16.419 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:16 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/490068672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:16 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:16.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:16 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-25T05:52:16.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:16.592 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:16.694 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.732 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:16.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:16.833 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:16.872 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:16.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:16.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:17.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:17.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:17.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[37492]: pgmap v969: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 882 op/s; 23458/320622 objects misplaced (7.316%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:52:17.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:17.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:17.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-25T05:52:17.136 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[37492]: osdmap e513: 8 total, 8 up, 8 in 2024-04-25T05:52:17.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:17.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:17.282 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:17.384 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[27010]: pgmap v969: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 882 op/s; 23458/320622 objects misplaced (7.316%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:52:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-25T05:52:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:17 smithi043 ceph-mon[27010]: osdmap e513: 8 total, 8 up, 8 in 2024-04-25T05:52:17.420 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:17 smithi060 ceph-mon[30898]: pgmap v969: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 882 op/s; 23458/320622 objects misplaced (7.316%); 16 MiB/s, 261 objects/s recovering 2024-04-25T05:52:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:17 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:17 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:17 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-25T05:52:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:17 smithi060 ceph-mon[30898]: osdmap e513: 8 total, 8 up, 8 in 2024-04-25T05:52:17.521 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:17.662 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:17.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:17.839 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:17.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:17.972 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:17.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:18.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:18.103 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:18.205 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:18.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:18.391 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:18.437 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:18 smithi043 ceph-mon[27010]: osdmap e514: 8 total, 8 up, 8 in 2024-04-25T05:52:18.437 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:18 smithi043 ceph-mon[37492]: osdmap e514: 8 total, 8 up, 8 in 2024-04-25T05:52:18.486 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:52:18.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:18 smithi060 ceph-mon[30898]: osdmap e514: 8 total, 8 up, 8 in 2024-04-25T05:52:18.577 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:18.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.728 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:18.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:18.829 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:18.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:18.971 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:19.002 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:19.093 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:19.103 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:19.111 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:19.138 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.151 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:19.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:19.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[27010]: pgmap v972: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 987 op/s; 22846/318726 objects misplaced (7.168%); 21 MiB/s, 335 objects/s recovering 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[27010]: osdmap e515: 8 total, 8 up, 8 in 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[37492]: pgmap v972: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 987 op/s; 22846/318726 objects misplaced (7.168%); 21 MiB/s, 335 objects/s recovering 2024-04-25T05:52:19.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[37492]: osdmap e515: 8 total, 8 up, 8 in 2024-04-25T05:52:19.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:19.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.303 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:19.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:19.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:19.460 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:19 smithi060 ceph-mon[30898]: pgmap v972: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 163 active+clean, 3 active+remapped+backfilling; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 987 op/s; 22846/318726 objects misplaced (7.168%); 21 MiB/s, 335 objects/s recovering 2024-04-25T05:52:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:19 smithi060 ceph-mon[30898]: osdmap e515: 8 total, 8 up, 8 in 2024-04-25T05:52:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:19.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.593 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:19.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:19.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:19.721 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:19.782 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:19.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:19.853 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:19.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:19.995 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:19.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:20.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:20.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:20.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:20.242 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.277 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:20.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:20.292 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:20.337 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:20.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:20.382 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:20.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:20 smithi043 ceph-mon[27010]: osdmap e516: 8 total, 8 up, 8 in 2024-04-25T05:52:20.383 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:20 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-25T05:52:20.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:20 smithi043 ceph-mon[37492]: osdmap e516: 8 total, 8 up, 8 in 2024-04-25T05:52:20.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:20 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-25T05:52:20.408 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:20.428 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:20.477 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:20.484 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:52:20.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:52:20.484 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:52:20.484 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:52:20.484 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:52:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:20 smithi060 ceph-mon[30898]: osdmap e516: 8 total, 8 up, 8 in 2024-04-25T05:52:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:20 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-25T05:52:20.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:20.530 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:20.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:20.581 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:20.589 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:20.590 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:20.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:20.679 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:20.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:20.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:20.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:20.955 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:20.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:20.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:52:21.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:21.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:21.221 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:52:21.222 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:52:21.222 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 518, 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "84884b5e-02c4-11ef-bc93-c7b262605968", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-25T05:28:34.192515+0000", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-25T05:52:21.166935+0000", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "last_up_change": "2024-04-25T05:33:46.700068+0000", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "last_in_change": "2024-04-25T05:51:39.176833+0000", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_num": 5806080, 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_set": [ 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "nodeep-scrub", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "noscrub", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pglog_hardlimit", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "purged_snapdirs", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "recovery_deletes", 2024-04-25T05:52:21.223 INFO:teuthology.orchestra.run.smithi043.stdout: "sortbitwise" 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_version": 18, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_snapshot": "", 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_max": 11, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "max_osd": 8, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "require_min_compat_client": "luminous", 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "min_compat_client": "luminous", 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "require_osd_release": "squid", 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "allow_crimson": false, 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: "pools": [ 2024-04-25T05:52:21.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 1, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".mgr", 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:31:44.863613+0000", 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.225 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 20, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 11, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 11, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 11, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 19, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "1.14", 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 515, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 514, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 514, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "79'128", 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "79'128" 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "517", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "517", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "517", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.227 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.228 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_max": 32, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_min": 1 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": {} 2024-04-25T05:52:21.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.230 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 2, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "datapool", 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:56.045664+0000", 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 8193, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.231 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 3, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 3, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 3, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 3, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 3, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "69", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 2, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 69, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.233 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.234 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:52:21.235 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "rbd": {} 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.6500000953674316, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.6500000953674316, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 3, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": ".rgw.root", 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:57.225173+0000", 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.237 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.238 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "68", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.239 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.240 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.5, 2024-04-25T05:52:21.241 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.5, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.5, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.5, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 4, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.log", 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:33:58.748103+0000", 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:52:21.243 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "70", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.244 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.245 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.246 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2.25, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2.25, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2.25, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2.25, 2024-04-25T05:52:21.247 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 5, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.control", 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:00.700767+0000", 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.248 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.249 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "72", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.250 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.252 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.25, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.25, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.25, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.25, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 6, 2024-04-25T05:52:21.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "default.rgw.meta", 2024-04-25T05:52:21.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:21.254 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:34:02.767681+0000", 2024-04-25T05:52:21.254 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.254 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 32, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 32, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 32, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 32, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 32, 2024-04-25T05:52:21.255 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "74", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.256 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.257 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.258 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "options": { 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_bias": 4 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": {} 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 1.75, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 1.75, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 1.75, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 1.75, 2024-04-25T05:52:21.259 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 11, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_name": "unique_pool_4", 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "create_time": "2024-04-25T05:50:10.050814+0000", 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 1, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "flags_names": "hashpspool", 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": 1, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "min_size": 2, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_rule": 0, 2024-04-25T05:52:21.260 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_count": 0, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_target": 0, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "object_hash": 2, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_autoscale_mode": "off", 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num": 16, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num": 16, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_placement_num_target": 16, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_target": 16, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_num_pending": 16, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "last_pg_merge_meta": { 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "source_pgid": "0.0", 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "ready_epoch": 0, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_started": 0, 2024-04-25T05:52:21.261 INFO:teuthology.orchestra.run.smithi043.stdout: "last_epoch_clean": 0, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "source_version": "0'0", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "target_version": "0'0" 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "last_change": "461", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend": "0", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "auid": 0, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_mode": "selfmanaged", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_seq": 0, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "snap_epoch": 0, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pool_snaps": [], 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps": "[]", 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_bytes": 0, 2024-04-25T05:52:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quota_max_objects": 0, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tiers": [], 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_of": -1, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "read_tier": -1, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "write_tier": -1, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_mode": "none", 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_bytes": 0, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "target_max_objects": 0, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_flush_age": 0, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "cache_min_evict_age": 0, 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profile": "", 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_params": { 2024-04-25T05:52:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "none" 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_period": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_count": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "use_gmt_hitset": true, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min_read_recency_for_promote": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "min_write_recency_for_promote": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "hit_set_search_last_n": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "grade_table": [], 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "stripe_width": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "expected_num_objects": 0, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "fast_read": false, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "options": {}, 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "application_metadata": { 2024-04-25T05:52:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "rados": {} 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "read_balance": { 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "score_type": "Fair distribution", 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "score_acting": 2, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "score_stable": 2, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "optimal_score": 1, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_acting": 2, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "raw_score_stable": 2, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity_weighted": 0.99945449829101562, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity": 1, 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "average_primary_affinity_weighted": 1 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ef370ce4-85d8-41ed-9655-2e2eaab3b48e", 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 8, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 500, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6801", 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6802", 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6804", 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6803", 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3557191248 2024-04-25T05:52:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6801/3557191248", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6802/3557191248", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6804/3557191248", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6803/3557191248", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "ed504d08-dc59-4584-bd5f-8059e02a4161", 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.269 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 13, 2024-04-25T05:52:21.270 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 517, 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6805", 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6806", 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6808", 2024-04-25T05:52:21.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6807", 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 235229295 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6805/235229295", 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6806/235229295", 2024-04-25T05:52:21.273 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6808/235229295", 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6807/235229295", 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "11ff9a46-8686-4b36-a530-5cc1dca1f088", 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.274 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 18, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 510, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6809", 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6810", 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6812", 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6811", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3416612512 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6809/3416612512", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6810/3416612512", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6812/3416612512", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6811/3416612512", 2024-04-25T05:52:21.277 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "f3314c45-11c4-46ee-be0b-6f639faa7da2", 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 26, 2024-04-25T05:52:21.278 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 515, 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6813", 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6814", 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6816", 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6815", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3391766005 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6813/3391766005", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.43:6814/3391766005", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.43:6816/3391766005", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.43:6815/3391766005", 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.281 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "8c5337b2-94d1-4784-9321-db771de75f4c", 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 0.995635986328125, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 31, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 511, 2024-04-25T05:52:21.282 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6800", 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6801", 2024-04-25T05:52:21.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6803", 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6802", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1310286745 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6800/1310286745", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6801/1310286745", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6803/1310286745", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6802/1310286745", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.285 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2", 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 40, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 511, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6804", 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6805", 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6807", 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.288 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[27010]: pgmap v975: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 20852/305256 objects misplaced (6.831%); 24 MiB/s, 381 objects/s recovering 2024-04-25T05:52:21.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-25T05:52:21.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[27010]: osdmap e517: 8 total, 8 up, 8 in 2024-04-25T05:52:21.288 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/290169809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[27010]: osdmap e518: 8 total, 8 up, 8 in 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[37492]: pgmap v975: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 20852/305256 objects misplaced (6.831%); 24 MiB/s, 381 objects/s recovering 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[37492]: osdmap e517: 8 total, 8 up, 8 in 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/290169809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:21.289 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:21 smithi043 ceph-mon[37492]: osdmap e518: 8 total, 8 up, 8 in 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6806", 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3586186657 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.290 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6804/3586186657", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6805/3586186657", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6807/3586186657", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6806/3586186657", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "5b946188-0dd1-4e68-bcfc-a3daa342edaa", 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.291 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 50, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 511, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6808", 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:52:21.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6809", 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6811", 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6810", 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 1675137598 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.294 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6808/1675137598", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6809/1675137598", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6811/1675137598", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6810/1675137598", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "uuid": "dec84bb8-9d2b-469f-9641-62ff40b9b9b2", 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "up": 1, 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "in": 1, 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 1, 2024-04-25T05:52:21.295 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_affinity": 1, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_begin": 0, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "last_clean_end": 0, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "up_from": 59, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "up_thru": 511, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "down_at": 0, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "lost_at": 0, 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6812", 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addrs": { 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6813", 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addrs": { 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6815", 2024-04-25T05:52:21.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addrs": { 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.60:6814", 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 3982012700 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.60:6812/3982012700", 2024-04-25T05:52:21.298 INFO:teuthology.orchestra.run.smithi043.stdout: "cluster_addr": "172.21.15.60:6813/3982012700", 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_back_addr": "172.21.15.60:6815/3982012700", 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "heartbeat_front_addr": "172.21.15.60:6814/3982012700", 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "state": [ 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "exists", 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "up" 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "osd_xinfo": [ 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 0, 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.299 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:43.706624+0000", 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 1, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:21.710639+0000", 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 2, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:58.548355+0000", 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 3, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.301 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:13.273647+0000", 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 4, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:30.212991+0000", 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 5, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:10.067291+0000", 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 6, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.303 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:28.450013+0000", 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": 7, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "down_stamp": "0.000000", 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_probability": 0, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "laggy_interval": 0, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "features": 4540701547738038271, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "old_weight": 0, 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "last_purged_snaps_scrub": "2024-04-25T05:51:37.706536+0000", 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: "dead_epoch": 0 2024-04-25T05:52:21.304 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap": [ 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.6", 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1c", 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:52:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: 5 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.7", 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_items": [ 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.d", 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 5 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.13", 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1b", 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 7 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "4.1f", 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:52:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 2 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.0", 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 1, 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 6 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "5.1a", 2024-04-25T05:52:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 4, 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 3 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "6.3", 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "mappings": [ 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "from": 7, 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "to": 1 2024-04-25T05:52:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_upmap_primaries": [], 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pg_temp": [ 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.1", 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: 2 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.2", 2024-04-25T05:52:21.311 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.6", 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.9", 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 1 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pgid": "11.b", 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: "osds": [ 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: 4 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:21.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "primary_temp": [], 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "blocklist": { 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3423182021": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2082036420": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3958950586": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3953560108": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4077364595": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/830746728": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/786479021": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/3735093998": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1835545567": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:6800/149142905": "2024-04-26T05:34:31.636703+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1408327650": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:52:21.314 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/2239747746": "2024-04-26T05:29:02.328794+0000", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/4084484879": "2024-04-26T05:29:30.781325+0000", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "172.21.15.43:0/1247294293": "2024-04-26T05:29:30.781325+0000" 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "range_blocklist": {}, 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "erasure_code_profiles": { 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "default": { 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "crush-failure-domain": "osd", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "k": "2", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "m": "1", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "plugin": "jerasure", 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "technique": "reed_sol_van" 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_snaps_queue": [], 2024-04-25T05:52:21.315 INFO:teuthology.orchestra.run.smithi043.stdout: "new_removed_snaps": [], 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "new_purged_snaps": [], 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_node_flags": {}, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "device_class_flags": {}, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": { 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_enabled": false, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_bucket_count": 0, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "degraded_stretch_mode": 0, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "recovering_stretch_mode": 0, 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode_bucket": 0 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:21.316 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:52:21.326 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:21.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.477 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:21.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:21 smithi060 ceph-mon[30898]: pgmap v975: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 20852/305256 objects misplaced (6.831%); 24 MiB/s, 381 objects/s recovering 2024-04-25T05:52:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:21 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-25T05:52:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:21 smithi060 ceph-mon[30898]: osdmap e517: 8 total, 8 up, 8 in 2024-04-25T05:52:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:21 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/290169809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:21 smithi060 ceph-mon[30898]: osdmap e518: 8 total, 8 up, 8 in 2024-04-25T05:52:21.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:21.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:21.714 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:21.746 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:21.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:21.877 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:21.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:21.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:22.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.159 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:22.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:22.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:52:22.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:22.287 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:22.388 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2504686398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:52:22.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2504686398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:52:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2504686398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-25T05:52:22.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:22.550 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:22.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:22.681 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:22.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:22.820 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:22.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:22.955 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:22.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:23.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:23.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.226 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:23.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:23.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:23.358 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:23.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.489 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:23.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:23 smithi060 ceph-mon[30898]: pgmap v978: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 20852/305256 objects misplaced (6.831%); 17 MiB/s, 269 objects/s recovering 2024-04-25T05:52:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:23 smithi060 ceph-mon[30898]: osdmap e519: 8 total, 8 up, 8 in 2024-04-25T05:52:23.590 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:23.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:23 smithi043 ceph-mon[27010]: pgmap v978: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 20852/305256 objects misplaced (6.831%); 17 MiB/s, 269 objects/s recovering 2024-04-25T05:52:23.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:23 smithi043 ceph-mon[27010]: osdmap e519: 8 total, 8 up, 8 in 2024-04-25T05:52:23.616 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:23 smithi043 ceph-mon[37492]: pgmap v978: 167 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 162 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 20852/305256 objects misplaced (6.831%); 17 MiB/s, 269 objects/s recovering 2024-04-25T05:52:23.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:23 smithi043 ceph-mon[37492]: osdmap e519: 8 total, 8 up, 8 in 2024-04-25T05:52:23.624 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:23.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.758 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:23.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:23.859 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:23.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:23.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:23.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:24.047 INFO:tasks.thrashosds.thrasher:j is {'epoch': 518, 'fsid': '84884b5e-02c4-11ef-bc93-c7b262605968', 'created': '2024-04-25T05:28:34.192515+0000', 'modified': '2024-04-25T05:52:21.166935+0000', 'last_up_change': '2024-04-25T05:33:46.700068+0000', 'last_in_change': '2024-04-25T05:51:39.176833+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T05:31:44.863613+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 20, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 19, 'last_pg_merge_meta': {'source_pgid': '1.14', 'ready_epoch': 515, 'last_epoch_started': 514, 'last_epoch_clean': 514, 'source_version': "79'128", 'target_version': "79'128"}, 'last_change': '517', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '517', 'last_force_op_resend_preluminous': '517', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-04-25T05:33:56.045664+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-04-25T05:33:57.225173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-25T05:33:58.748103+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-25T05:34:00.700767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-25T05:34:02.767681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-04-25T05:50:10.050814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '461', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9994544982910156, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ef370ce4-85d8-41ed-9655-2e2eaab3b48e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 500, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6801', 'nonce': 3557191248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6802', 'nonce': 3557191248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6804', 'nonce': 3557191248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6803', 'nonce': 3557191248}]}, 'public_addr': '172.21.15.43:6801/3557191248', 'cluster_addr': '172.21.15.43:6802/3557191248', 'heartbeat_back_addr': '172.21.15.43:6804/3557191248', 'heartbeat_front_addr': '172.21.15.43:6803/3557191248', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ed504d08-dc59-4584-bd5f-8059e02a4161', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6805', 'nonce': 235229295}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6806', 'nonce': 235229295}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6808', 'nonce': 235229295}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6807', 'nonce': 235229295}]}, 'public_addr': '172.21.15.43:6805/235229295', 'cluster_addr': '172.21.15.43:6806/235229295', 'heartbeat_back_addr': '172.21.15.43:6808/235229295', 'heartbeat_front_addr': '172.21.15.43:6807/235229295', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '11ff9a46-8686-4b36-a530-5cc1dca1f088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 510, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6809', 'nonce': 3416612512}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6810', 'nonce': 3416612512}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6812', 'nonce': 3416612512}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6811', 'nonce': 3416612512}]}, 'public_addr': '172.21.15.43:6809/3416612512', 'cluster_addr': '172.21.15.43:6810/3416612512', 'heartbeat_back_addr': '172.21.15.43:6812/3416612512', 'heartbeat_front_addr': '172.21.15.43:6811/3416612512', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f3314c45-11c4-46ee-be0b-6f639faa7da2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 515, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6813', 'nonce': 3391766005}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6814', 'nonce': 3391766005}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6816', 'nonce': 3391766005}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.43:6815', 'nonce': 3391766005}]}, 'public_addr': '172.21.15.43:6813/3391766005', 'cluster_addr': '172.21.15.43:6814/3391766005', 'heartbeat_back_addr': '172.21.15.43:6816/3391766005', 'heartbeat_front_addr': '172.21.15.43:6815/3391766005', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '8c5337b2-94d1-4784-9321-db771de75f4c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.995635986328125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 511, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6800', 'nonce': 1310286745}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6801', 'nonce': 1310286745}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6803', 'nonce': 1310286745}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6802', 'nonce': 1310286745}]}, 'public_addr': '172.21.15.60:6800/1310286745', 'cluster_addr': '172.21.15.60:6801/1310286745', 'heartbeat_back_addr': '172.21.15.60:6803/1310286745', 'heartbeat_front_addr': '172.21.15.60:6802/1310286745', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 511, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6804', 'nonce': 3586186657}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6805', 'nonce': 3586186657}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6807', 'nonce': 3586186657}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6806', 'nonce': 3586186657}]}, 'public_addr': '172.21.15.60:6804/3586186657', 'cluster_addr': '172.21.15.60:6805/3586186657', 'heartbeat_back_addr': '172.21.15.60:6807/3586186657', 'heartbeat_front_addr': '172.21.15.60:6806/3586186657', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '5b946188-0dd1-4e68-bcfc-a3daa342edaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 511, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6808', 'nonce': 1675137598}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6809', 'nonce': 1675137598}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6811', 'nonce': 1675137598}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6810', 'nonce': 1675137598}]}, 'public_addr': '172.21.15.60:6808/1675137598', 'cluster_addr': '172.21.15.60:6809/1675137598', 'heartbeat_back_addr': '172.21.15.60:6811/1675137598', 'heartbeat_front_addr': '172.21.15.60:6810/1675137598', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'dec84bb8-9d2b-469f-9641-62ff40b9b9b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 511, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6812', 'nonce': 3982012700}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6813', 'nonce': 3982012700}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6815', 'nonce': 3982012700}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6814', 'nonce': 3982012700}]}, 'public_addr': '172.21.15.60:6812/3982012700', 'cluster_addr': '172.21.15.60:6813/3982012700', 'heartbeat_back_addr': '172.21.15.60:6815/3982012700', 'heartbeat_front_addr': '172.21.15.60:6814/3982012700', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:43.706624+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:21.710639+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:58.548355+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:13.273647+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:30.212991+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:10.067291+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:28.450013+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T05:51:37.706536+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.6', 'osds': [5, 1, 7]}, {'pgid': '5.1c', 'osds': [6, 0, 5]}, {'pgid': '6.7', 'osds': [1, 2, 4]}], 'pg_upmap_items': [{'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.0', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1a', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.3', 'mappings': [{'from': 7, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.1', 'osds': [6, 3, 2]}, {'pgid': '11.2', 'osds': [2, 3, 7]}, {'pgid': '11.6', 'osds': [1, 2, 4]}, {'pgid': '11.9', 'osds': [3, 6, 1]}, {'pgid': '11.b', 'osds': [3, 1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.43:0/3423182021': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/2082036420': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3958950586': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/3953560108': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/4077364595': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/830746728': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:6800/786479021': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/3735093998': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1835545567': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:6800/149142905': '2024-04-26T05:34:31.636703+0000', '172.21.15.43:0/1408327650': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/2239747746': '2024-04-26T05:29:02.328794+0000', '172.21.15.43:0/4084484879': '2024-04-26T05:29:30.781325+0000', '172.21.15.43:0/1247294293': '2024-04-26T05:29:30.781325+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T05:52:24.047 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.6 2024-04-25T05:52:24.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd rm-pg-upmap 4.6 2024-04-25T05:52:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:24.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:24.093 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:24.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:24.220 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:24.221 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:24.322 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:24.355 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:24.394 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:24 smithi043 ceph-mon[27010]: osdmap e520: 8 total, 8 up, 8 in 2024-04-25T05:52:24.395 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:24 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-25T05:52:24.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:24 smithi043 ceph-mon[37492]: osdmap e520: 8 total, 8 up, 8 in 2024-04-25T05:52:24.395 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:24 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-25T05:52:24.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:24 smithi060 ceph-mon[30898]: osdmap e520: 8 total, 8 up, 8 in 2024-04-25T05:52:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:24 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-25T05:52:24.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:24.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:24.654 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:24.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:24.788 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:24.890 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:24.938 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:24.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:25.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:25.068 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:25.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:25.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:25.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:25.325 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:25.350 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[27010]: pgmap v981: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 730 op/s; 19599/296499 objects misplaced (6.610%); 16 MiB/s, 252 objects/s recovering 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[27010]: osdmap e521: 8 total, 8 up, 8 in 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[27010]: osdmap e522: 8 total, 8 up, 8 in 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[37492]: pgmap v981: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 730 op/s; 19599/296499 objects misplaced (6.610%); 16 MiB/s, 252 objects/s recovering 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-25T05:52:25.351 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[37492]: osdmap e521: 8 total, 8 up, 8 in 2024-04-25T05:52:25.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:25 smithi043 ceph-mon[37492]: osdmap e522: 8 total, 8 up, 8 in 2024-04-25T05:52:25.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:52:25.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.466 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:25.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:25 smithi060 ceph-mon[30898]: pgmap v981: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 730 op/s; 19599/296499 objects misplaced (6.610%); 16 MiB/s, 252 objects/s recovering 2024-04-25T05:52:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:25 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-25T05:52:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:25 smithi060 ceph-mon[30898]: osdmap e521: 8 total, 8 up, 8 in 2024-04-25T05:52:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:25 smithi060 ceph-mon[30898]: osdmap e522: 8 total, 8 up, 8 in 2024-04-25T05:52:25.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:25.590 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:25.692 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:25.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:25.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:25.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:25.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155160 9.6 9:68737ed0:::benchmark_data_smithi043_129820_object33846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.24634889999999, 2024-04-25T05:52:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4685459839999999, 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155160 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:52:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054726+0000", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4890000000000001e-06 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054732+0000", 2024-04-25T05:52:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0260000000000004e-06 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054747+0000", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4762e-05 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054756+0000", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2070000000000001e-06 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.056987+0000", 2024-04-25T05:52:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022311950000000001 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.057103+0000", 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115751 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6,7", 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.062978+0000", 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058742639999999997 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.065122+0000", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021449189999999999 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.065931+0000", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080820200000000005 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.081687+0000", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015756503000000002 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:25.996 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.081698+0000", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0569e-05 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519331+0000", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4376332760000001 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519417+0000", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5554000000000001e-05 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:25.997 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523215+0000", 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037980840000000002 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523271+0000", 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6183000000000001e-05 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155182 9.6 9:64768de4:::benchmark_data_smithi043_129820_object33868:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.22505482700001, 2024-04-25T05:52:25.998 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.44742573, 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155182 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:25.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076020+0000", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3370000000000001e-06 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076026+0000", 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1090000000000003e-06 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.000 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076038+0000", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1559999999999999e-05 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076046+0000", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7530000000000001e-06 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083212+0000", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071663600000000001 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.001 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083220+0000", 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3010000000000007e-06 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519718+0000", 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4364973889999999 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519802+0000", 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4505000000000005e-05 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.002 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519898+0000", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5798999999999995e-05 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523005+0000", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031065419999999999 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523303+0000", 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297977 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.003 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523339+0000", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6943999999999997e-05 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523369+0000", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9410000000000001e-05 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523445+0000", 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5743999999999999e-05 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155187 9.6 9:62aab14c:::benchmark_data_smithi043_129820_object33873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.22015722899999, 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.445533349, 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155187 2024-04-25T05:52:26.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080917+0000", 2024-04-25T05:52:26.006 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080922+0000", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.989e-06 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080933+0000", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0903e-05 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080941+0000", 2024-04-25T05:52:26.007 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.96e-06 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083245+0000", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023036889999999998 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083252+0000", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8340000000000001e-06 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521237+0000", 2024-04-25T05:52:26.008 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4379850830000001 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521310+0000", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3540000000000004e-05 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521382+0000", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1266000000000005e-05 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523269+0000", 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001887122 2024-04-25T05:52:26.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526171+0000", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029016379999999998 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526300+0000", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012934099999999999 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526353+0000", 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3245999999999998e-05 2024-04-25T05:52:26.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526450+0000", 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6738999999999997e-05 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155197 9.6 9:60d19fde:::benchmark_data_smithi043_129820_object33883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.21407435999998, 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.439703379, 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155197 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.012 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087000+0000", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4300000000000003e-07 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087003+0000", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0079999999999998e-06 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087010+0000", 2024-04-25T05:52:26.013 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.793e-06 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087016+0000", 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7549999999999997e-06 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087222+0000", 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000205167 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.014 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087231+0000", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0329999999999997e-06 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522178+0000", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4349474170000001 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522246+0000", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7964000000000002e-05 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.015 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522316+0000", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0087999999999997e-05 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523459+0000", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011426279999999999 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526260+0000", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002801455 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.016 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526528+0000", 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00026746199999999999 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526589+0000", 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1754999999999994e-05 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526703+0000", 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113311 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155207 9.6 9:6f7e51c0:::benchmark_data_smithi043_129820_object33893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.20255892400002, 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4315835130000001, 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155207 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.018 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098516+0000", 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.423e-06 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098521+0000", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.809e-06 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098530+0000", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3079999999999997e-06 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098538+0000", 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3000000000000004e-06 2024-04-25T05:52:26.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098698+0000", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000159975 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098707+0000", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3749999999999992e-06 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523520+0000", 2024-04-25T05:52:26.021 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4248128410000001 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523674+0000", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154444 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523838+0000", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000164067 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526225+0000", 2024-04-25T05:52:26.022 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023871679999999998 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526772+0000", 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00054641399999999995 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.529943+0000", 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031711209999999998 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.023 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.529996+0000", 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3053000000000002e-05 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530098+0000", 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000102215 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155209 9.6 9:69e43cc6:::benchmark_data_smithi043_129820_object33895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.20250533400002, 2024-04-25T05:52:26.024 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.432188107, 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155209 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098569+0000", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1900000000000002e-07 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098572+0000", 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1999999999999999e-06 2024-04-25T05:52:26.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098579+0000", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8499999999999996e-06 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098585+0000", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4659999999999998e-06 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098738+0000", 2024-04-25T05:52:26.027 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015333 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098745+0000", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1080000000000004e-06 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525155+0000", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.42640999 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525248+0000", 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3037999999999999e-05 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525316+0000", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8225999999999994e-05 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526464+0000", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011481219999999999 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530363+0000", 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038985299999999999 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530405+0000", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1757e-05 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530499+0000", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4128000000000004e-05 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530756+0000", 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000257743 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155210 9.6 9:62a79c7f:::benchmark_data_smithi043_129820_object33896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.20248079200002, 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4385105970000001, 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155210 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.031 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3300000000000003e-07 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098596+0000", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1619999999999998e-06 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098601+0000", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6149999999999996e-06 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098606+0000", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1189999999999996e-06 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.033 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098767+0000", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016068300000000001 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098773+0000", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4409999999999998e-06 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526824+0000", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4280502369999999 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.034 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526982+0000", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015850099999999999 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.527143+0000", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016125800000000001 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530317+0000", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031736619999999998 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.035 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.533960+0000", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036430019999999998 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536942+0000", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029821040000000002 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537000+0000", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7896999999999998e-05 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537104+0000", 2024-04-25T05:52:26.036 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000103483 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155231 9.6 9:6c10c19d:::benchmark_data_smithi043_129820_object33917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.18199148799999, 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4224660419999999, 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155231 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.038 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119084+0000", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4279999999999999e-06 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119090+0000", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2899999999999999e-06 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119101+0000", 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1277999999999999e-05 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.039 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119110+0000", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1130000000000007e-06 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119205+0000", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4975000000000005e-05 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119216+0000", 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0268000000000001e-05 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.040 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528403+0000", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4091876059999999 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528524+0000", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012039799999999999 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528655+0000", 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013160099999999999 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.041 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530322+0000", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001667167 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536752+0000", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064295860000000002 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541312+0000", 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045599179999999996 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.042 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541439+0000", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012706499999999999 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541548+0000", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109349 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155286 9.6 9:62058b0b:::benchmark_data_smithi043_129820_object33972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.07543316700003, 2024-04-25T05:52:26.043 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3162657389999999, 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155286 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.75e-07 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225646+0000", 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4710000000000001e-06 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.045 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225654+0000", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7190000000000007e-06 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225662+0000", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6510000000000001e-06 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225716+0000", 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4165000000000001e-05 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.046 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225725+0000", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3580000000000003e-06 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530799+0000", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.305074265 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530941+0000", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014229499999999999 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.047 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.531083+0000", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014207999999999999 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.533984+0000", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029003129999999998 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536825+0000", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002841784 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541749+0000", 2024-04-25T05:52:26.048 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049237810000000003 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541805+0000", 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6110000000000003e-05 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541906+0000", 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101072 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.049 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155290 9.6 9:6b6bab7a:::benchmark_data_smithi043_129820_object33976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.07277855000001, 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.317358917, 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155290 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.050 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228297+0000", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.359e-06 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.051 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228301+0000", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0860000000000004e-06 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228309+0000", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9799999999999998e-06 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228315+0000", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8519999999999998e-06 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228486+0000", 2024-04-25T05:52:26.052 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017009600000000001 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228494+0000", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.123e-06 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532331+0000", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3038373779999999 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532444+0000", 2024-04-25T05:52:26.053 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113113 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532608+0000", 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000163707 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536773+0000", 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041653539999999996 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.054 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537161+0000", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00038807199999999999 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545511+0000", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0083501449999999998 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545562+0000", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0943999999999999e-05 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.055 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545654+0000", 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1707999999999997e-05 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155323 9.6 9:60343a60:::benchmark_data_smithi043_129820_object34009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.05197071999999, 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2987439169999999, 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.056 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155323 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249104+0000", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5000000000000002e-07 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249108+0000", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3080000000000001e-06 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249117+0000", 2024-04-25T05:52:26.058 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5180000000000004e-06 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249124+0000", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249210+0000", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6537000000000002e-05 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249220+0000", 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8220000000000002e-06 2024-04-25T05:52:26.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534015+0000", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2847948680000001 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534132+0000", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117316 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534248+0000", 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011565699999999999 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537116+0000", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028682479999999999 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537205+0000", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8677999999999994e-05 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547761+0000", 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010556204 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547809+0000", 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7682000000000002e-05 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547847+0000", 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8117000000000001e-05 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155332 9.6 9:63fa8193:::benchmark_data_smithi043_129820_object34018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:52:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 429.04419105900001, 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2911138019999999, 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155332 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256884+0000", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6099999999999999e-07 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256888+0000", 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5789999999999997e-06 2024-04-25T05:52:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256899+0000", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1071e-05 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256907+0000", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7109999999999995e-06 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256960+0000", 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2632e-05 2024-04-25T05:52:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256965+0000", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8270000000000003e-06 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535461+0000", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.278495739 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535548+0000", 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7292000000000004e-05 2024-04-25T05:52:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535700+0000", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015115699999999999 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537122+0000", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014224750000000001 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541968+0000", 2024-04-25T05:52:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0048458190000000003 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547897+0000", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059294289999999999 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547940+0000", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3050000000000003e-05 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547996+0000", 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6060000000000002e-05 2024-04-25T05:52:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94216 11.0 11:04369c26:::benchmark_data_smithi043_179003_object94215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.817188889999997, 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.239603538, 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94216 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483886+0000", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.68e-06 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483911+0000", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4275000000000001e-05 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483927+0000", 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6209e-05 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483936+0000", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6270000000000006e-06 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483995+0000", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9590999999999999e-05 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.484129+0000", 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000134184 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.484266+0000", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000136938 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.487255+0000", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002988465 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.489356+0000", 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002100874 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723380+0000", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.23402442500000001 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723437+0000", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6498e-05 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723488+0000", 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1771999999999998e-05 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94230 11.0 11:0c3febce:::benchmark_data_smithi043_179003_object94229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.766935626000006, 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.193247635, 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94230 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534140+0000", 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3350000000000001e-06 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534168+0000", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7818999999999999e-05 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534190+0000", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1359e-05 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534203+0000", 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3040000000000001e-05 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534285+0000", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2291000000000002e-05 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534464+0000", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017857600000000001 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534640+0000", 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017681899999999999 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.539282+0000", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046420999999999997 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.539676+0000", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00039339799999999999 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727295+0000", 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18761967500000001 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727360+0000", 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4937000000000005e-05 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727386+0000", 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5286000000000002e-05 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94244 11.0 11:09c229f5:::benchmark_data_smithi043_179003_object94243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.749581398999993, 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17984397599999999, 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94244 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:52:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551494+0000", 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551516+0000", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2483000000000001e-05 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551546+0000", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9730000000000002e-05 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551784+0000", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000238142 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552058+0000", 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027341399999999998 2024-04-25T05:52:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552330+0000", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027213399999999999 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552513+0000", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018327 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.555341+0000", 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028275739999999998 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562661+0000", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0073205220000000003 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731226+0000", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16856490299999999 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731286+0000", 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9462000000000001e-05 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731336+0000", 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0702999999999999e-05 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94258 11.0 11:0e232a3d:::benchmark_data_smithi043_179003_object94257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.738682893999993, 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.169532664, 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94258 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:52:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3e-07 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562445+0000", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3516000000000001e-05 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562453+0000", 2024-04-25T05:52:26.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3319999999999992e-06 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562461+0000", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5270000000000001e-06 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562694+0000", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000233327 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562833+0000", 2024-04-25T05:52:26.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138596 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.563001+0000", 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016854399999999999 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.565736+0000", 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002735038 2024-04-25T05:52:26.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.570245+0000", 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045089120000000003 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731812+0000", 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.161567138 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731883+0000", 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0382000000000005e-05 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731923+0000", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0722000000000001e-05 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94295 11.0 11:00ae873d:::benchmark_data_smithi043_179003_object94294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.710294070000003, 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14333959800000001, 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.095 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94295 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590781+0000", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.265e-06 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590801+0000", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9935000000000001e-05 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590811+0000", 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0329999999999999e-05 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590817+0000", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.964e-06 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590891+0000", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3915e-05 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.591038+0000", 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014694799999999999 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.591208+0000", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000170464 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.593800+0000", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025914570000000001 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598539+0000", 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047392720000000001 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734007+0000", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.135468218 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734077+0000", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9395999999999998e-05 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734119+0000", 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2434000000000001e-05 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94309 11.0 11:02925c71:::benchmark_data_smithi043_179003_object94308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.702162572999995, 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14004730400000001, 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94309 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598913+0000", 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3999999999999999e-06 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.102 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598970+0000", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7605999999999997e-05 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598981+0000", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1052e-05 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598989+0000", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599043+0000", 2024-04-25T05:52:26.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3981999999999999e-05 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599210+0000", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000166799 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599359+0000", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000149071 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.602209+0000", 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002850024 2024-04-25T05:52:26.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.609505+0000", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0072964589999999999 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738834+0000", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.129328478 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738911+0000", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7008999999999994e-05 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738958+0000", 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7487999999999997e-05 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94332 11.0 11:0724872d:::benchmark_data_smithi043_179003_object94331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.618728+0000", 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.682345294000001, 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.122714588, 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.106 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94332 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618728+0000", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618728+0000", 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618730+0000", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.305e-06 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618748+0000", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8681999999999998e-05 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618758+0000", 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2839999999999995e-06 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.108 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618763+0000", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7939999999999999e-06 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.618822+0000", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8695e-05 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.619031+0000", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020923200000000001 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.619424+0000", 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000392266 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.621970+0000", 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025466830000000001 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.623065+0000", 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0010942529999999999 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741349+0000", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11828406399999999 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741400+0000", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1230000000000002e-05 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741443+0000", 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3099999999999997e-05 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.111 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94351 11.0 11:0bea7d55:::benchmark_data_smithi043_179003_object94350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.633885+0000", 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 68.667188769000006, 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.10770543, 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94351 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:26.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633885+0000", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633885+0000", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633886+0000", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.345e-06 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:26.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633913+0000", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6406000000000001e-05 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633924+0000", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1459e-05 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633930+0000", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3040000000000001e-06 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:26.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.633987+0000", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6623999999999999e-05 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.634137+0000", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015020800000000001 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.634325+0000", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000187908 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.636946+0000", 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002621047 2024-04-25T05:52:26.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.639979+0000", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030323849999999999 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741491+0000", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.101512449 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741548+0000", 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7309999999999998e-05 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:26.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.741590+0000", 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1984999999999999e-05 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:26.117 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:52:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.119 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:26.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:26.219 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:26.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:26.389 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:26.491 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:26.519 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:26.621 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:26.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:26.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.784 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:26.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:26.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:26.924 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:26.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:27.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.068 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:27.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:27.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.203 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:27.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:27.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:27 smithi043 ceph-mon[27010]: pgmap v984: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 18369/289920 objects misplaced (6.336%); 23 MiB/s, 364 objects/s recovering 2024-04-25T05:52:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:27 smithi043 ceph-mon[27010]: osdmap e523: 8 total, 8 up, 8 in 2024-04-25T05:52:27.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:27 smithi043 ceph-mon[37492]: pgmap v984: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 18369/289920 objects misplaced (6.336%); 23 MiB/s, 364 objects/s recovering 2024-04-25T05:52:27.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:27 smithi043 ceph-mon[37492]: osdmap e523: 8 total, 8 up, 8 in 2024-04-25T05:52:27.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.475 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:27.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:27 smithi060 ceph-mon[30898]: pgmap v984: 166 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 161 active+clean, 3 active+remapped+backfilling; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 18369/289920 objects misplaced (6.336%); 23 MiB/s, 364 objects/s recovering 2024-04-25T05:52:27.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:27 smithi060 ceph-mon[30898]: osdmap e523: 8 total, 8 up, 8 in 2024-04-25T05:52:27.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:27.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:27.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:27.749 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:27.851 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:27.887 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:27.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:27.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:28.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:28.142 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:28.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:28.277 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:28.379 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:28 smithi043 ceph-mon[27010]: osdmap e524: 8 total, 8 up, 8 in 2024-04-25T05:52:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:28 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-25T05:52:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:28 smithi043 ceph-mon[37492]: osdmap e524: 8 total, 8 up, 8 in 2024-04-25T05:52:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:28 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-25T05:52:28.409 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:28 smithi060 ceph-mon[30898]: osdmap e524: 8 total, 8 up, 8 in 2024-04-25T05:52:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:28 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-25T05:52:28.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:28.544 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:28.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:28.687 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:28.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:28.818 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:28.919 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:28.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:29.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:29.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:29.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:29.229 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:29.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:29.369 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:29.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[27010]: pgmap v987: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 160 active+clean, 3 active+remapped+backfilling; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 17786/287844 objects misplaced (6.179%); 14 MiB/s, 226 objects/s recovering 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[27010]: osdmap e525: 8 total, 8 up, 8 in 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[37492]: pgmap v987: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 160 active+clean, 3 active+remapped+backfilling; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 17786/287844 objects misplaced (6.179%); 14 MiB/s, 226 objects/s recovering 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-25T05:52:29.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-mon[37492]: osdmap e525: 8 total, 8 up, 8 in 2024-04-25T05:52:29.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:29 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:29 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:29 smithi060 ceph-mon[30898]: pgmap v987: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 160 active+clean, 3 active+remapped+backfilling; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 17786/287844 objects misplaced (6.179%); 14 MiB/s, 226 objects/s recovering 2024-04-25T05:52:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:29 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-25T05:52:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:29 smithi060 ceph-mon[30898]: osdmap e525: 8 total, 8 up, 8 in 2024-04-25T05:52:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:29.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:29.615 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:29.657 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:29.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:29.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:29.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:29.936 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:29.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:30.038 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:30.076 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:30.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:30.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:30.307 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:30.341 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:30.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:30 smithi043 ceph-mon[37492]: osdmap e526: 8 total, 8 up, 8 in 2024-04-25T05:52:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:30 smithi043 ceph-mon[27010]: osdmap e526: 8 total, 8 up, 8 in 2024-04-25T05:52:30.443 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:30.472 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:30 smithi060 ceph-mon[30898]: osdmap e526: 8 total, 8 up, 8 in 2024-04-25T05:52:30.574 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:30.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:30.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:30.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:30.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:30.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:30.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:30.986 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:31.012 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:31.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:31.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:31.289 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:31.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:52:31.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:31 smithi043 ceph-mon[37492]: pgmap v990: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 15891/274434 objects misplaced (5.790%); 25 MiB/s, 394 objects/s recovering 2024-04-25T05:52:31.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:31 smithi043 ceph-mon[37492]: osdmap e527: 8 total, 8 up, 8 in 2024-04-25T05:52:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:31 smithi043 ceph-mon[27010]: pgmap v990: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 15891/274434 objects misplaced (5.790%); 25 MiB/s, 394 objects/s recovering 2024-04-25T05:52:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:31 smithi043 ceph-mon[27010]: osdmap e527: 8 total, 8 up, 8 in 2024-04-25T05:52:31.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:31 smithi060 ceph-mon[30898]: pgmap v990: 165 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 2 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 15891/274434 objects misplaced (5.790%); 25 MiB/s, 394 objects/s recovering 2024-04-25T05:52:31.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:31 smithi060 ceph-mon[30898]: osdmap e527: 8 total, 8 up, 8 in 2024-04-25T05:52:31.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:31.543 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:31.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:31.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:31.927 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:31.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:32.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.115 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:32.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:32.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:32.244 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:32.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:32.383 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:32 smithi043 ceph-mon[27010]: osdmap e528: 8 total, 8 up, 8 in 2024-04-25T05:52:32.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:32 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-25T05:52:32.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:32 smithi043 ceph-mon[37492]: osdmap e528: 8 total, 8 up, 8 in 2024-04-25T05:52:32.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:32 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-25T05:52:32.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:32 smithi060 ceph-mon[30898]: osdmap e528: 8 total, 8 up, 8 in 2024-04-25T05:52:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:32 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-25T05:52:32.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:32.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:32.637 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:32.739 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:32.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:32.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:32.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:32.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:33.007 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:33.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:33.019 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:33.101 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:33.185 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:52:33.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:33.231 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:33.333 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:33.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:33.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: pgmap v993: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 15891/274434 objects misplaced (5.790%); 17 MiB/s, 279 objects/s recovering 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: osdmap e529: 8 total, 8 up, 8 in 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/810865662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[27010]: osdmap e530: 8 total, 8 up, 8 in 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: pgmap v993: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 15891/274434 objects misplaced (5.790%); 17 MiB/s, 279 objects/s recovering 2024-04-25T05:52:33.489 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: osdmap e529: 8 total, 8 up, 8 in 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/810865662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:52:33.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:33 smithi043 ceph-mon[37492]: osdmap e530: 8 total, 8 up, 8 in 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: pgmap v993: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 158 active+clean, 3 active+remapped+backfilling; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 15891/274434 objects misplaced (5.790%); 17 MiB/s, 279 objects/s recovering 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: osdmap e529: 8 total, 8 up, 8 in 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/810865662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:52:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-mon[30898]: osdmap e530: 8 total, 8 up, 8 in 2024-04-25T05:52:33.518 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:33.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.655 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:33.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:33.756 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:33.796 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:33.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:33.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:34.036 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:34.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:34.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:34.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:34.310 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:34.351 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:34.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:34.480 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:34 smithi060 ceph-mon[30898]: osdmap e531: 8 total, 8 up, 8 in 2024-04-25T05:52:34.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:34.601 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:34.607 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:34.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:34.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:34 smithi043 ceph-mon[37492]: osdmap e531: 8 total, 8 up, 8 in 2024-04-25T05:52:34.616 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:34.617 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:34 smithi043 ceph-mon[27010]: osdmap e531: 8 total, 8 up, 8 in 2024-04-25T05:52:34.709 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:34.755 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:34.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:34.889 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:34.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:34.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:35.031 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:35.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.159 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:35.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:35.260 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:35.393 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: pgmap v996: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 clean+premerge+peered, 1 active+remapped+backfill_wait, 157 active+clean, 3 active+remapped+backfilling; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 662 op/s; 14683/266490 objects misplaced (5.510%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.d scrub starts 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: 1.d scrub ok 2024-04-25T05:52:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:35 smithi060 ceph-mon[30898]: osdmap e532: 8 total, 8 up, 8 in 2024-04-25T05:52:35.524 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:35.551 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: pgmap v996: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 clean+premerge+peered, 1 active+remapped+backfill_wait, 157 active+clean, 3 active+remapped+backfilling; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 662 op/s; 14683/266490 objects misplaced (5.510%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:52:35.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.d scrub starts 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: 1.d scrub ok 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[27010]: osdmap e532: 8 total, 8 up, 8 in 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: pgmap v996: 164 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 clean+premerge+peered, 1 active+remapped+backfill_wait, 157 active+clean, 3 active+remapped+backfilling; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 662 op/s; 14683/266490 objects misplaced (5.510%); 17 MiB/s, 278 objects/s recovering 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.d scrub starts 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: 1.d scrub ok 2024-04-25T05:52:35.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:35 smithi043 ceph-mon[37492]: osdmap e532: 8 total, 8 up, 8 in 2024-04-25T05:52:35.653 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:35.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:35.806 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:35.908 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:35.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:36.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:36.058 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:36.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:36.190 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:36.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:36.322 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:36.424 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:36.465 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-25T05:52:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:36 smithi060 ceph-mon[30898]: osdmap e533: 8 total, 8 up, 8 in 2024-04-25T05:52:36.567 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:36.599 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:52:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:52:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:52:36.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[27010]: osdmap e533: 8 total, 8 up, 8 in 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-25T05:52:36.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:36 smithi043 ceph-mon[37492]: osdmap e533: 8 total, 8 up, 8 in 2024-04-25T05:52:36.701 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:36.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:36.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:36.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:36.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:36.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:37.000 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:37.102 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:37.245 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:37.272 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:37.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: pgmap v999: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 13517/260088 objects misplaced (5.197%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:37 smithi060 ceph-mon[30898]: osdmap e534: 8 total, 8 up, 8 in 2024-04-25T05:52:37.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:37.546 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: pgmap v999: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 13517/260088 objects misplaced (5.197%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[27010]: osdmap e534: 8 total, 8 up, 8 in 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: pgmap v999: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 13517/260088 objects misplaced (5.197%); 25 MiB/s, 392 objects/s recovering 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:37 smithi043 ceph-mon[37492]: osdmap e534: 8 total, 8 up, 8 in 2024-04-25T05:52:37.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:37.683 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:37.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:37.819 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:37.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:37.950 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:37.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:38.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:38.089 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:38.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.233 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:38.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:38.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:38.376 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[37492]: 1.e scrub starts 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[37492]: 1.e scrub ok 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[37492]: 11.5 scrub starts 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[37492]: osdmap e535: 8 total, 8 up, 8 in 2024-04-25T05:52:38.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[27010]: 1.e scrub starts 2024-04-25T05:52:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[27010]: 1.e scrub ok 2024-04-25T05:52:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[27010]: 11.5 scrub starts 2024-04-25T05:52:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:52:38.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:38 smithi043 ceph-mon[27010]: osdmap e535: 8 total, 8 up, 8 in 2024-04-25T05:52:38.478 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:38.505 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:38 smithi060 ceph-mon[30898]: 1.e scrub starts 2024-04-25T05:52:38.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:38 smithi060 ceph-mon[30898]: 1.e scrub ok 2024-04-25T05:52:38.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:38 smithi060 ceph-mon[30898]: 11.5 scrub starts 2024-04-25T05:52:38.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:38 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:52:38.506 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:38 smithi060 ceph-mon[30898]: osdmap e535: 8 total, 8 up, 8 in 2024-04-25T05:52:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:38.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:38.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:38.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:38.791 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:38.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:38.929 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:38.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:39.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:39.055 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:39.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:39.184 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:39.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:39.291 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:39.314 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:39.343 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:39.413 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:39.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:39.444 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:39.473 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:52:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:52:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: pgmap v1002: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 671 op/s; 12716/258432 objects misplaced (4.920%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:52:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: osdmap e536: 8 total, 8 up, 8 in 2024-04-25T05:52:39.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2204645433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:39.520 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:39.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:52:39.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:52:39.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: pgmap v1002: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 671 op/s; 12716/258432 objects misplaced (4.920%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:52:39.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:39.520 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: osdmap e536: 8 total, 8 up, 8 in 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2204645433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: pgmap v1002: 163 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_wait, 1 clean+premerge+peered, 157 active+clean, 3 active+remapped+backfilling; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 671 op/s; 12716/258432 objects misplaced (4.920%); 19 MiB/s, 304 objects/s recovering 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:39.521 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:52:39.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:52:39.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:52:39.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:52:39.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: osdmap e536: 8 total, 8 up, 8 in 2024-04-25T05:52:39.522 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2204645433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:39.533 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:52:39.533 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:52:39.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:39.569 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:39.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:39.614 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:39.666 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:39.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:39.702 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:52:39.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:39.776 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:39.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:39.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:39.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:39.984 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:40.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:40.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:40.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:40.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: 11.5 scrub ok 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: 11.5 scrub ok 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:52:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:52:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:52:40.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:40 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-25T05:52:40.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:40.442 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: 11.5 scrub ok 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:52:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:40 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-25T05:52:40.544 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:40.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:40.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:40.715 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:40.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:40.843 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:40.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:40.979 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:40.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:41.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:41.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:41.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.238 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:41.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:41.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:41.482 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.587 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:41.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: pgmap v1005: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 10461/247431 objects misplaced (4.228%); 39 MiB/s, 625 objects/s recovering 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: osdmap e537: 8 total, 8 up, 8 in 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[27010]: 11.e scrub starts 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:52:41.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: pgmap v1005: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 10461/247431 objects misplaced (4.228%); 39 MiB/s, 625 objects/s recovering 2024-04-25T05:52:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-25T05:52:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: osdmap e537: 8 total, 8 up, 8 in 2024-04-25T05:52:41.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:41 smithi043 ceph-mon[37492]: 11.e scrub starts 2024-04-25T05:52:41.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:41.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: pgmap v1005: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 10461/247431 objects misplaced (4.228%); 39 MiB/s, 625 objects/s recovering 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: osdmap e537: 8 total, 8 up, 8 in 2024-04-25T05:52:41.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:41 smithi060 ceph-mon[30898]: 11.e scrub starts 2024-04-25T05:52:41.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:41.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:41.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:41.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:41.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:42.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:42.135 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:42.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:42.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:52:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:42.326 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:42.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:42.422 INFO:teuthology.orchestra.run.smithi043.stderr:clear 4.6 pg_upmap mapping 2024-04-25T05:52:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:42.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:42.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: osdmap e538: 8 total, 8 up, 8 in 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3106266194' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:52:42.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: osdmap e538: 8 total, 8 up, 8 in 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3106266194' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:52:42.600 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:42 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:52:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:42.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:42.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:52:42.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:42.740 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: osdmap e538: 8 total, 8 up, 8 in 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3106266194' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]: dispatch 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:52:42.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:42 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:52:42.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:42.863 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:42.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:42.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:43.010 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:43.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:43.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:43.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:43.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:43.423 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:43.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:43.560 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:43.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:43.687 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: pgmap v1008: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10461/247431 objects misplaced (4.228%); 27 MiB/s, 434 objects/s recovering 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:52:43.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-25T05:52:43.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: osdmap e539: 8 total, 8 up, 8 in 2024-04-25T05:52:43.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 11.e scrub ok 2024-04-25T05:52:43.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:52:43.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:52:43.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.811 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:43.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: pgmap v1008: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10461/247431 objects misplaced (4.228%); 27 MiB/s, 434 objects/s recovering 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: osdmap e539: 8 total, 8 up, 8 in 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 11.e scrub ok 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:52:43.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: pgmap v1008: 162 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 1 peering, 156 active+clean, 3 active+remapped+backfilling; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10461/247431 objects misplaced (4.228%); 27 MiB/s, 434 objects/s recovering 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.6"}]': finished 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: osdmap e539: 8 total, 8 up, 8 in 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 11.e scrub ok 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:52:43.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:43 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:52:43.913 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:43.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:44.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:44.057 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:44.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:44.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:44.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:44.310 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:44.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:44.444 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:44.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:44.584 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:44.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:44.711 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: osdmap e540: 8 total, 8 up, 8 in 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-25T05:52:44.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:44 smithi060 ceph-mon[30898]: osdmap e541: 8 total, 8 up, 8 in 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: osdmap e540: 8 total, 8 up, 8 in 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-25T05:52:44.761 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[27010]: osdmap e541: 8 total, 8 up, 8 in 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: osdmap e540: 8 total, 8 up, 8 in 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-25T05:52:44.762 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:44 smithi043 ceph-mon[37492]: osdmap e541: 8 total, 8 up, 8 in 2024-04-25T05:52:44.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:44.835 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:44.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:44.960 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:44.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:45.061 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:45.107 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:45.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:45.249 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:45.351 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:45.376 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:45.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:45.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:45.626 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:45.656 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: pgmap v1011: 161 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 157 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 633 op/s; 9961/239844 objects misplaced (4.153%); 7.5 MiB/s, 120 objects/s recovering 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 5.16 scrub starts 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 5.16 scrub ok 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:52:45.724 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 11.d scrub starts 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 1.b scrub starts 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 1.b scrub ok 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: osdmap e542: 8 total, 8 up, 8 in 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2674974617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:45.725 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: pgmap v1011: 161 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 157 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 633 op/s; 9961/239844 objects misplaced (4.153%); 7.5 MiB/s, 120 objects/s recovering 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 5.16 scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 5.16 scrub ok 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 11.d scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 1.b scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 1.b scrub ok 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:52:45.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: osdmap e542: 8 total, 8 up, 8 in 2024-04-25T05:52:45.727 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2674974617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.727 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: pgmap v1011: 161 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 157 active+clean, 2 active+remapped+backfilling; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 633 op/s; 9961/239844 objects misplaced (4.153%); 7.5 MiB/s, 120 objects/s recovering 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 5.16 scrub starts 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 5.16 scrub ok 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:52:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 11.d scrub starts 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 1.b scrub starts 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 1.b scrub ok 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: osdmap e542: 8 total, 8 up, 8 in 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2674974617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:52:45.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:45.784 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:45.842 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:52:45.842 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:52:45.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:45.914 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:45.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:46.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:46.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:46.107 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:52:46.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:46.183 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:46.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.315 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:46.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:46.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:46.440 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:46.542 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:46.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:46.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.725 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:46.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 11.d scrub ok 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: 6.17 scrub starts 2024-04-25T05:52:46.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:52:46.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:46.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:46 smithi060 ceph-mon[30898]: osdmap e543: 8 total, 8 up, 8 in 2024-04-25T05:52:46.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:46.855 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:52:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:52:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:52:46.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 11.d scrub ok 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: 6.17 scrub starts 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[27010]: osdmap e543: 8 total, 8 up, 8 in 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:52:46.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:52:46.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 11.d scrub ok 2024-04-25T05:52:46.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: 6.17 scrub starts 2024-04-25T05:52:46.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:52:46.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:52:46.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:46 smithi043 ceph-mon[37492]: osdmap e543: 8 total, 8 up, 8 in 2024-04-25T05:52:46.957 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:46.991 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:46.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:47.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:47.137 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:47.239 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:47.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:47.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:47.399 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:47.501 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:47.530 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:47.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:47.672 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: pgmap v1014: 161 pgs: 2 active+remapped+backfilling, 2 active+clean+remapped, 157 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 7703/234216 objects misplaced (3.289%); 35 MiB/s, 564 objects/s recovering 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: 6.17 scrub ok 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:52:47.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:47 smithi060 ceph-mon[30898]: osdmap e544: 8 total, 8 up, 8 in 2024-04-25T05:52:47.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: pgmap v1014: 161 pgs: 2 active+remapped+backfilling, 2 active+clean+remapped, 157 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 7703/234216 objects misplaced (3.289%); 35 MiB/s, 564 objects/s recovering 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: 6.17 scrub ok 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[27010]: osdmap e544: 8 total, 8 up, 8 in 2024-04-25T05:52:47.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: pgmap v1014: 161 pgs: 2 active+remapped+backfilling, 2 active+clean+remapped, 157 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 7703/234216 objects misplaced (3.289%); 35 MiB/s, 564 objects/s recovering 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: 6.17 scrub ok 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:52:47.790 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:47 smithi043 ceph-mon[37492]: osdmap e544: 8 total, 8 up, 8 in 2024-04-25T05:52:47.794 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:47.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:47.846 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:47.847 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:52:47.888 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:47.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:47.944 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:47.946 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:47.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:47.988 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:48.031 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:48.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:48.076 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:48.108 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:48.123 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:48.126 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:48.127 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:48.211 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:48.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:48.415 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:48.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:48.549 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[27010]: 3.1c deep-scrub starts 2024-04-25T05:52:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[27010]: 3.1c deep-scrub ok 2024-04-25T05:52:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3242596041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:48.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-25T05:52:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[37492]: 3.1c deep-scrub starts 2024-04-25T05:52:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[37492]: 3.1c deep-scrub ok 2024-04-25T05:52:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3242596041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:48.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:48 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-25T05:52:48.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:48 smithi060 ceph-mon[30898]: 3.1c deep-scrub starts 2024-04-25T05:52:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:48 smithi060 ceph-mon[30898]: 3.1c deep-scrub ok 2024-04-25T05:52:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3242596041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:48 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-25T05:52:48.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:48.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:48.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:48.960 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:48.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:49.062 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.103 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:49.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:49.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:49.218 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:49.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:49.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:49.394 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:49.495 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:49.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:49.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:49.674 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 7, 6, 2, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T05:52:49.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:52:49.684 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: pgmap v1017: 160 pgs: 2 active+remapped+backfilling, 1 active+clean+remapped, 157 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 7703/232734 objects misplaced (3.310%); 28 MiB/s, 443 objects/s recovering 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: osdmap e545: 8 total, 8 up, 8 in 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:49.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:49.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:49.814 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: pgmap v1017: 160 pgs: 2 active+remapped+backfilling, 1 active+clean+remapped, 157 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 7703/232734 objects misplaced (3.310%); 28 MiB/s, 443 objects/s recovering 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: osdmap e545: 8 total, 8 up, 8 in 2024-04-25T05:52:49.830 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: pgmap v1017: 160 pgs: 2 active+remapped+backfilling, 1 active+clean+remapped, 157 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 7703/232734 objects misplaced (3.310%); 28 MiB/s, 443 objects/s recovering 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: osdmap e545: 8 total, 8 up, 8 in 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:52:49.831 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:52:49.916 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:49.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:49.973 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:49.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232177 10.18 10:1a382fa4:::benchmark_data_smithi043_152558_object103827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.70738438699999, 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95601884800000003, 2024-04-25T05:52:50.066 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232177 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355127+0000", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4000000000000001e-07 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355130+0000", 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.811e-06 2024-04-25T05:52:50.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355139+0000", 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.687e-06 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355146+0000", 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9190000000000002e-06 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.069 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355285+0000", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013905100000000001 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355296+0000", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0637e-05 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307193+0000", 2024-04-25T05:52:50.070 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95189739500000004 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307333+0000", 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013948199999999999 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307609+0000", 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027630100000000001 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.071 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.310940+0000", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003331493 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311007+0000", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6475000000000003e-05 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311024+0000", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7076999999999999e-05 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.072 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311094+0000", 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9638e-05 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311145+0000", 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1242000000000002e-05 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232184 10.18 10:1f984730:::benchmark_data_smithi043_152558_object103834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:52:50.073 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.699193334, 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94951154000000004, 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232184 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363318+0000", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.181e-06 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363324+0000", 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6590000000000004e-06 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363333+0000", 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7019999999999996e-06 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363341+0000", 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4930000000000001e-06 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363402+0000", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1212999999999998e-05 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363416+0000", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4008e-05 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309166+0000", 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94575008000000005 2024-04-25T05:52:50.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309307+0000", 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014065700000000001 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309544+0000", 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023717000000000001 2024-04-25T05:52:50.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312676+0000", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031320359999999999 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312690+0000", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3619e-05 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312725+0000", 2024-04-25T05:52:50.079 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5490000000000001e-05 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312787+0000", 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2051999999999995e-05 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:50.081 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.081 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312828+0000", 2024-04-25T05:52:50.081 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1180000000000002e-05 2024-04-25T05:52:50.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232190 10.18 10:1c12a50d:::benchmark_data_smithi043_152558_object103840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.69264703600001, 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94767507500000003, 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.082 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232190 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.083 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369865+0000", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4619999999999999e-06 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369871+0000", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3749999999999999e-06 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369882+0000", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0515e-05 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.084 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369889+0000", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.481e-06 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369947+0000", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7445999999999999e-05 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369956+0000", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3000000000000007e-06 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.085 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311176+0000", 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.941220371 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311289+0000", 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112652 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311488+0000", 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019917399999999999 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.086 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315758+0000", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042704539999999999 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317049+0000", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012905200000000001 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317411+0000", 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036197200000000001 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.087 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317495+0000", 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4339999999999995e-05 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317538+0000", 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3013000000000003e-05 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.088 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232212 10.18 10:1d7e6992:::benchmark_data_smithi043_152558_object103862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.66966313399999, 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93768464399999996, 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232212 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.089 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392848+0000", 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.265e-06 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.090 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392862+0000", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4076999999999999e-05 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392874+0000", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1736000000000001e-05 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392883+0000", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5909999999999996e-06 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.091 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392949+0000", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5753999999999994e-05 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392958+0000", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7180000000000008e-06 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312940+0000", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.919981771 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.092 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.313071+0000", 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013091099999999999 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.313274+0000", 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020262899999999999 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317004+0000", 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037301209999999999 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.093 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317372+0000", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036868600000000001 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330370+0000", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012997140000000001 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330481+0000", 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111634 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330532+0000", 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0611e-05 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232280 10.18 10:1f825152:::benchmark_data_smithi043_152558_object103930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.60092012000001, 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.87212884099999999, 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.095 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232280 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461591+0000", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.24e-06 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461597+0000", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4500000000000003e-06 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461607+0000", 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9990000000000003e-06 2024-04-25T05:52:50.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461615+0000", 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8429999999999993e-06 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461674+0000", 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9651999999999997e-05 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.098 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461683+0000", 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9549999999999998e-06 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314427+0000", 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.85274352399999997 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314544+0000", 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117234 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.099 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314753+0000", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020859700000000001 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317062+0000", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002308994 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317596+0000", 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00053388299999999997 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.100 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333567+0000", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015971346000000001 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333670+0000", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000102645 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333719+0000", 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9478999999999998e-05 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232299 10.18 10:1cc40e50:::benchmark_data_smithi043_152558_object103949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.58534736799999, 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.856721332, 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232299 2024-04-25T05:52:50.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477164+0000", 2024-04-25T05:52:50.103 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4190000000000001e-06 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477171+0000", 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1929999999999998e-06 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477184+0000", 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3501e-05 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.104 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477193+0000", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7329999999999992e-06 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477262+0000", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9710999999999997e-05 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477269+0000", 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2580000000000001e-06 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.105 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315793+0000", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.83852447299999999 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315900+0000", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001072 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.316075+0000", 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017479799999999999 2024-04-25T05:52:50.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320733+0000", 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046573889999999996 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323425+0000", 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026927019999999999 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.107 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333779+0000", 2024-04-25T05:52:50.108 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010353968 2024-04-25T05:52:50.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333832+0000", 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2717999999999999e-05 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333884+0000", 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2268999999999997e-05 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232324 10.18 10:1aeaad8b:::benchmark_data_smithi043_152558_object103974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.56284720299999, 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.83436993100000001, 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232324 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.110 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499665+0000", 2024-04-25T05:52:50.111 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5820000000000001e-06 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499670+0000", 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7880000000000003e-06 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499682+0000", 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1754e-05 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.112 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499697+0000", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4946999999999999e-05 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499757+0000", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9710999999999998e-05 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499766+0000", 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8470000000000002e-06 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317631+0000", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.81786536399999998 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.318088+0000", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000456828 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.318348+0000", 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000260294 2024-04-25T05:52:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320775+0000", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024265430000000002 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324784+0000", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040088809999999997 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333952+0000", 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0091684090000000006 2024-04-25T05:52:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333997+0000", 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5451999999999998e-05 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334033+0000", 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5531000000000003e-05 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232342 10.18 10:1cb472a9:::benchmark_data_smithi043_152558_object103992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.548573871, 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.82022194299999995, 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232342 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513937+0000", 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9400000000000004e-07 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513942+0000", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8919999999999999e-06 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513951+0000", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6090000000000001e-06 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513958+0000", 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5399999999999998e-06 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.515008+0000", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001050193 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.515016+0000", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0369999999999995e-06 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319428+0000", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.80441196400000003 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319549+0000", 2024-04-25T05:52:50.120 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012084200000000001 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319732+0000", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000182597 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324789+0000", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050572589999999997 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324835+0000", 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5748000000000003e-05 2024-04-25T05:52:50.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334075+0000", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0092403490000000001 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334122+0000", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6983000000000001e-05 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334158+0000", 2024-04-25T05:52:50.122 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6136000000000001e-05 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232354 10.18 10:1d2e927d:::benchmark_data_smithi043_152558_object104004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.53362535400001, 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.80545024600000004, 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232354 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.124 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528887+0000", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.792e-06 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528892+0000", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.271e-06 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528906+0000", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3696999999999999e-05 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528912+0000", 2024-04-25T05:52:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9519999999999999e-06 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528967+0000", 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5707e-05 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528977+0000", 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.009e-05 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320806+0000", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.79182896999999997 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320924+0000", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011754100000000001 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.321095+0000", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000171349 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324794+0000", 2024-04-25T05:52:50.127 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036986520000000002 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327302+0000", 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025076109999999999 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334244+0000", 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0069425119999999996 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.128 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334295+0000", 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0688e-05 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334335+0000", 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0414000000000003e-05 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232355 10.18 10:1899febd:::benchmark_data_smithi043_152558_object104005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.53358151699999, 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.805527047, 2024-04-25T05:52:50.129 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232355 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.06e-07 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528932+0000", 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9289999999999998e-06 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528939+0000", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4899999999999997e-06 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528945+0000", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7349999999999999e-06 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.529003+0000", 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7460999999999998e-05 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.132 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.529010+0000", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6320000000000002e-06 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322040+0000", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.79303036000000005 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322147+0000", 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106777 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.133 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322304+0000", 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000157009 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327251+0000", 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049475370000000001 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.135 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327352+0000", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010056999999999999 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334372+0000", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070206349999999999 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334420+0000", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7917999999999998e-05 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.136 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334456+0000", 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5488000000000003e-05 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232443 10.18 10:1a37d6af:::benchmark_data_smithi043_152558_object104093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.440345057, 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.712427268, 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.137 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232443 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.138 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622166+0000", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1209999999999999e-06 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622171+0000", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1220000000000003e-06 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.139 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622190+0000", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8165e-05 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622197+0000", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7640000000000005e-06 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622280+0000", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2244999999999996e-05 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622289+0000", 2024-04-25T05:52:50.140 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0820000000000005e-06 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323472+0000", 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.70118327999999996 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323581+0000", 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109454 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.141 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323751+0000", 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000169116 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327257+0000", 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003506412 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330319+0000", 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030615999999999998 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.142 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334494+0000", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041753980000000003 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334539+0000", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4817999999999997e-05 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334593+0000", 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3690999999999998e-05 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232545 10.18 10:1949a72c:::benchmark_data_smithi043_152558_object104195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.29801461400001, 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.57027235399999998, 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232545 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.144 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764497+0000", 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.176e-06 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.145 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764502+0000", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8420000000000004e-06 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764512+0000", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0516999999999999e-05 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764518+0000", 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8989999999999998e-06 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.146 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764575+0000", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6471999999999997e-05 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764586+0000", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1054e-05 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324873+0000", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.56028686800000005 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.147 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324974+0000", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101086 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.325137+0000", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016339300000000001 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327314+0000", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021773650000000001 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330579+0000", 2024-04-25T05:52:50.148 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003264922 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334633+0000", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040539970000000002 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334729+0000", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5847000000000006e-05 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334768+0000", 2024-04-25T05:52:50.149 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8915999999999998e-05 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232637 10.18 10:1e18b6bd:::benchmark_data_smithi043_152558_object104287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.19538379799999, 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.46779147700000001, 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.150 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232637 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867128+0000", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.46e-06 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867134+0000", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0689999999999998e-06 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867149+0000", 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4642e-05 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867158+0000", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1460000000000002e-06 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867240+0000", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1972999999999996e-05 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867249+0000", 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5070000000000006e-06 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.325969+0000", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.45871979299999999 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.326081+0000", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112388 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.326255+0000", 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017321900000000001 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330277+0000", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040226740000000004 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333354+0000", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030765520000000002 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334830+0000", 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014759689999999999 2024-04-25T05:52:50.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334881+0000", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1400999999999998e-05 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334918+0000", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6684000000000003e-05 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232658 10.18 10:1ec2bb35:::benchmark_data_smithi043_152558_object104308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:52:50.156 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.171591745, 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.44412062099999999, 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232658 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890920+0000", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.294e-06 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890926+0000", 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3069999999999999e-06 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.158 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890939+0000", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2842e-05 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890948+0000", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8300000000000002e-06 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.891010+0000", 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2336999999999996e-05 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.891022+0000", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1616e-05 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327388+0000", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.43636574500000003 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327501+0000", 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011370900000000001 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.160 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.161 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327676+0000", 2024-04-25T05:52:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.161 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:50.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:50.162 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017499599999999999 2024-04-25T05:52:50.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.162 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.162 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330331+0000", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026544569999999998 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333412+0000", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030809420000000001 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334958+0000", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001546635 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335003+0000", 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4320000000000003e-05 2024-04-25T05:52:50.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335039+0000", 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6591000000000003e-05 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232694 10.18 10:19db009e:::benchmark_data_smithi043_152558_object104344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.10884363100001, 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.38153700099999999, 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.164 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232694 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953668+0000", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2610000000000001e-06 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953673+0000", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6829999999999999e-06 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953683+0000", 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0689e-05 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953692+0000", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7800000000000006e-06 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953744+0000", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1499000000000003e-05 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953751+0000", 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6709999999999999e-06 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.167 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.328723+0000", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.37497164700000002 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.328841+0000", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117623 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.329030+0000", 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000189676 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.168 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333431+0000", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0044009139999999997 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333492+0000", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1301000000000001e-05 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335119+0000", 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001626901 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.169 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335167+0000", 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7777e-05 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335204+0000", 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6579000000000003e-05 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232721 10.18 10:1e695aa9:::benchmark_data_smithi043_152558_object104371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:52:50.170 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 188.03116815000001, 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.31091695600000002, 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232721 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.171 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031344+0000", 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.609e-06 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.172 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031350+0000", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3180000000000003e-06 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031366+0000", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6240999999999999e-05 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031376+0000", 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4760000000000004e-06 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.173 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031440+0000", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3822e-05 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031447+0000", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6490000000000008e-06 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331065+0000", 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.29961802999999998 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.174 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331187+0000", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012126299999999999 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331525+0000", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033875700000000002 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335248+0000", 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003722235 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.175 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.337203+0000", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019557929999999999 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342156+0000", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049530540000000001 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342213+0000", 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6521000000000003e-05 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342259+0000", 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6187999999999999e-05 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:127712 11.d 11:b9029a75:::benchmark_data_smithi043_179003_object127711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:47.567613+0000", 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 62.494897084999998, 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.092190297000000004, 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.177 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 127712 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567613+0000", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.178 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567613+0000", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567614+0000", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.92e-07 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567636+0000", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2079999999999999e-05 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.179 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567644+0000", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8550000000000002e-06 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.567651+0000", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6050000000000003e-06 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.646537+0000", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.078886267999999996 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.180 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.646679+0000", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000142452 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.646818+0000", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000139159 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.653068+0000", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062499840000000001 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.181 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.653122+0000", 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3102000000000001e-05 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.659693+0000", 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0065719109999999997 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.659755+0000", 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1301000000000001e-05 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:47.659804+0000", 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8788000000000001e-05 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:168496 11.2 11:474b32a1:::benchmark_data_smithi043_179003_object38207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:52:27.863521+0000", 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 22.198989502, 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.089029188999999995, 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.183 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 168496 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863521+0000", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.184 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863521+0000", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863522+0000", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3009999999999999e-06 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863527+0000", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4920000000000004e-06 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.185 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863538+0000", 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1810999999999999e-05 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863548+0000", 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2059999999999996e-06 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.863975+0000", 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00042736100000000002 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.186 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.864112+0000", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013671000000000001 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,7", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.864634+0000", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00052273800000000002 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.877960+0000", 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.013326044 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.187 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.188 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.877976+0000", 2024-04-25T05:52:50.188 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5546999999999999e-05 2024-04-25T05:52:50.188 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:50.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.878589+0000", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000612694 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.951984+0000", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.073394986999999995 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.952328+0000", 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00034472499999999999 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:27.952550+0000", 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000221573 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:169381 11.2 11:494234f6:::benchmark_data_smithi043_179003_object39092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:52:28.784426+0000", 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 21.278084408000002, 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.093397445999999995, 2024-04-25T05:52:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 169381 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784426+0000", 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784426+0000", 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784427+0000", 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.291e-06 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.192 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784432+0000", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2939999999999996e-06 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784447+0000", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4694e-05 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.784456+0000", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1519999999999998e-06 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.193 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.789845+0000", 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053891329999999999 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.789982+0000", 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013606100000000001 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3,7", 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.790114+0000", 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000132805 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.798322+0000", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0082077810000000008 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.798419+0000", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7402e-05 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.805069+0000", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0066490610000000004 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:52:50.195 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.877483+0000", 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.072413951000000004 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.877694+0000", 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000211638 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:28.877823+0000", 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129183 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "MOSDPGPushReply(11.2 532/498 [PushReplyOp(11:4dbf1020:::benchmark_data_smithi043_179003_object119999:head)])", 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:52:35.819586+0000", 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 14.242924109, 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.11081748700000001, 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819586+0000", 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.197 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819586+0000", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819587+0000", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2500000000000001e-07 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819590+0000", 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1250000000000001e-06 2024-04-25T05:52:50.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819597+0000", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0260000000000002e-06 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.819604+0000", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5129999999999999e-06 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.930108+0000", 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.110503724 2024-04-25T05:52:50.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:52:35.930404+0000", 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000295774 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:52:50.200 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:52:50.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:50.292 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:50.394 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.425 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:50.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:50.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:52:50.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:50.551 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:50.653 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:50.679 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:50.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: 11.0 scrub starts 2024-04-25T05:52:50.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: osdmap e546: 8 total, 8 up, 8 in 2024-04-25T05:52:50.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:50.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:50.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: 5.9 deep-scrub starts 2024-04-25T05:52:50.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:50 smithi060 ceph-mon[30898]: 5.9 deep-scrub ok 2024-04-25T05:52:50.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: 11.0 scrub starts 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: osdmap e546: 8 total, 8 up, 8 in 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: 5.9 deep-scrub starts 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[27010]: 5.9 deep-scrub ok 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: 11.0 scrub starts 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: osdmap e546: 8 total, 8 up, 8 in 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:50.757 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:50.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: 5.9 deep-scrub starts 2024-04-25T05:52:50.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:50 smithi043 ceph-mon[37492]: 5.9 deep-scrub ok 2024-04-25T05:52:50.781 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:50.836 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:50.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:50.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:50.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:51.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.144 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:51.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:51.230 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:52:51.245 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:51.332 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:51.434 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:51.508 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:51.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:51.633 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:51.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:51.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: pgmap v1020: 160 pgs: 1 active+clean+scrubbing, 2 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 5799/221850 objects misplaced (2.614%); 24 MiB/s, 376 objects/s recovering 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 11.2 scrub starts 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: osdmap e547: 8 total, 8 up, 8 in 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:52:51.765 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[27010]: 6.5 deep-scrub starts 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: pgmap v1020: 160 pgs: 1 active+clean+scrubbing, 2 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 5799/221850 objects misplaced (2.614%); 24 MiB/s, 376 objects/s recovering 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 11.2 scrub starts 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: osdmap e547: 8 total, 8 up, 8 in 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:52:51.766 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:51 smithi043 ceph-mon[37492]: 6.5 deep-scrub starts 2024-04-25T05:52:51.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:51.889 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:51.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:51.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:51.946 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:52:51.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: pgmap v1020: 160 pgs: 1 active+clean+scrubbing, 2 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 5799/221850 objects misplaced (2.614%); 24 MiB/s, 376 objects/s recovering 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 11.2 scrub starts 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: osdmap e547: 8 total, 8 up, 8 in 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:52:51.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:51 smithi060 ceph-mon[30898]: 6.5 deep-scrub starts 2024-04-25T05:52:52.008 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:52.070 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:52.172 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:52.199 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:52.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:52.325 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:52.388 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:52.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:52.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:52.529 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:52:52.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.637 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:52.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:52.738 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: 11.c scrub starts 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: 11.0 scrub ok 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: osdmap e548: 8 total, 8 up, 8 in 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: 6.5 deep-scrub ok 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:52:52.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:52:52.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: 11.c scrub starts 2024-04-25T05:52:52.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: 11.0 scrub ok 2024-04-25T05:52:52.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: osdmap e548: 8 total, 8 up, 8 in 2024-04-25T05:52:52.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: 6.5 deep-scrub ok 2024-04-25T05:52:52.764 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:52:52.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:52.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:52:52.765 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:52 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:52:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.776 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:52.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:52.877 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:52.922 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:52.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: 11.c scrub starts 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: 11.0 scrub ok 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: osdmap e548: 8 total, 8 up, 8 in 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: 6.5 deep-scrub ok 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:52:52.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:52 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:52:53.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.059 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:53.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:53.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:53.201 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:53.302 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:53.329 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:53.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:53.458 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:53.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:53.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:53.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:53.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:53.782 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:52:53.803 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: pgmap v1023: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 5799/221847 objects misplaced (2.614%); 23 MiB/s, 374 objects/s recovering 2024-04-25T05:52:53.803 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 11.c scrub ok 2024-04-25T05:52:53.803 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:52:53.803 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:52:53.803 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: osdmap e549: 8 total, 8 up, 8 in 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 11.2 scrub ok 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.7 scrub starts 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.7 scrub ok 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3793860876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:52:53.804 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:52:53.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:53.883 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:53.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: pgmap v1023: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 5799/221847 objects misplaced (2.614%); 23 MiB/s, 374 objects/s recovering 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 11.c scrub ok 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:52:53.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: osdmap e549: 8 total, 8 up, 8 in 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 11.2 scrub ok 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.7 scrub starts 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.7 scrub ok 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3793860876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: pgmap v1023: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 5799/221847 objects misplaced (2.614%); 23 MiB/s, 374 objects/s recovering 2024-04-25T05:52:53.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 11.c scrub ok 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/548261066' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: osdmap e549: 8 total, 8 up, 8 in 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 11.2 scrub ok 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.7 scrub starts 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.7 scrub ok 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3793860876' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:52:53.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:53 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:52:53.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:54.011 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:54.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:54.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:54.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:52:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:52:54.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:54.290 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:54.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:54.433 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:54.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.583 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:54.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:54.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:54.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:54.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:54.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:54.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:52:54.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[27010]: osdmap e550: 8 total, 8 up, 8 in 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:54.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:54 smithi043 ceph-mon[37492]: osdmap e550: 8 total, 8 up, 8 in 2024-04-25T05:52:54.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:54.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:54 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:52:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:54 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:52:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:54 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:52:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:54 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:52:54.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:54 smithi060 ceph-mon[30898]: osdmap e550: 8 total, 8 up, 8 in 2024-04-25T05:52:55.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:55.112 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:55.214 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:55.250 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:55.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:55.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:55.522 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:55.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:55.650 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:55.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:55.779 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:55.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[27010]: pgmap v1026: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 5799/213531 objects misplaced (2.716%) 2024-04-25T05:52:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[27010]: osdmap e551: 8 total, 8 up, 8 in 2024-04-25T05:52:55.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[27010]: osdmap e552: 8 total, 8 up, 8 in 2024-04-25T05:52:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[37492]: pgmap v1026: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 5799/213531 objects misplaced (2.716%) 2024-04-25T05:52:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[37492]: osdmap e551: 8 total, 8 up, 8 in 2024-04-25T05:52:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:55 smithi043 ceph-mon[37492]: osdmap e552: 8 total, 8 up, 8 in 2024-04-25T05:52:55.918 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:55.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:52:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:55 smithi060 ceph-mon[30898]: pgmap v1026: 159 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 693 op/s; 5799/213531 objects misplaced (2.716%) 2024-04-25T05:52:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:55 smithi060 ceph-mon[30898]: osdmap e551: 8 total, 8 up, 8 in 2024-04-25T05:52:55.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:55 smithi060 ceph-mon[30898]: osdmap e552: 8 total, 8 up, 8 in 2024-04-25T05:52:56.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:56.047 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:56.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:52:56.178 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:52:56.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:56.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:56.446 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:56.547 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:52:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:52:56.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:56.724 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:56.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:56.861 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:56.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:56.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:52:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:57.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:52:57.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:57.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:57.262 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:57.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:57.409 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:57.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:57.540 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:57.643 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:57.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:57.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:57.804 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:57.815 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:57 smithi060 ceph-mon[30898]: pgmap v1029: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 4626/208629 objects misplaced (2.217%); 14 MiB/s, 213 objects/s recovering 2024-04-25T05:52:57.873 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:57 smithi043 ceph-mon[27010]: pgmap v1029: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 4626/208629 objects misplaced (2.217%); 14 MiB/s, 213 objects/s recovering 2024-04-25T05:52:57.874 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:57 smithi043 ceph-mon[37492]: pgmap v1029: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 4626/208629 objects misplaced (2.217%); 14 MiB/s, 213 objects/s recovering 2024-04-25T05:52:57.906 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:57.942 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:57.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:58.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:58.094 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:58.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:58.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:58.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:58.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:58.463 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:52:58.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:58.499 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:58.505 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:52:58.544 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:52:58.588 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:52:58.601 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:58.626 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:58.638 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:52:58.684 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:52:58.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:58.729 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3565294852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:58.729 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3565294852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:58.729 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:52:58.752 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:58.767 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:52:58.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:52:58.777 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:52:58.778 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:52:58.853 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:52:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:58.884 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:58.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:52:58.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:58.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3565294852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:52:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.016 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:59.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:59.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.158 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:59.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:59.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:59.293 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:59.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:52:59.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:52:59.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:59.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:52:59.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:52:59.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:52:59.719 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:52:59.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:52:59.849 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:52:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:52:59.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:52:59 smithi043 ceph-mon[27010]: pgmap v1030: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 904 op/s; 4626/206712 objects misplaced (2.238%); 9.7 MiB/s, 152 objects/s recovering 2024-04-25T05:52:59.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:52:59 smithi043 ceph-mon[37492]: pgmap v1030: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 904 op/s; 4626/206712 objects misplaced (2.238%); 9.7 MiB/s, 152 objects/s recovering 2024-04-25T05:52:59.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:52:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:52:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:52:59.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:52:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:52:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:52:59.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:52:59.998 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:52:59 smithi060 ceph-mon[30898]: pgmap v1030: 158 pgs: 1 clean+premerge+peered, 2 active+remapped+backfilling, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 904 op/s; 4626/206712 objects misplaced (2.238%); 9.7 MiB/s, 152 objects/s recovering 2024-04-25T05:53:00.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:00.134 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:00.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:00.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:00.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:00.515 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:00.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.671 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:00.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:00.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:00.810 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:00.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:00 smithi043 ceph-mon[27010]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.14k op/s; 3401/193620 objects misplaced (1.757%); 17 MiB/s, 274 objects/s recovering 2024-04-25T05:53:00.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:00 smithi043 ceph-mon[37492]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.14k op/s; 3401/193620 objects misplaced (1.757%); 17 MiB/s, 274 objects/s recovering 2024-04-25T05:53:00.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:00.943 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:00.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:00.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:00 smithi060 ceph-mon[30898]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 779 B/s rd, 0 B/s wr, 1.14k op/s; 3401/193620 objects misplaced (1.757%); 17 MiB/s, 274 objects/s recovering 2024-04-25T05:53:01.046 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:01.085 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:01.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:01.216 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:01.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:01.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:01.445 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:01.487 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:01.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:01.625 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:01.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:01.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:01.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:01.896 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:01.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:01.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:02.156 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:02.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:02.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:02.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:02.432 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:02.534 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:02.555 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:02.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:02.688 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:02.790 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:02.813 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:02.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:02.939 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:02.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:03.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.070 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:03.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:03.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:03.195 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:03.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:03.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:03.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:03.463 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:03.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:03 smithi060 ceph-mon[30898]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 828 op/s; 3401/193620 objects misplaced (1.757%); 14 MiB/s, 224 objects/s recovering 2024-04-25T05:53:03.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:03.605 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:03 smithi043 ceph-mon[27010]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 828 op/s; 3401/193620 objects misplaced (1.757%); 14 MiB/s, 224 objects/s recovering 2024-04-25T05:53:03.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:03 smithi043 ceph-mon[37492]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 828 op/s; 3401/193620 objects misplaced (1.757%); 14 MiB/s, 224 objects/s recovering 2024-04-25T05:53:03.707 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:03.742 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:03.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:03.882 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:03.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:03.984 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:04.010 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:04.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:04.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:04.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:04.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:04.287 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:04.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.421 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:04.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:04.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:04.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:04.656 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:04.682 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:04.784 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:04.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:04.846 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:53:04.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[27010]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 864 op/s; 3401/183924 objects misplaced (1.849%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:04.920 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:04 smithi043 ceph-mon[37492]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 864 op/s; 3401/183924 objects misplaced (1.849%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:53:04.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:04.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:04.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:04 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:04.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:04 smithi060 ceph-mon[30898]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 864 op/s; 3401/183924 objects misplaced (1.849%); 6.2 MiB/s, 99 objects/s recovering 2024-04-25T05:53:05.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:05.091 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:05.101 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:05.189 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:05.189 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:05.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.249 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:05.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:05.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:05.388 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:05.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:05.518 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:05.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:05.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:05.746 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:05.772 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:05.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:05.904 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:05.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:06.006 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:06.031 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:06.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.163 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:06.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:06.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:06.290 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:06.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:06.418 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:06.519 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:06.562 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:06.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:06.690 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:06.792 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:06.827 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:06.929 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:06.964 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:06.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:07.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:07.091 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:07.182 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:53:07.194 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:07.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:07.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:07.357 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:07.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:07 smithi043 ceph-mon[37492]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 923 op/s; 2192/178161 objects misplaced (1.230%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:07.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:53:07.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:07 smithi043 ceph-mon[27010]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 923 op/s; 2192/178161 objects misplaced (1.230%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:07.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:53:07.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:07.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:07.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:07 smithi060 ceph-mon[30898]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 930 B/s rd, 0 B/s wr, 923 op/s; 2192/178161 objects misplaced (1.230%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:53:07.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.631 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:07.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:07.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:07.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:07.889 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:07.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:07.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:08.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:53:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:08.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:08.170 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:08.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:08.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:53:08.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:08 smithi043 ceph-mon[27010]: osdmap e553: 8 total, 8 up, 8 in 2024-04-25T05:53:08.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:53:08.297 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:08 smithi043 ceph-mon[37492]: osdmap e553: 8 total, 8 up, 8 in 2024-04-25T05:53:08.314 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:08.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:08.432 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:08.461 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3689310983' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:53:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:08 smithi060 ceph-mon[30898]: osdmap e553: 8 total, 8 up, 8 in 2024-04-25T05:53:08.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.592 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:08.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:08.693 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:08.720 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:08.822 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:08.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:08.954 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:53:08.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:09.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:09.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:09.211 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:53:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:09.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:09.375 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:09.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:09 smithi060 ceph-mon[30898]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 2192/176361 objects misplaced (1.243%); 12 MiB/s, 193 objects/s recovering 2024-04-25T05:53:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:09.511 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:09.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:09 smithi043 ceph-mon[27010]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 2192/176361 objects misplaced (1.243%); 12 MiB/s, 193 objects/s recovering 2024-04-25T05:53:09.563 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:09.563 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:09 smithi043 ceph-mon[37492]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s; 2192/176361 objects misplaced (1.243%); 12 MiB/s, 193 objects/s recovering 2024-04-25T05:53:09.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:09.613 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:09.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:09.781 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:09.806 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:09.846 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:53:09.848 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:09.882 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:09.884 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:09.912 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:09.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:09.925 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:09.969 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:10.014 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:10.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:10.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:10.060 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:10.107 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:10.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:10.186 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:10.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:10.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:10.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:10.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:10.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[37492]: osdmap e554: 8 total, 8 up, 8 in 2024-04-25T05:53:10.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4251856815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[27010]: osdmap e554: 8 total, 8 up, 8 in 2024-04-25T05:53:10.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4251856815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:10.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:10.461 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:10 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4070615196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:10 smithi060 ceph-mon[30898]: osdmap e554: 8 total, 8 up, 8 in 2024-04-25T05:53:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4251856815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:10.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:10.595 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:10.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:10.733 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:10.835 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:10.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:10.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:10.964 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:11.001 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:11.058 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:11.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:11.129 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:11.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:11.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:11.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:11.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:11 smithi043 ceph-mon[27010]: pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:11.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:11 smithi043 ceph-mon[37492]: pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:11.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:11 smithi060 ceph-mon[30898]: pgmap v1038: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:11.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.540 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:11.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:11.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:11.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:11.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:11.819 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:11.921 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:11.950 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:11.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:11.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:12.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:12.095 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:12.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.224 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:12.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:12.256 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:53:12.325 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:12.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:12.452 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.494 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:12.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:12.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.665 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:12.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:12.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:12.792 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:12.861 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:53:12.861 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:53:12.861 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:53:12.861 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:53:12.861 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:53:12.893 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:12.954 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:12.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:13.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:13.067 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:53:13.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:13.195 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:13.221 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:53:13.323 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:13.350 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:13.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:13.482 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:13.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:13.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:13.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[27010]: pgmap v1039: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 868 op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:13.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2992275460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[37492]: pgmap v1039: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 868 op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2992275460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:13 smithi060 ceph-mon[30898]: pgmap v1039: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 868 op/s; 1106/163077 objects misplaced (0.678%); 15 MiB/s, 242 objects/s recovering 2024-04-25T05:53:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2992275460' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:53:13.760 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:13.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:13.887 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:13.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:13.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:14.017 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:14.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:14.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:14.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:14.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.278 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:14.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:14.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:14.410 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:14.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.533 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:14.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:14.634 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:14.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:53:14.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:14 smithi043 ceph-mon[27010]: osdmap e555: 8 total, 8 up, 8 in 2024-04-25T05:53:14.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:53:14.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:14 smithi043 ceph-mon[37492]: osdmap e555: 8 total, 8 up, 8 in 2024-04-25T05:53:14.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:53:14.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:14 smithi060 ceph-mon[30898]: osdmap e555: 8 total, 8 up, 8 in 2024-04-25T05:53:14.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.808 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:14.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:14.909 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:14.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:15.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.088 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:15.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:15.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:15.210 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:15.312 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:15.336 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:15.438 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:15.459 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:15.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:15.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:15.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:15.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[37492]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s; 1106/154518 objects misplaced (0.716%); 8.6 MiB/s, 138 objects/s recovering 2024-04-25T05:53:15.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:15.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:15.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[37492]: osdmap e556: 8 total, 8 up, 8 in 2024-04-25T05:53:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[27010]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s; 1106/154518 objects misplaced (0.716%); 8.6 MiB/s, 138 objects/s recovering 2024-04-25T05:53:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:15.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:15 smithi043 ceph-mon[27010]: osdmap e556: 8 total, 8 up, 8 in 2024-04-25T05:53:15.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:15.705 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:15 smithi060 ceph-mon[30898]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.14k op/s; 1106/154518 objects misplaced (0.716%); 8.6 MiB/s, 138 objects/s recovering 2024-04-25T05:53:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:15 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:15 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:15 smithi060 ceph-mon[30898]: osdmap e556: 8 total, 8 up, 8 in 2024-04-25T05:53:15.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:15.828 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:15.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:15.951 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:15.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:16.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:16.076 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:16.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:16.200 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:16.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:16.333 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:16.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:16.456 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:16.558 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:16.579 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:16.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:16.703 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:16.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:16.828 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:16.930 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:16.953 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:16.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:17.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:17.077 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:17.179 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.216 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:17.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:17.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:17.343 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:17 smithi043 ceph-mon[27010]: osdmap e557: 8 total, 8 up, 8 in 2024-04-25T05:53:17.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:17 smithi043 ceph-mon[27010]: pgmap v1044: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 190/149046 objects misplaced (0.127%); 8.9 MiB/s, 142 objects/s recovering 2024-04-25T05:53:17.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:17 smithi043 ceph-mon[37492]: osdmap e557: 8 total, 8 up, 8 in 2024-04-25T05:53:17.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:17 smithi043 ceph-mon[37492]: pgmap v1044: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 190/149046 objects misplaced (0.127%); 8.9 MiB/s, 142 objects/s recovering 2024-04-25T05:53:17.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:17.470 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:17.485 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:53:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:17 smithi060 ceph-mon[30898]: osdmap e557: 8 total, 8 up, 8 in 2024-04-25T05:53:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:17 smithi060 ceph-mon[30898]: pgmap v1044: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 190/149046 objects misplaced (0.127%); 8.9 MiB/s, 142 objects/s recovering 2024-04-25T05:53:17.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.614 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:17.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:17.714 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:17.756 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:17.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:17.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:17.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:18.037 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:18.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:18.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:18.198 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:18.247 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:53:18.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:18.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:18.359 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:18 smithi043 ceph-mon[27010]: osdmap e558: 8 total, 8 up, 8 in 2024-04-25T05:53:18.359 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:18 smithi043 ceph-mon[37492]: osdmap e558: 8 total, 8 up, 8 in 2024-04-25T05:53:18.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:18.433 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:18 smithi060 ceph-mon[30898]: osdmap e558: 8 total, 8 up, 8 in 2024-04-25T05:53:18.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:18.562 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:18.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:18.692 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:18.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:18.822 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:18.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:18.993 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:18.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:19.078 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:19.094 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:19.119 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:19.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:19 smithi043 ceph-mon[27010]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 883 op/s; 190/147762 objects misplaced (0.129%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:19.121 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:19.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:19 smithi043 ceph-mon[37492]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 883 op/s; 190/147762 objects misplaced (0.129%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:19.121 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:19.221 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:19.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:19.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:19.391 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:19.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:19 smithi060 ceph-mon[30898]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 883 op/s; 190/147762 objects misplaced (0.129%); 11 MiB/s, 175 objects/s recovering 2024-04-25T05:53:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.530 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:19.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:19.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:19.663 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:19.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:19.801 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:19.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:19.929 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:19.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:19.932 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:53:20.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:20.060 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:20.153 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:53:20.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:20.186 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:20.226 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:20 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:20.226 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:20 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:20.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:20.314 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:20.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:20.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:20 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:20.553 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.596 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:20.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:20.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:20.738 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:20.840 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:20.877 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:20.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:20.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:21.007 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:21.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:21.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[27010]: pgmap v1047: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 892 op/s; 11 MiB/s, 173 objects/s recovering 2024-04-25T05:53:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[27010]: osdmap e559: 8 total, 8 up, 8 in 2024-04-25T05:53:21.141 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:53:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[37492]: pgmap v1047: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 892 op/s; 11 MiB/s, 173 objects/s recovering 2024-04-25T05:53:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[37492]: osdmap e559: 8 total, 8 up, 8 in 2024-04-25T05:53:21.142 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:21 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:53:21.151 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:53:21.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:21.265 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:21.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:21 smithi060 ceph-mon[30898]: pgmap v1047: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 892 op/s; 11 MiB/s, 173 objects/s recovering 2024-04-25T05:53:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:21 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:21 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/724084662' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:21 smithi060 ceph-mon[30898]: osdmap e559: 8 total, 8 up, 8 in 2024-04-25T05:53:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:21 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:53:21.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:21.535 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:21.637 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:21.686 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:21.726 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:21.770 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:21.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:21.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:21.815 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:21.857 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:21.899 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:21.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:21.932 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:21.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:21.939 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:21.981 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:21.992 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:21.993 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:22.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.063 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:22.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:22.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:22.197 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:22.299 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:53:22.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: osdmap e560: 8 total, 8 up, 8 in 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3969836706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: osdmap e560: 8 total, 8 up, 8 in 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:53:22.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3969836706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:22.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:22.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:53:22.441 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3687010713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: osdmap e560: 8 total, 8 up, 8 in 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:53:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:53:22.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3969836706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:22.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:22.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:22.631 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:22.732 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:22.745 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:22.745 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.782 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:22.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:22.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:22.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:23.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.053 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:23.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:23.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:23.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:23.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:23.319 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:53:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[27010]: pgmap v1050: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 588 op/s; 1.9 MiB/s, 31 objects/s recovering 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:53:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:23 smithi043 ceph-mon[37492]: pgmap v1050: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 588 op/s; 1.9 MiB/s, 31 objects/s recovering 2024-04-25T05:53:23.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:23.457 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:53:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-mon[30898]: pgmap v1050: 158 pgs: 158 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 588 op/s; 1.9 MiB/s, 31 objects/s recovering 2024-04-25T05:53:23.559 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.619 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:23.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:23.720 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:23.764 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:23.866 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:23.897 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:23.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:23.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:53:23.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:24.029 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:24.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:24.141 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:24.155 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:24.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:24.285 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:24.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:53:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:53:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:53:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:53:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:53:24.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:24 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:53:24.413 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:53:24.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:24 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:53:24.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:53:24.516 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:24.559 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:24.580 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:24.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.718 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:24.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:24.868 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:24.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:24.896 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:24.970 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:25.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:25.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:25.133 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:25.176 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:25.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:25.272 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:25.327 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: pgmap v1051: 158 pgs: 158 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 988 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: pgmap v1051: 158 pgs: 158 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 988 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:53:25.328 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:25 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:53:25.343 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:25.343 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:53:25.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:53:25.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: pgmap v1051: 158 pgs: 158 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 988 op/s; 1.7 MiB/s, 26 objects/s recovering 2024-04-25T05:53:25.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:53:25.344 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:25 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:53:25.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:25.405 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:25.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:25.539 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:25.542 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:25.542 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:25.604 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:53:25.641 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:25.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:25.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:25.871 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:25.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:25.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:26.004 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:26.071 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:26.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:26.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:26.174 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:53:26.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:26.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:26.362 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:53:26.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.436 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:26.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1655296304' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:26 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:26.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1655296304' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1655296304' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:26 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:53:26.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:26.720 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:26.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.849 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:26.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:26.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:26.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:27.083 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:27.110 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:27.194 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:53:27.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:27.250 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:27.318 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:27.318 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:27.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:27.387 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:27.389 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:53:27.389 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:53:27.389 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: pgmap v1052: 158 pgs: 158 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1.3 MiB/s, 22 objects/s recovering 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: osdmap e561: 8 total, 8 up, 8 in 2024-04-25T05:53:27.390 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: pgmap v1052: 158 pgs: 158 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1.3 MiB/s, 22 objects/s recovering 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: osdmap e561: 8 total, 8 up, 8 in 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:53:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: pgmap v1052: 158 pgs: 158 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 1.3 MiB/s, 22 objects/s recovering 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: osdmap e561: 8 total, 8 up, 8 in 2024-04-25T05:53:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:27.489 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:27.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:27.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:27.624 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:27.663 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:27.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.796 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:27.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:27.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:27.924 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:27.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:27.928 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:27.983 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:28.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:28.026 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:28.062 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:28.068 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:28.114 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:28.160 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:28.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:28.198 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:28.206 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:28.243 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:28.251 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:28.252 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:28.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:28.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:53:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:53:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:53:28.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: osdmap e562: 8 total, 8 up, 8 in 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3457330987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: osdmap e562: 8 total, 8 up, 8 in 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3457330987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:53:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:28 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:53:28.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1122235094' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: osdmap e562: 8 total, 8 up, 8 in 2024-04-25T05:53:28.432 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3457330987' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:28.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:53:28.433 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:28 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:53:28.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:28.483 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:28.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:28.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:28.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:53:28.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:28.786 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:28.821 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:28.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:28.923 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:28.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:29.025 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:29.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.211 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:29.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:29.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:29.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:29.453 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:29.486 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:29 smithi060 ceph-mon[30898]: pgmap v1055: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 901 op/s 2024-04-25T05:53:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:29 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:53:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:29 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195454 10.1 10:8ac0cd3c:::benchmark_data_smithi043_152558_object67104:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.95157535200002, 2024-04-25T05:53:29.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044723351, 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195454 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552782+0000", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7039999999999999e-06 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552789+0000", 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8340000000000001e-06 2024-04-25T05:53:29.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552799+0000", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0064999999999999e-05 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552808+0000", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3190000000000001e-06 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558556+0000", 2024-04-25T05:53:29.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057473109999999997 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558565+0000", 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3989999999999993e-06 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591223+0000", 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0326579060000001 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.597504+0000", 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062808129999999997 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195456 10.1 10:80db890b:::benchmark_data_smithi043_152558_object67106:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94967402399999, 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0479446400000001, 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.512 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195456 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554683+0000", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554687+0000", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3379999999999998e-06 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554696+0000", 2024-04-25T05:53:29.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0720000000000006e-06 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554703+0000", 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3239999999999997e-06 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558586+0000", 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038829329999999999 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558699+0000", 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011292 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602367+0000", 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.04366767 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602626+0000", 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025938399999999998 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195470 10.4 10:2594b5ae:::benchmark_data_smithi043_152558_object67120:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94964225500001, 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0253313630000001, 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195470 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7500000000000001e-07 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554716+0000", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3029999999999998e-06 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554721+0000", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.181e-06 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554726+0000", 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9300000000000002e-06 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555243+0000", 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051627300000000004 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555251+0000", 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-04-25T05:53:29.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.578884+0000", 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0236325479999999 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.580045+0000", 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001161048 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195477 10.e 10:7c0c42f7:::benchmark_data_smithi043_152558_object67127:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94961789000001, 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0658362960000001, 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195477 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3299999999999998e-07 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554740+0000", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2330000000000001e-06 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554746+0000", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.13e-06 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554751+0000", 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3750000000000002e-06 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554809+0000", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8230000000000003e-05 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554820+0000", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0581000000000001e-05 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596186+0000", 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0413658189999999 2024-04-25T05:53:29.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.598902+0000", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027164310000000001 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.599221+0000", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000319031 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.611133+0000", 2024-04-25T05:53:29.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011912219999999999 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.614119+0000", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029860350000000002 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620485+0000", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063658589999999998 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620537+0000", 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1572e-05 2024-04-25T05:53:29.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620574+0000", 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7447000000000002e-05 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195487 10.4 10:295dbb8c:::benchmark_data_smithi043_152558_object67137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94959672099998, 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0346735600000001, 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195487 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.529 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8599999999999999e-07 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554761+0000", 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0310000000000001e-06 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554766+0000", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9450000000000001e-06 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554771+0000", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0819999999999998e-06 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555283+0000", 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051210299999999997 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555291+0000", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1690000000000003e-06 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.581742+0000", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0264516699999999 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582176+0000", 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043403700000000002 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582318+0000", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000142003 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589262+0000", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069436080000000004 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589327+0000", 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4770000000000005e-05 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589367+0000", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9824999999999999e-05 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589405+0000", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8486999999999999e-05 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589433+0000", 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7543999999999999e-05 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.534 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195490 10.4 10:28cf0ec2:::benchmark_data_smithi043_152558_object67140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94606759499999, 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0313008349999999, 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195490 2024-04-25T05:53:29.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558289+0000", 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1960000000000001e-06 2024-04-25T05:53:29.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558293+0000", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8619999999999996e-06 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558301+0000", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.396e-06 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558308+0000", 2024-04-25T05:53:29.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3629999999999996e-06 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558920+0000", 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061224599999999999 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558926+0000", 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.891e-06 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584147+0000", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0252211170000001 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584221+0000", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4258999999999995e-05 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584294+0000", 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2475e-05 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589475+0000", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051810500000000004 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589486+0000", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1388e-05 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589536+0000", 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0037999999999998e-05 2024-04-25T05:53:29.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589566+0000", 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9144000000000001e-05 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589589+0000", 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3410000000000001e-05 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.541 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195491 10.4 10:22b59bea:::benchmark_data_smithi043_152558_object67141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94603365900002, 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0361324119999999, 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195491 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4999999999999998e-07 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558325+0000", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2699999999999999e-06 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558330+0000", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0189999999999997e-06 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558335+0000", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0799999999999996e-06 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558939+0000", 2024-04-25T05:53:29.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060434400000000004 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558943+0000", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.036e-06 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.585996+0000", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0270530360000001 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586084+0000", 2024-04-25T05:53:29.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7683999999999996e-05 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586156+0000", 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1772000000000003e-05 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589499+0000", 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033435169999999998 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594275+0000", 2024-04-25T05:53:29.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047758619999999996 2024-04-25T05:53:29.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594326+0000", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0866999999999999e-05 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594406+0000", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9572000000000003e-05 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594455+0000", 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9002999999999998e-05 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195496 10.4 10:23a4e9e2:::benchmark_data_smithi043_152558_object67146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.94177926999998, 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.032050406, 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195496 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.549 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562578+0000", 2024-04-25T05:53:29.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0470000000000001e-06 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562581+0000", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.839e-06 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562590+0000", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2190000000000004e-06 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562595+0000", 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.468e-06 2024-04-25T05:53:29.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562692+0000", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7392999999999994e-05 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562699+0000", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7109999999999998e-06 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587618+0000", 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.024919208 2024-04-25T05:53:29.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587708+0000", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0075999999999996e-05 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587795+0000", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6837999999999998e-05 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594287+0000", 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064916610000000001 2024-04-25T05:53:29.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594498+0000", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021106299999999999 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594545+0000", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.655e-05 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594590+0000", 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5574000000000001e-05 2024-04-25T05:53:29.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594627+0000", 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6758999999999998e-05 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195505 10.1 10:82918380:::benchmark_data_smithi043_152558_object67155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.92853627599999, 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.053126166, 2024-04-25T05:53:29.555 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195505 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575821+0000", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.08e-06 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575827+0000", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6819999999999997e-06 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575835+0000", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2160000000000006e-06 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575841+0000", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2499999999999997e-06 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575897+0000", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6360000000000002e-05 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575904+0000", 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1749999999999999e-06 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602674+0000", 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.02677006 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603270+0000", 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00059573700000000005 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603459+0000", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018862499999999999 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615485+0000", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012026722 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621556+0000", 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060705749999999999 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628800+0000", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072444809999999997 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628892+0000", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1471000000000006e-05 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628946+0000", 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3732e-05 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.561 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195517 10.4 10:24331628:::benchmark_data_smithi043_152558_object67167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.91753625799998, 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007987988, 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.562 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195517 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.563 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586821+0000", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.252e-06 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586825+0000", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586836+0000", 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0988999999999999e-05 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586845+0000", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586911+0000", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5667000000000007e-05 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586923+0000", 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2104e-05 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589644+0000", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0027209029999999 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589733+0000", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9178000000000006e-05 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589854+0000", 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120975 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594292+0000", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044382069999999996 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594675+0000", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038344300000000002 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594716+0000", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1041999999999998e-05 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594767+0000", 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0926e-05 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594807+0000", 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.019e-05 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195519 10.4 10:29710d26:::benchmark_data_smithi043_152558_object67169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:53:29.568 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.91336205499999, 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.005228536, 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195519 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590995+0000", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6339999999999999e-06 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591000+0000", 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.147e-06 2024-04-25T05:53:29.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591014+0000", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3421000000000001e-05 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591022+0000", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1899999999999995e-06 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591079+0000", 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6852999999999999e-05 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591090+0000", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1545999999999999e-05 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591626+0000", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.000535591 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591729+0000", 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102774 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591850+0000", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121255 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594508+0000", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026583100000000001 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596021+0000", 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015122289999999999 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596061+0000", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0339999999999997e-05 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596174+0000", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112823 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596222+0000", 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8423e-05 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.574 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195520 10.1 10:8be1fae5:::benchmark_data_smithi043_152558_object67170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.91331594000002, 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0383997030000001, 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195520 2024-04-25T05:53:29.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:53:29.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1899999999999998e-07 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591045+0000", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7779999999999998e-06 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591051+0000", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.784e-06 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591056+0000", 2024-04-25T05:53:29.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591110+0000", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3823000000000003e-05 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591117+0000", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0520000000000004e-06 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612643+0000", 2024-04-25T05:53:29.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0215256699999999 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612745+0000", 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102353 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612856+0000", 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011074900000000001 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615525+0000", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002668784 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628786+0000", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013261088000000001 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629032+0000", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024633800000000003 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629115+0000", 2024-04-25T05:53:29.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2599e-05 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629440+0000", 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00032487500000000002 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195526 10.1 10:8fa6926a:::benchmark_data_smithi043_152558_object67176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.909296982, 2024-04-25T05:53:29.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.037524509, 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195526 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595060+0000", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.488e-06 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595066+0000", 2024-04-25T05:53:29.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6559999999999997e-06 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595080+0000", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3925e-05 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595089+0000", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6810000000000004e-06 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595171+0000", 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2888999999999999e-05 2024-04-25T05:53:29.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595186+0000", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4477e-05 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615572+0000", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0203861359999999 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615673+0000", 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101087 2024-04-25T05:53:29.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.616535+0000", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00086202200000000005 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621514+0000", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049784570000000004 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629450+0000", 2024-04-25T05:53:29.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0079360639999999996 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632427+0000", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029770019999999999 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632527+0000", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100078 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632583+0000", 2024-04-25T05:53:29.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6546999999999999e-05 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195534 10.4 10:2bc4d148:::benchmark_data_smithi043_152558_object67184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.89891253899998, 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.004935758, 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.589 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195534 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605445+0000", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.59e-06 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605451+0000", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3500000000000002e-06 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605466+0000", 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4555e-05 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605474+0000", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8639999999999996e-06 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605539+0000", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5467000000000002e-05 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605553+0000", 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4328000000000001e-05 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596256+0000", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99070311899999997 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596366+0000", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109079 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596569+0000", 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020323699999999999 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.600295+0000", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00372573 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602218+0000", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001922972 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609480+0000", 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072625119999999996 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609538+0000", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7531999999999997e-05 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.610379+0000", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000841423 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195540 10.1 10:8978d76f:::benchmark_data_smithi043_152558_object67190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:53:29.595 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 260.88898404899999, 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.017365112, 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195540 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615373+0000", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.311e-06 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615378+0000", 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3249999999999998e-06 2024-04-25T05:53:29.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615391+0000", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2245e-05 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615397+0000", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7499999999999997e-06 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615462+0000", 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4750000000000002e-05 2024-04-25T05:53:29.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615470+0000", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6550000000000004e-06 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.618963+0000", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.003493653 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619188+0000", 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022493599999999999 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619317+0000", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012912399999999999 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621613+0000", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022951650000000001 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629456+0000", 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0078430449999999999 2024-04-25T05:53:29.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632635+0000", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003178883 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632691+0000", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5965999999999999e-05 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632737+0000", 2024-04-25T05:53:29.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6304000000000003e-05 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232724 10.17 10:eb2728fc:::benchmark_data_smithi043_152558_object104374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 227.18694914400001, 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0079804210000001, 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:29.602 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232724 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317409+0000", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9240000000000001e-06 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317418+0000", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3579999999999992e-06 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317437+0000", 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9550000000000001e-05 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317449+0000", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1626e-05 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317528+0000", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9333999999999996e-05 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317539+0000", 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0848e-05 2024-04-25T05:53:29.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322050+0000", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0045109210000001 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322151+0000", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101123 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322389+0000", 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023737899999999999 2024-04-25T05:53:29.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325247+0000", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002858401 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325260+0000", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3125999999999999e-05 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325296+0000", 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6239000000000003e-05 2024-04-25T05:53:29.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325348+0000", 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1251000000000001e-05 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325387+0000", 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9341000000000002e-05 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:210734 11.e 11:72abaf1b:::benchmark_data_smithi043_179003_object80445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:13.352572+0000", 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 16.151783850000001, 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.048615506000000003, 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 210734 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352572+0000", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352572+0000", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352573+0000", 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2699999999999998e-07 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352578+0000", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0810000000000001e-06 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352594+0000", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5858999999999998e-05 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.352601+0000", 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3610000000000003e-06 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.359127+0000", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0065262920000000004 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.359273+0000", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000145403 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.359407+0000", 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013408199999999999 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.612 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.361864+0000", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024566179999999998 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.361885+0000", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.1140000000000001e-05 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.401073+0000", 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039188005999999997 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.613 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.401132+0000", 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9768000000000001e-05 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.401187+0000", 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4969000000000002e-05 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211897 11.e 11:7dd02b31:::benchmark_data_smithi043_179003_object81608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:53:29.614 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 15.007322526999999, 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.108093957, 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211897 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:53:29.615 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497035+0000", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.587e-06 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497042+0000", 2024-04-25T05:53:29.616 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7710000000000001e-06 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497053+0000", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1044e-05 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497061+0000", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4859999999999997e-06 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497615+0000", 2024-04-25T05:53:29.617 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055385700000000005 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497748+0000", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013329999999999999 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497865+0000", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116537 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502484+0000", 2024-04-25T05:53:29.618 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046191119999999999 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504633+0000", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002148736 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605053+0000", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100420537 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605093+0000", 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9514999999999997e-05 2024-04-25T05:53:29.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605127+0000", 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4474999999999998e-05 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211899 11.e 11:79b67a63:::benchmark_data_smithi043_179003_object81610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 15.006199989000001, 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.107157866, 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.620 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211899 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.621 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498157+0000", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0729999999999999e-06 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498161+0000", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0770000000000001e-06 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.622 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498172+0000", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0849999999999999e-05 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498181+0000", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7420000000000003e-06 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500317+0000", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021363480000000002 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.623 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500437+0000", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120166 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500515+0000", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8140999999999996e-05 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504641+0000", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041259390000000003 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.624 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504707+0000", 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6043000000000004e-05 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605187+0000", 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10047949 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.625 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.625 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605249+0000", 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2296000000000001e-05 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605314+0000", 2024-04-25T05:53:29.626 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4701000000000003e-05 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211902 11.e 11:7cddbeec:::benchmark_data_smithi043_179003_object81613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 15.003034645, 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.104091134, 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:29.627 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211902 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.628 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501322+0000", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1540000000000001e-06 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501327+0000", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9200000000000003e-06 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501339+0000", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2099000000000001e-05 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:29.629 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501347+0000", 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4730000000000003e-06 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502526+0000", 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011796899999999999 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502660+0000", 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013350800000000001 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502787+0000", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012737499999999999 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504782+0000", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019942509999999998 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.505325+0000", 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054379099999999996 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.631 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605345+0000", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100019497 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605374+0000", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9099999999999999e-05 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605412+0000", 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8275999999999997e-05 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.632 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:29.633 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:29.633 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[27010]: pgmap v1055: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 901 op/s 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[37492]: pgmap v1055: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 901 op/s 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:53:29.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:53:29.670 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:29.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:29.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:29.917 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:29.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:29.974 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:29.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:30.061 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:53:30.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.114 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:30.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:30.215 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:30.260 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 3.18 scrub starts 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 3.18 scrub ok 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 4.4 scrub starts 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 4.4 scrub ok 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 3.9 deep-scrub starts 2024-04-25T05:53:30.357 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[27010]: 3.9 deep-scrub ok 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 3.18 scrub starts 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 3.18 scrub ok 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 4.4 scrub starts 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 4.4 scrub ok 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 3.9 deep-scrub starts 2024-04-25T05:53:30.358 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:30 smithi043 ceph-mon[37492]: 3.9 deep-scrub ok 2024-04-25T05:53:30.362 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:30.515 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.553 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:30.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:30.654 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:30.702 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 3.18 scrub starts 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 3.18 scrub ok 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 4.4 scrub starts 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 4.4 scrub ok 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 3.9 deep-scrub starts 2024-04-25T05:53:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:30 smithi060 ceph-mon[30898]: 3.9 deep-scrub ok 2024-04-25T05:53:30.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:30.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:30.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:30.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:30.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:31.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:53:31.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:31.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:31.269 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: pgmap v1056: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 4.f scrub starts 2024-04-25T05:53:31.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 4.f scrub ok 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 4.d scrub starts 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: 4.d scrub ok 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1924195492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: pgmap v1056: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:53:31.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 4.f scrub starts 2024-04-25T05:53:31.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 4.f scrub ok 2024-04-25T05:53:31.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 4.d scrub starts 2024-04-25T05:53:31.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: 4.d scrub ok 2024-04-25T05:53:31.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1924195492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:31 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:31.428 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:31.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:53:31.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: pgmap v1056: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 4.f scrub starts 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 4.f scrub ok 2024-04-25T05:53:31.547 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 4.d scrub starts 2024-04-25T05:53:31.548 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: 4.d scrub ok 2024-04-25T05:53:31.548 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1924195492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.548 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:31 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:53:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.564 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:31.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:31.709 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:31.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:31.820 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:53:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.847 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:31.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:31.949 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:31.998 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:31.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:32.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.157 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:32.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:32.259 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:32.269 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:53:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:32.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:32.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: osdmap e563: 8 total, 8 up, 8 in 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: 4.9 scrub starts 2024-04-25T05:53:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:32 smithi060 ceph-mon[30898]: 4.9 scrub ok 2024-04-25T05:53:32.504 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: osdmap e563: 8 total, 8 up, 8 in 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: 4.9 scrub starts 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[27010]: 4.9 scrub ok 2024-04-25T05:53:32.558 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:53:32.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:53:32.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:53:32.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: osdmap e563: 8 total, 8 up, 8 in 2024-04-25T05:53:32.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: 4.9 scrub starts 2024-04-25T05:53:32.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:32 smithi043 ceph-mon[37492]: 4.9 scrub ok 2024-04-25T05:53:32.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:32.635 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:32.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:32.792 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:32.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:32.938 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:32.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:33.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:33.068 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:53:33.068 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:53:33.069 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:53:33.069 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:53:33.069 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:33.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:33.205 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:33.306 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:33.346 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:33.429 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:33.450 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:33.478 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.479 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:33.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:33.525 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:33.556 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[27010]: pgmap v1058: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[27010]: 4.16 scrub starts 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[27010]: 4.16 scrub ok 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[37492]: pgmap v1058: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[37492]: 4.16 scrub starts 2024-04-25T05:53:33.557 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:33 smithi043 ceph-mon[37492]: 4.16 scrub ok 2024-04-25T05:53:33.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:33.593 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.621 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:33.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:33.629 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:33.670 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:53:33.670 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:53:33.670 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-mon[30898]: pgmap v1058: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:33.670 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-mon[30898]: 4.16 scrub starts 2024-04-25T05:53:33.670 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-mon[30898]: 4.16 scrub ok 2024-04-25T05:53:33.674 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:33.715 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:33.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:33.758 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:33.771 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:33.772 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.778 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:33.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:33.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:33.911 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:33.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:33.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:34.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:34.041 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:34.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:53:34.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:34.184 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:34.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:34.327 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:34.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:34.467 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:34.497 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/943778718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:34.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:53:34.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:34 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:53:34.569 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:53:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:53:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/943778718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:53:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/943778718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:53:34.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:34 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:53:34.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:34.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:34.887 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:34.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:34.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:35.013 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:35.048 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:35.128 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:53:35.145 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:35.145 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:35.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.194 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:35.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:35.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.324 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:35.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:35.426 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:35.453 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:35.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:35.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: pgmap v1059: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 11.3 scrub starts 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3703090062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: 11.3 scrub ok 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[27010]: osdmap e564: 8 total, 8 up, 8 in 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: pgmap v1059: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:53:35.593 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 11.3 scrub starts 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3703090062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: 11.3 scrub ok 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:53:35.594 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:35 smithi043 ceph-mon[37492]: osdmap e564: 8 total, 8 up, 8 in 2024-04-25T05:53:35.668 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:53:35.668 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:53:35.668 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:35.668 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: pgmap v1059: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 11.3 scrub starts 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3703090062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: 11.3 scrub ok 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:53:35.669 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:35 smithi060 ceph-mon[30898]: osdmap e564: 8 total, 8 up, 8 in 2024-04-25T05:53:35.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:35.721 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:35.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:35.856 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:35.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:35.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:35.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:36.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:36.132 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:36.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:36.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:36.315 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2734363605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:53:36.332 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:53:36.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2734363605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:36 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.377 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:36.419 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:36.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:36.557 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:36.584 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:53:36.584 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:36 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:53:36.584 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:36 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:53:36.584 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2734363605' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.584 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:36 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:53:36.658 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:36.697 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:36.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:36.806 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:53:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.847 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:36.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:36.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:36.985 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:36.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:37.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:37.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:37.240 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:37.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.295 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:37.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:37.396 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.435 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:37.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:37 smithi060 ceph-mon[30898]: pgmap v1061: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:37 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:53:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:37 smithi060 ceph-mon[30898]: osdmap e565: 8 total, 8 up, 8 in 2024-04-25T05:53:37.536 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.573 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:37.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[27010]: pgmap v1061: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:37.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:53:37.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[27010]: osdmap e565: 8 total, 8 up, 8 in 2024-04-25T05:53:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[37492]: pgmap v1061: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:37.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:53:37.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:37 smithi043 ceph-mon[37492]: osdmap e565: 8 total, 8 up, 8 in 2024-04-25T05:53:37.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:37.707 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:37.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:37.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:37.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:37.996 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:37.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:38.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:38.233 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:38.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:53:38.364 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:38.391 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:38.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.550 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:38.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:38.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:38.776 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:38.800 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:38.800 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.822 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:38.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:38.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-25T05:53:38.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:38.953 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:38.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:39.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:39.083 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:39.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:39.213 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:39.315 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:39.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:39.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:39 smithi043 ceph-mon[27010]: pgmap v1063: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 879 op/s 2024-04-25T05:53:39.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:39 smithi043 ceph-mon[37492]: pgmap v1063: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 879 op/s 2024-04-25T05:53:39.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:39.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.628 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:39.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:39.672 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:53:39.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:39 smithi060 ceph-mon[30898]: pgmap v1063: 158 pgs: 158 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 879 op/s 2024-04-25T05:53:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.792 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:39.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:39.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:39.922 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:39.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:39.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:40.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:40.055 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:40.066 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:40.100 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:40.151 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:40.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:40.197 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:40.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:40.247 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:40.291 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:40.338 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:40.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:40.350 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:40.368 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:40.386 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:40.405 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:40.406 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:40.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:40.495 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:40.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:53:40.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/899275699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:40.503 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[27010]: osdmap e566: 8 total, 8 up, 8 in 2024-04-25T05:53:40.504 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:53:40.504 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/899275699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:40.504 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:40 smithi043 ceph-mon[37492]: osdmap e566: 8 total, 8 up, 8 in 2024-04-25T05:53:40.596 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:40.636 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:40 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:53:40.636 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/899275699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:40.636 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:40 smithi060 ceph-mon[30898]: osdmap e566: 8 total, 8 up, 8 in 2024-04-25T05:53:40.638 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:40.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:40.865 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:40.891 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:40.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:40.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:53:40.993 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:41.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:41.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:41.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:41.336 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:53:41.353 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:41.356 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:41.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:41.503 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:41.605 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:41.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[27010]: pgmap v1065: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-04-25T05:53:41.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:41.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:53:41.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:53:41.614 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:41.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[37492]: pgmap v1065: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-04-25T05:53:41.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:41.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:53:41.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:53:41.615 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:41 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:41.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:41 smithi060 ceph-mon[30898]: pgmap v1065: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-04-25T05:53:41.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:41 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:41.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:41 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:53:41.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:41 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:53:41.618 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:41 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:41.645 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:41.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:41.782 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:41.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:41.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:42.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:42.051 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:42.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:42.302 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:42.311 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:239431 11.1 11:8ab95134:::benchmark_data_smithi043_179003_object109142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:42.304569+0000", 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0059999299999999997, 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060253210000000001, 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 239431 2024-04-25T05:53:42.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304569+0000", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304569+0000", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304570+0000", 2024-04-25T05:53:42.313 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.02e-06 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304575+0000", 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3940000000000004e-06 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304589+0000", 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3202e-05 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:42.314 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304596+0000", 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.695e-06 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304645+0000", 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8878999999999997e-05 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304772+0000", 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000126441 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.315 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.304891+0000", 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011906800000000001 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.307926+0000", 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030356580000000001 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.316 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:239433 11.3 11:c614b67f:::benchmark_data_smithi043_179003_object109144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:42.305785+0000", 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.0047837160000000004, 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048644530000000004, 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 239433 2024-04-25T05:53:42.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305785+0000", 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305785+0000", 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305786+0000", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9999999999999997e-07 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305789+0000", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4879999999999999e-06 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305798+0000", 2024-04-25T05:53:42.319 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.208e-06 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305803+0000", 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1900000000000003e-06 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305849+0000", 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6103999999999998e-05 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.320 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.305962+0000", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000113478 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.306086+0000", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012406 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.308764+0000", 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002677722 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.321 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.308821+0000", 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7197999999999997e-05 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:239428 11.3 11:c87229fc:::benchmark_data_smithi043_179003_object109139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:42.297243+0000", 2024-04-25T05:53:42.322 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 0.013325574999999999, 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013458843, 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 239428 2024-04-25T05:53:42.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297243+0000", 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297243+0000", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297244+0000", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4499999999999997e-07 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297248+0000", 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.873e-06 2024-04-25T05:53:42.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297257+0000", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2909999999999996e-06 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297264+0000", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6120000000000004e-06 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297473+0000", 2024-04-25T05:53:42.326 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020886900000000001 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297559+0000", 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6518000000000001e-05 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.297625+0000", 2024-04-25T05:53:42.327 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6204999999999994e-05 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.299712+0000", 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020861999999999999 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:42.300016+0000", 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00030430600000000001 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:42.328 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-04-25T05:53:42.329 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 3 2024-04-25T05:53:42.329 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:53:42.340 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:42.442 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:42.477 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:42.580 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:42.613 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:42.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:42 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:42.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:42.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:42 smithi060 ceph-mon[30898]: osdmap e567: 8 total, 8 up, 8 in 2024-04-25T05:53:42.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:42.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:42.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[27010]: osdmap e567: 8 total, 8 up, 8 in 2024-04-25T05:53:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1643431279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:42.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:42 smithi043 ceph-mon[37492]: osdmap e567: 8 total, 8 up, 8 in 2024-04-25T05:53:42.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:42.748 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:42.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:42.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:42.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:43.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:53:43.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.169 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:43.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:43.270 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:43.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.0 to 1 (10000) 2024-04-25T05:53:43.392 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:53:43.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:43.480 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:43.483 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:43.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:43.615 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:43.634 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:43.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:43.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: pgmap v1067: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:43.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: osdmap e568: 8 total, 8 up, 8 in 2024-04-25T05:53:43.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4015737556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.635 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: pgmap v1067: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: osdmap e568: 8 total, 8 up, 8 in 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4015737556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: pgmap v1067: 158 pgs: 158 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: osdmap e568: 8 total, 8 up, 8 in 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4015737556' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:43 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:53:43.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:43.762 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:43.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:43.888 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:43.899 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:43.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:44.000 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:44.041 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:44.092 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:53:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:53:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:53:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:53:44.093 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:53:44.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:44.272 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.313 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:44.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:44.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:44.444 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:44.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:44.581 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:44.638 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:44 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:53:44.638 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:44 smithi060 ceph-mon[30898]: osdmap e569: 8 total, 8 up, 8 in 2024-04-25T05:53:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:44 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:53:44.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:44 smithi043 ceph-mon[27010]: osdmap e569: 8 total, 8 up, 8 in 2024-04-25T05:53:44.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:44 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:53:44.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:44 smithi043 ceph-mon[37492]: osdmap e569: 8 total, 8 up, 8 in 2024-04-25T05:53:44.683 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:44.711 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:44.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:44.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:53:44.844 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:44.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:44.987 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:44.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:45.089 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:45.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:53:45.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:45.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:45.279 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:45.381 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.431 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:45.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:45.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:45.564 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[27010]: pgmap v1070: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:53:45.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[27010]: osdmap e570: 8 total, 8 up, 8 in 2024-04-25T05:53:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[37492]: pgmap v1070: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:53:45.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:45 smithi043 ceph-mon[37492]: osdmap e570: 8 total, 8 up, 8 in 2024-04-25T05:53:45.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:45.706 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:45 smithi060 ceph-mon[30898]: pgmap v1070: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-25T05:53:45.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:45 smithi060 ceph-mon[30898]: osdmap e570: 8 total, 8 up, 8 in 2024-04-25T05:53:45.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:45.833 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:45.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:45.964 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:45.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:46.066 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.104 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:46.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:46.163 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:53:46.163 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:53:46.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:53:46.163 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232758 10.16 10:6a536367:::benchmark_data_smithi043_152558_object104408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 243.80698703300001, 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99219000999999996, 2024-04-25T05:53:46.164 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.165 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232758 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350943+0000", 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3699999999999999e-07 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350948+0000", 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2919999999999997e-06 2024-04-25T05:53:46.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350957+0000", 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7089999999999997e-06 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350965+0000", 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5610000000000001e-06 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351038+0000", 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3561999999999998e-05 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351047+0000", 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4549999999999995e-06 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323496+0000", 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97244899900000004 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323612+0000", 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011592200000000001 2024-04-25T05:53:46.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323850+0000", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023792799999999999 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326511+0000", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026609490000000001 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329075+0000", 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002564607 2024-04-25T05:53:46.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343036+0000", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013961031 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343089+0000", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2522999999999998e-05 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343132+0000", 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3634999999999998e-05 2024-04-25T05:53:46.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232859 10.16 10:6a2a3b0c:::benchmark_data_smithi043_152558_object104509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 243.71942579899999, 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90477883999999997, 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.174 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232859 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:53:46.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438505+0000", 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5820000000000001e-06 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438514+0000", 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3529999999999995e-06 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438522+0000", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7870000000000007e-06 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438528+0000", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6030000000000004e-06 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438592+0000", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4029000000000006e-05 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438602+0000", 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8349999999999999e-06 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325262+0000", 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88666051999999995 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325370+0000", 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010740599999999999 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325516+0000", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000146389 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329163+0000", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036465989999999999 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329175+0000", 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2473999999999999e-05 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343188+0000", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014012754000000001 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343242+0000", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3992000000000001e-05 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343283+0000", 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0516999999999998e-05 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232861 10.16 10:6d117552:::benchmark_data_smithi043_152558_object104511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 243.71569646, 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90115485500000003, 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.181 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232861 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442234+0000", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2589999999999999e-06 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442239+0000", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442248+0000", 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4299999999999995e-06 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442257+0000", 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2199999999999998e-06 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442330+0000", 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2401e-05 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442341+0000", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1063999999999999e-05 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326560+0000", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88421923400000002 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326658+0000", 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7768000000000003e-05 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326835+0000", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000177527 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329180+0000", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023450599999999999 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330157+0000", 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00097623399999999998 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343317+0000", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013160420000000001 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343352+0000", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5067000000000002e-05 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343388+0000", 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5781000000000002e-05 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.187 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232866 10.16 10:6de63d91:::benchmark_data_smithi043_152558_object104516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 243.70128657500001, 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88762149599999995, 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.188 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232866 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.189 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456644+0000", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5700000000000001e-07 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456648+0000", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.69e-06 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456658+0000", 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0780000000000002e-06 2024-04-25T05:53:46.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456663+0000", 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5609999999999998e-06 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456723+0000", 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9577999999999998e-05 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456731+0000", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1929999999999993e-06 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327815+0000", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.871084317 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327953+0000", 2024-04-25T05:53:46.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000137389 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.328167+0000", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000214324 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329185+0000", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001018166 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330231+0000", 2024-04-25T05:53:46.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010455309999999999 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344181+0000", 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013950071 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344227+0000", 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6589999999999999e-05 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344264+0000", 2024-04-25T05:53:46.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7150999999999997e-05 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232888 10.12 10:4f53b376:::benchmark_data_smithi043_152558_object104538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 242.810200976, 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96954986499999996, 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.196 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232888 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347731+0000", 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.295e-06 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347736+0000", 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347752+0000", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6121e-05 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347760+0000", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1659999999999994e-06 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347880+0000", 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932199999999999 2024-04-25T05:53:46.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347889+0000", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.482e-06 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314176+0000", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96628726499999995 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314286+0000", 2024-04-25T05:53:46.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010990899999999999 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314365+0000", 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9061999999999996e-05 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317147+0000", 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027818740000000002 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317183+0000", 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5639000000000002e-05 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317219+0000", 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6324e-05 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.202 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317250+0000", 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0596000000000003e-05 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317278+0000", 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8479999999999998e-05 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232966 10.12 10:49a888b9:::benchmark_data_smithi043_152558_object104616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 242.76274359999999, 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92239090000000001, 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232966 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395187+0000", 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2399999999999997e-07 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395190+0000", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4920000000000002e-06 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395200+0000", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7589999999999992e-06 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395208+0000", 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8110000000000003e-06 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395287+0000", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9079000000000006e-05 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395295+0000", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8080000000000005e-06 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315576+0000", 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92028153000000001 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315678+0000", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102095 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315768+0000", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0304999999999997e-05 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317349+0000", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00158056 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.208 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317389+0000", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9878e-05 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317472+0000", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2762000000000005e-05 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317523+0000", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1730000000000001e-05 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.209 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317577+0000", 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3366999999999999e-05 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233055 10.12 10:4ec1d645:::benchmark_data_smithi043_152558_object104705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 242.71082219199999, 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87067929700000002, 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.210 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233055 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447109+0000", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4980000000000001e-06 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447114+0000", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1510000000000003e-06 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447128+0000", 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4012000000000001e-05 2024-04-25T05:53:46.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447136+0000", 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7579999999999992e-06 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447275+0000", 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013975899999999999 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.213 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447283+0000", 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6590000000000007e-06 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316413+0000", 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.86913016700000001 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316518+0000", 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010422800000000001 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.214 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316597+0000", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9097000000000005e-05 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317594+0000", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099745499999999996 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317631+0000", 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7342999999999998e-05 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.215 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317700+0000", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8861000000000004e-05 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317752+0000", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1872000000000001e-05 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317787+0000", 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4437000000000002e-05 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175433 11.1 11:8fba1ab6:::benchmark_data_smithi043_179003_object45144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.067154677000005, 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0345604850000001, 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175433 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090776+0000", 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.429e-06 2024-04-25T05:53:46.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090784+0000", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2649999999999999e-06 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090794+0000", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0552e-05 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090802+0000", 2024-04-25T05:53:46.219 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4819999999999997e-06 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.095151+0000", 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043491149999999998 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.095160+0000", 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4530000000000008e-06 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.220 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113721+0000", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0185605760000001 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113832+0000", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011136 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113914+0000", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2316000000000003e-05 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.221 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120575+0000", 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066603720000000003 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122121+0000", 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001545886 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125210+0000", 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030893130000000002 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125277+0000", 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7334000000000003e-05 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125335+0000", 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8032000000000003e-05 2024-04-25T05:53:46.223 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175439 11.1 11:8a7d81ef:::benchmark_data_smithi043_179003_object45150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.055253022000002, 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0228662150000001, 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.224 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175439 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102678+0000", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.451e-06 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102683+0000", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5180000000000004e-06 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102698+0000", 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4681999999999999e-05 2024-04-25T05:53:46.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102706+0000", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.157e-06 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102768+0000", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1227000000000002e-05 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102775+0000", 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3830000000000003e-06 2024-04-25T05:53:46.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116168+0000", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0133934600000001 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116237+0000", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8560999999999996e-05 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116308+0000", 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1373999999999998e-05 2024-04-25T05:53:46.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122048+0000", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005739971 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125378+0000", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033293210000000001 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125435+0000", 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7147e-05 2024-04-25T05:53:46.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125495+0000", 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0485000000000001e-05 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125543+0000", 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7478000000000002e-05 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.230 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175466 11.1 11:8d3c0ad7:::benchmark_data_smithi043_179003_object45177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.012037591999999, 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98401796699999999, 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175466 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.231 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145894+0000", 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.637e-06 2024-04-25T05:53:46.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145899+0000", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6579999999999999e-06 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145909+0000", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9199999999999999e-06 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145917+0000", 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1589999999999993e-06 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.146013+0000", 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6075000000000004e-05 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.146022+0000", 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4169999999999992e-06 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.234 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117397+0000", 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97137537299999999 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117465+0000", 2024-04-25T05:53:46.235 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7768000000000006e-05 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117530+0000", 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5222000000000007e-05 2024-04-25T05:53:46.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.237 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.237 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122052+0000", 2024-04-25T05:53:46.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:46.238 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:46.240 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045222259999999999 2024-04-25T05:53:46.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.240 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125583+0000", 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035309809999999999 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129846+0000", 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042627150000000003 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129881+0000", 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4703000000000003e-05 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129910+0000", 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9113000000000001e-05 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175468 11.1 11:81a32ea0:::benchmark_data_smithi043_179003_object45179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.242 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 71.009646253, 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.985623044, 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175468 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.243 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.244 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148284+0000", 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.569999999999999e-07 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148289+0000", 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7129999999999996e-06 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.245 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148302+0000", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2636999999999999e-05 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148310+0000", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148364+0000", 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4471000000000001e-05 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148374+0000", 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5480000000000007e-06 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118654+0000", 2024-04-25T05:53:46.247 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.970280166 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118718+0000", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4387000000000005e-05 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118893+0000", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017421799999999999 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123418+0000", 2024-04-25T05:53:46.248 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045254049999999997 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129746+0000", 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063283230000000003 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.249 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133841+0000", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040942069999999999 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133874+0000", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3290000000000001e-05 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133906+0000", 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2415000000000001e-05 2024-04-25T05:53:46.250 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175488 11.1 11:883cd18e:::benchmark_data_smithi043_179003_object45199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.990276049000002, 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96641953199999997, 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.251 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175488 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.252 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167655+0000", 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5680000000000001e-06 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167660+0000", 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7600000000000002e-06 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167671+0000", 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0910000000000001e-05 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167676+0000", 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4809999999999997e-06 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167732+0000", 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5695999999999997e-05 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167741+0000", 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4979999999999995e-06 2024-04-25T05:53:46.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120665+0000", 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95292317699999995 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120734+0000", 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9831000000000003e-05 2024-04-25T05:53:46.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120808+0000", 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3175000000000003e-05 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.257 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125163+0000", 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004355261 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129797+0000", 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046343199999999999 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133968+0000", 2024-04-25T05:53:46.258 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041712779999999996 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134028+0000", 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9709e-05 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134073+0000", 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4867999999999998e-05 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.259 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175491 11.1 11:86b5808b:::benchmark_data_smithi043_179003_object45202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.986653602999993, 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.974324462, 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.260 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175491 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171277+0000", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6899999999999996e-07 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171281+0000", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2169999999999996e-06 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171290+0000", 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8130000000000002e-06 2024-04-25T05:53:46.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171295+0000", 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-06 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171346+0000", 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0958000000000002e-05 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.263 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171356+0000", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3519999999999996e-06 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122164+0000", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95080842499999996 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122231+0000", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7368999999999998e-05 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.264 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122299+0000", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7893999999999997e-05 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125386+0000", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030862279999999999 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129945+0000", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045593869999999998 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.265 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145473+0000", 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0155277 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145548+0000", 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5539999999999998e-05 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145600+0000", 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.219e-05 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175505 11.1 11:8036578e:::benchmark_data_smithi043_179003_object45216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.975049655999996, 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96335497000000003, 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175505 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.267 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182881+0000", 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.054e-06 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.268 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182886+0000", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8539999999999996e-06 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182895+0000", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6309999999999998e-06 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182904+0000", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1529999999999997e-06 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.269 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182990+0000", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6946000000000004e-05 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182999+0000", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7150000000000004e-06 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123438+0000", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94043926 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.270 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123505+0000", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6879999999999997e-05 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123591+0000", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5607999999999998e-05 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125590+0000", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019993179999999999 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.271 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130005+0000", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044150659999999996 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146160+0000", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.016154294999999999 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146193+0000", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3114999999999997e-05 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146235+0000", 2024-04-25T05:53:46.272 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2074999999999999e-05 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175516 11.1 11:8230b7a3:::benchmark_data_smithi043_179003_object45227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.964332235000001, 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95606739399999996, 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.273 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175516 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.274 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193599+0000", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4360000000000001e-06 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193605+0000", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2310000000000001e-06 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193619+0000", 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4037e-05 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.275 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193627+0000", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2109999999999998e-06 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193690+0000", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3099e-05 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193696+0000", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2020000000000001e-06 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.276 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125612+0000", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.93191594499999997 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125737+0000", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012487699999999999 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125873+0000", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000135437 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129755+0000", 2024-04-25T05:53:46.277 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038821340000000002 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.140090+0000", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010335247000000001 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149600+0000", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0095101639999999998 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149636+0000", 2024-04-25T05:53:46.278 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5814e-05 2024-04-25T05:53:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.279 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149665+0000", 2024-04-25T05:53:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8560000000000001e-05 2024-04-25T05:53:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175586 11.1 11:81fb6593:::benchmark_data_smithi043_179003_object45297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.895865008000001, 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88770023200000003, 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175586 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262066+0000", 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6810000000000001e-06 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262072+0000", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1630000000000001e-06 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262088+0000", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5690999999999999e-05 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262096+0000", 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3829999999999992e-06 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262160+0000", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3906999999999996e-05 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262169+0000", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6840000000000002e-06 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.127891+0000", 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.86572203599999997 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.128019+0000", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127787 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.128139+0000", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119997 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129956+0000", 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018173009999999999 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.286 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.140176+0000", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010219788 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149703+0000", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0095273489999999992 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149735+0000", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2048000000000002e-05 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.287 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149765+0000", 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9417e-05 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175599 11.1 11:8d1f405c:::benchmark_data_smithi043_179003_object45310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.880177270000004, 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87487917199999998, 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.288 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175599 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.289 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277753+0000", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7000000000000003e-07 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277761+0000", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5660000000000001e-06 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277770+0000", 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5920000000000007e-06 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277776+0000", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4199999999999998e-06 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277820+0000", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4669000000000002e-05 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277826+0000", 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.291 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130039+0000", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85221334000000004 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130112+0000", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3145000000000005e-05 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130178+0000", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5712999999999999e-05 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.292 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134087+0000", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003909432 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149796+0000", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015708895 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152515+0000", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002718416 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.293 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152587+0000", 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2083999999999996e-05 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152631+0000", 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4761000000000001e-05 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175618 11.1 11:812e2f85:::benchmark_data_smithi043_179003_object45329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 70.859044159999996, 2024-04-25T05:53:46.294 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85387319399999995, 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175618 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.295 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298886+0000", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0330000000000001e-06 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298891+0000", 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.296 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298899+0000", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0129999999999993e-06 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298904+0000", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9989999999999999e-06 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298961+0000", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7679000000000001e-05 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:53:46.297 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298971+0000", 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2359999999999993e-06 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132062+0000", 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.83309193000000004 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132184+0000", 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121699 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.298 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132318+0000", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000134268 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.138055+0000", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057369680000000003 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149845+0000", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011789714999999999 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.299 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152668+0000", 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0028230780000000001 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152723+0000", 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5148999999999997e-05 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152759+0000", 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5185999999999998e-05 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:194360 11.1 11:803a97fb:::benchmark_data_smithi043_179003_object64071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.435221146000004, 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.085195066999999999, 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 194360 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722709+0000", 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0690000000000001e-06 2024-04-25T05:53:46.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722714+0000", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6859999999999997e-06 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722723+0000", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3850000000000008e-06 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722729+0000", 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.587e-06 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722779+0000", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0232000000000003e-05 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722871+0000", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2007000000000002e-05 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722998+0000", 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012624899999999999 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.725628+0000", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026307570000000001 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.725649+0000", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0752000000000001e-05 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807786+0000", 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082136946000000002 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.305 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807852+0000", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6313e-05 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807903+0000", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1084000000000001e-05 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211279 11.1 11:85df86fe:::benchmark_data_smithi043_179003_object80990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:13.860751+0000", 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 32.297178680000002, 2024-04-25T05:53:46.306 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.084514769000000003, 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211279 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860751+0000", 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.307 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860751+0000", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860752+0000", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.33e-06 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860758+0000", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7690000000000002e-06 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:53:46.308 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860774+0000", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6104e-05 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860783+0000", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5909999999999996e-06 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.860878+0000", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5542000000000001e-05 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.861031+0000", 2024-04-25T05:53:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00015330699999999999 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.861163+0000", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131249 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.866163+0000", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049998050000000004 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.866175+0000", 2024-04-25T05:53:46.310 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2245e-05 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.945157+0000", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.078981843999999995 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.945217+0000", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0748999999999997e-05 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:13.945266+0000", 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8233999999999999e-05 2024-04-25T05:53:46.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:53:46.312 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:53:46.312 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:53:46.312 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:53:46.312 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:53:46.338 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:46.440 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:46.471 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:46.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:46.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:46.712 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:46.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:46.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:46.878 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:46.898 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:46.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:46.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:47.012 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.050 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:47.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:47.053 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:47.098 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:47.139 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:47.151 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:47.180 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:47.194 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:47.216 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:53:47.220 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:47.267 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:47.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:47.328 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:47.332 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.338 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:47.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:47.439 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:47 smithi060 ceph-mon[30898]: pgmap v1072: 158 pgs: 158 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:53:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/307766004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:47.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:47.593 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.612 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:47.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:47.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:47 smithi043 ceph-mon[27010]: pgmap v1072: 158 pgs: 158 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:53:47.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/307766004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:47.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:47 smithi043 ceph-mon[37492]: pgmap v1072: 158 pgs: 158 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-25T05:53:47.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/307766004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:47.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.747 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:47.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:47.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:47.890 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:47.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:47.898 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:47.991 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:48.033 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:48.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.173 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:48.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:48.253 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:48.253 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:48.274 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:48.305 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:48.407 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:48.434 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:53:48.437 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:48.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:48.576 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:48.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:48.708 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:48 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3296457693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:48 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:48.838 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:48.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:48 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3296457693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:48 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:48 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3296457693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:48 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:53:48.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:48.962 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:48.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:48.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:53:49.063 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:49.093 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:49.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:49.228 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:49.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:49.340 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.365 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:49.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:49.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:49.496 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:49.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:49.630 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:49.675 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: pgmap v1073: 158 pgs: 158 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 958 op/s 2024-04-25T05:53:49.675 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:49.675 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:49.675 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: osdmap e571: 8 total, 8 up, 8 in 2024-04-25T05:53:49.675 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:49.676 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:49.676 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:53:49.676 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:49 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: pgmap v1073: 158 pgs: 158 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 958 op/s 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: osdmap e571: 8 total, 8 up, 8 in 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: pgmap v1073: 158 pgs: 158 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 958 op/s 2024-04-25T05:53:49.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: osdmap e571: 8 total, 8 up, 8 in 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:53:49.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:53:49.732 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:49.771 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:49.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:49.906 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:49.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:49.933 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:53:49.975 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:49.975 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:50.007 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:50.045 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:50.147 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:50.178 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:50.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:50.317 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:50.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:50.456 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:50.468 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[27010]: osdmap e572: 8 total, 8 up, 8 in 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:53:50.469 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:50 smithi043 ceph-mon[37492]: osdmap e572: 8 total, 8 up, 8 in 2024-04-25T05:53:50.558 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:50.610 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:50.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:50 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:53:50.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:50 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:53:50.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:50 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:53:50.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:50 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:53:50.631 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:50 smithi060 ceph-mon[30898]: osdmap e572: 8 total, 8 up, 8 in 2024-04-25T05:53:50.674 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:53:50.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:50.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:50.858 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:50.896 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:50.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:50.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:51.034 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:51.056 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:53:51.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:51.166 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:51.268 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:51.308 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:51.383 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:53:51.389 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:53:51.410 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:51.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: pgmap v1076: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 1.7 scrub starts 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 1.7 scrub ok 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:53:51.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:53:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:53:51.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:51 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:53:51.533 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:53:51.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:51.576 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:51.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:51.723 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: pgmap v1076: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 1.7 scrub starts 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 1.7 scrub ok 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:51.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: pgmap v1076: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.23k op/s 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 1.7 scrub starts 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 1.7 scrub ok 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:53:51.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:53:51.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:51.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:51.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:53:51.795 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:51 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:53:51.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:51.862 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:51.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:51.997 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:51.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:52.026 INFO:teuthology.orchestra.run.smithi060.stdout:ERROR: (22) Invalid argument 2024-04-25T05:53:52.026 INFO:teuthology.orchestra.run.smithi060.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:53:52.098 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.133 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:52.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:52.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:52.270 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:52.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:53:52.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:52.401 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:52.433 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1076,"stamp":"2024-04-25T05:53:50.123177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":915544627,"num_objects":14192,"num_object_clones":0,"num_object_copies":42576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14192,"num_whiteouts":0,"num_read":2584,"num_read_kb":2304,"num_write":247526,"num_write_kb":8338350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28897,"num_bytes_recovered":1850560081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81804,"ondisk_log_size":81804,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10352776,"kb_used_data":3309184,"kb_used_omap":12,"kb_used_meta":7042803,"kb_avail":739608440,"statfs":{"total":767960285184,"available":757359042560,"internally_reserved":0,"allocated":3388604416,"data_stored":3383316893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7211830832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,12,5,5,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-484048896,"num_objects":-7386,"num_object_clones":0,"num_object_copies":-22158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7386,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001481"},"pg_stats":[{"pgid":"11.3","version":"571'15452","reported_seq":15795,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.660196+0000","last_change":"2024-04-25T05:53:34.830889+0000","last_active":"2024-04-25T05:53:48.660196+0000","last_peered":"2024-04-25T05:53:48.660196+0000","last_clean":"2024-04-25T05:53:48.660196+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:53:48.660196+0000","last_undegraded":"2024-04-25T05:53:48.660196+0000","last_fullsized":"2024-04-25T05:53:48.660196+0000","mapping_epoch":456,"log_start":"520'10246","ondisk_log_start":"520'10246","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'14507","last_scrub_stamp":"2024-04-25T05:53:34.830724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.830724+0000","objects_scrubbed":1862,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:55.510615+0000","scrub_duration":573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15452,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":565,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.117409+0000","last_change":"2024-04-25T05:52:35.929420+0000","last_active":"2024-04-25T05:53:45.117409+0000","last_peered":"2024-04-25T05:53:45.117409+0000","last_clean":"2024-04-25T05:53:45.117409+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:53:45.117409+0000","last_undegraded":"2024-04-25T05:53:45.117409+0000","last_fullsized":"2024-04-25T05:53:45.117409+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:52:35.929307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.929307+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.731090+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1082,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437355+0000","last_change":"2024-04-25T05:51:40.101355+0000","last_active":"2024-04-25T05:53:48.437355+0000","last_peered":"2024-04-25T05:53:48.437355+0000","last_clean":"2024-04-25T05:53:48.437355+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:53:48.437355+0000","last_undegraded":"2024-04-25T05:53:48.437355+0000","last_fullsized":"2024-04-25T05:53:48.437355+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":526,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.120900+0000","last_change":"2024-04-25T05:52:52.397154+0000","last_active":"2024-04-25T05:53:45.120900+0000","last_peered":"2024-04-25T05:53:45.120900+0000","last_clean":"2024-04-25T05:53:45.120900+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:53:45.120900+0000","last_undegraded":"2024-04-25T05:53:45.120900+0000","last_fullsized":"2024-04-25T05:53:45.120900+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:52:52.394206+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.394206+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:54.399800+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":642,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.116832+0000","last_change":"2024-04-25T05:52:08.909918+0000","last_active":"2024-04-25T05:53:45.116832+0000","last_peered":"2024-04-25T05:53:45.116832+0000","last_clean":"2024-04-25T05:53:45.116832+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:53:45.116832+0000","last_undegraded":"2024-04-25T05:53:45.116832+0000","last_fullsized":"2024-04-25T05:53:45.116832+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:08.909664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.909664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.033061+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1424,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434083+0000","last_change":"2024-04-25T05:51:58.168288+0000","last_active":"2024-04-25T05:53:48.434083+0000","last_peered":"2024-04-25T05:53:48.434083+0000","last_clean":"2024-04-25T05:53:48.434083+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:53:48.434083+0000","last_undegraded":"2024-04-25T05:53:48.434083+0000","last_fullsized":"2024-04-25T05:53:48.434083+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:58.168142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:58.168142+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"570'15530","reported_seq":29264,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:47.998419+0000","last_change":"2024-04-25T05:52:52.524759+0000","last_active":"2024-04-25T05:53:47.998419+0000","last_peered":"2024-04-25T05:53:47.998419+0000","last_clean":"2024-04-25T05:53:47.998419+0000","last_became_active":"2024-04-25T05:52:44.443285+0000","last_became_peered":"2024-04-25T05:52:44.443285+0000","last_unstale":"2024-04-25T05:53:47.998419+0000","last_undegraded":"2024-04-25T05:53:47.998419+0000","last_fullsized":"2024-04-25T05:53:47.998419+0000","mapping_epoch":540,"log_start":"507'9324","ondisk_log_start":"507'9324","created":456,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11957","last_scrub_stamp":"2024-04-25T05:52:52.524605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.524605+0000","objects_scrubbed":4563,"log_size":6206,"log_dups_size":0,"ondisk_log_size":6206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.589578+0000","scrub_duration":2115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15530,"num_write_kb":527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6713,"num_bytes_recovered":439877632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":574,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.120227+0000","last_change":"2024-04-25T05:53:30.906380+0000","last_active":"2024-04-25T05:53:45.120227+0000","last_peered":"2024-04-25T05:53:45.120227+0000","last_clean":"2024-04-25T05:53:45.120227+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:53:45.120227+0000","last_undegraded":"2024-04-25T05:53:45.120227+0000","last_fullsized":"2024-04-25T05:53:45.120227+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:53:30.906231+0000","last_deep_scrub":"320'39","last_deep_scrub_stamp":"2024-04-25T05:51:57.812833+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.906231+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:58.223526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":797,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426140+0000","last_change":"2024-04-25T05:51:22.353776+0000","last_active":"2024-04-25T05:53:48.426140+0000","last_peered":"2024-04-25T05:53:48.426140+0000","last_clean":"2024-04-25T05:53:48.426140+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:53:48.426140+0000","last_undegraded":"2024-04-25T05:53:48.426140+0000","last_fullsized":"2024-04-25T05:53:48.426140+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.353535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.353535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1095,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442927+0000","last_change":"2024-04-25T05:51:40.109308+0000","last_active":"2024-04-25T05:53:48.442927+0000","last_peered":"2024-04-25T05:53:48.442927+0000","last_clean":"2024-04-25T05:53:48.442927+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:53:48.442927+0000","last_undegraded":"2024-04-25T05:53:48.442927+0000","last_fullsized":"2024-04-25T05:53:48.442927+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444319+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":431,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.120040+0000","last_change":"2024-04-25T05:53:34.769498+0000","last_active":"2024-04-25T05:53:45.120040+0000","last_peered":"2024-04-25T05:53:45.120040+0000","last_clean":"2024-04-25T05:53:45.120040+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:53:45.120040+0000","last_undegraded":"2024-04-25T05:53:45.120040+0000","last_fullsized":"2024-04-25T05:53:45.120040+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.769373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.769373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.388264+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":1020,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122131+0000","last_change":"2024-04-25T05:52:07.018629+0000","last_active":"2024-04-25T05:53:45.122131+0000","last_peered":"2024-04-25T05:53:45.122131+0000","last_clean":"2024-04-25T05:53:45.122131+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:53:45.122131+0000","last_undegraded":"2024-04-25T05:53:45.122131+0000","last_fullsized":"2024-04-25T05:53:45.122131+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:52:07.018495+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:52:07.018495+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"571'15383","reported_seq":29260,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.677356+0000","last_change":"2024-04-25T05:52:36.116043+0000","last_active":"2024-04-25T05:53:48.677356+0000","last_peered":"2024-04-25T05:53:48.677356+0000","last_clean":"2024-04-25T05:53:48.677356+0000","last_became_active":"2024-04-25T05:52:36.097202+0000","last_became_peered":"2024-04-25T05:52:36.097202+0000","last_unstale":"2024-04-25T05:53:48.677356+0000","last_undegraded":"2024-04-25T05:53:48.677356+0000","last_fullsized":"2024-04-25T05:53:48.677356+0000","mapping_epoch":532,"log_start":"518'10165","ondisk_log_start":"518'10165","created":456,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'6932","last_scrub_stamp":"2024-04-25T05:51:30.948125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.948125+0000","objects_scrubbed":6847,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58392576,"num_objects":891,"num_object_clones":0,"num_object_copies":2673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15383,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6773,"num_bytes_recovered":443875328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":874,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.116237+0000","last_change":"2024-04-25T05:52:41.908835+0000","last_active":"2024-04-25T05:53:45.116237+0000","last_peered":"2024-04-25T05:53:45.116237+0000","last_clean":"2024-04-25T05:53:45.116237+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:53:45.116237+0000","last_undegraded":"2024-04-25T05:53:45.116237+0000","last_fullsized":"2024-04-25T05:53:45.116237+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:52:41.908693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.908693+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:56.986259+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1109,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114475+0000","last_change":"2024-04-25T05:53:29.887090+0000","last_active":"2024-04-25T05:53:45.114475+0000","last_peered":"2024-04-25T05:53:45.114475+0000","last_clean":"2024-04-25T05:53:45.114475+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:53:45.114475+0000","last_undegraded":"2024-04-25T05:53:45.114475+0000","last_fullsized":"2024-04-25T05:53:45.114475+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.887008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.890083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":406,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122878+0000","last_change":"2024-04-25T05:53:33.021664+0000","last_active":"2024-04-25T05:53:45.122878+0000","last_peered":"2024-04-25T05:53:45.122878+0000","last_clean":"2024-04-25T05:53:45.122878+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:53:45.122878+0000","last_undegraded":"2024-04-25T05:53:45.122878+0000","last_fullsized":"2024-04-25T05:53:45.122878+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.021506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.021506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.233074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1070,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436353+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:53:48.436353+0000","last_peered":"2024-04-25T05:53:48.436353+0000","last_clean":"2024-04-25T05:53:48.436353+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:53:48.436353+0000","last_undegraded":"2024-04-25T05:53:48.436353+0000","last_fullsized":"2024-04-25T05:53:48.436353+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"571'15378","reported_seq":15674,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.427481+0000","last_change":"2024-04-25T05:52:50.663942+0000","last_active":"2024-04-25T05:53:49.427481+0000","last_peered":"2024-04-25T05:53:49.427481+0000","last_clean":"2024-04-25T05:53:49.427481+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:53:49.427481+0000","last_undegraded":"2024-04-25T05:53:49.427481+0000","last_fullsized":"2024-04-25T05:53:49.427481+0000","mapping_epoch":456,"log_start":"527'10613","ondisk_log_start":"527'10613","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'11702","last_scrub_stamp":"2024-04-25T05:52:50.663796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.663796+0000","objects_scrubbed":4552,"log_size":4765,"log_dups_size":0,"ondisk_log_size":4765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:50.935234+0000","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53477376,"num_objects":816,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15378,"num_write_kb":518208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":1119,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.450643+0000","last_change":"2024-04-25T05:53:30.416648+0000","last_active":"2024-04-25T05:53:48.450643+0000","last_peered":"2024-04-25T05:53:48.450643+0000","last_clean":"2024-04-25T05:53:48.450643+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:53:48.450643+0000","last_undegraded":"2024-04-25T05:53:48.450643+0000","last_fullsized":"2024-04-25T05:53:48.450643+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:30.416481+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.416481+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.310035+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1067,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.439658+0000","last_change":"2024-04-25T05:51:20.429881+0000","last_active":"2024-04-25T05:53:48.439658+0000","last_peered":"2024-04-25T05:53:48.439658+0000","last_clean":"2024-04-25T05:53:48.439658+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:53:48.439658+0000","last_undegraded":"2024-04-25T05:53:48.439658+0000","last_fullsized":"2024-04-25T05:53:48.439658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:20.429760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:20.429760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":774,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.116640+0000","last_change":"2024-04-25T05:51:46.715282+0000","last_active":"2024-04-25T05:53:45.116640+0000","last_peered":"2024-04-25T05:53:45.116640+0000","last_clean":"2024-04-25T05:53:45.116640+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:53:45.116640+0000","last_undegraded":"2024-04-25T05:53:45.116640+0000","last_fullsized":"2024-04-25T05:53:45.116640+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:51:46.715159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:46.715159+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.032942+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":412,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122657+0000","last_change":"2024-04-25T05:53:21.122301+0000","last_active":"2024-04-25T05:53:45.122657+0000","last_peered":"2024-04-25T05:53:45.122657+0000","last_clean":"2024-04-25T05:53:45.122657+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:53:45.122657+0000","last_undegraded":"2024-04-25T05:53:45.122657+0000","last_fullsized":"2024-04-25T05:53:45.122657+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.122207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.122207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:28.763652+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":984,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121756+0000","last_change":"2024-04-25T05:52:45.117231+0000","last_active":"2024-04-25T05:53:45.121756+0000","last_peered":"2024-04-25T05:53:45.121756+0000","last_clean":"2024-04-25T05:53:45.121756+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:53:45.121756+0000","last_undegraded":"2024-04-25T05:53:45.121756+0000","last_fullsized":"2024-04-25T05:53:45.121756+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.117108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.117108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.151932+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"571'15599","reported_seq":15918,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.447867+0000","last_change":"2024-04-25T05:52:09.488584+0000","last_active":"2024-04-25T05:53:48.447867+0000","last_peered":"2024-04-25T05:53:48.447867+0000","last_clean":"2024-04-25T05:53:48.447867+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:53:48.447867+0000","last_undegraded":"2024-04-25T05:53:48.447867+0000","last_fullsized":"2024-04-25T05:53:48.447867+0000","mapping_epoch":456,"log_start":"527'10869","ondisk_log_start":"527'10869","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'9496","last_scrub_stamp":"2024-04-25T05:52:09.488306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:09.488306+0000","objects_scrubbed":7109,"log_size":4730,"log_dups_size":0,"ondisk_log_size":4730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60751872,"num_objects":927,"num_object_clones":0,"num_object_copies":2781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15599,"num_write_kb":528832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1089,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114399+0000","last_change":"2024-04-25T05:53:33.920196+0000","last_active":"2024-04-25T05:53:45.114399+0000","last_peered":"2024-04-25T05:53:45.114399+0000","last_clean":"2024-04-25T05:53:45.114399+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:53:45.114399+0000","last_undegraded":"2024-04-25T05:53:45.114399+0000","last_fullsized":"2024-04-25T05:53:45.114399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.920077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.920077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.068133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1249,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.439761+0000","last_change":"2024-04-25T05:51:40.149035+0000","last_active":"2024-04-25T05:53:48.439761+0000","last_peered":"2024-04-25T05:53:48.439761+0000","last_clean":"2024-04-25T05:53:48.439761+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:53:48.439761+0000","last_undegraded":"2024-04-25T05:53:48.439761+0000","last_fullsized":"2024-04-25T05:53:48.439761+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:51:29.363198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.363198+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1105,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.438638+0000","last_change":"2024-04-25T05:53:20.767176+0000","last_active":"2024-04-25T05:53:48.438638+0000","last_peered":"2024-04-25T05:53:48.438638+0000","last_clean":"2024-04-25T05:53:48.438638+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:53:48.438638+0000","last_undegraded":"2024-04-25T05:53:48.438638+0000","last_fullsized":"2024-04-25T05:53:48.438638+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:20.767051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:20.767051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.097716+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":636,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114890+0000","last_change":"2024-04-25T05:52:42.951239+0000","last_active":"2024-04-25T05:53:45.114890+0000","last_peered":"2024-04-25T05:53:45.114890+0000","last_clean":"2024-04-25T05:53:45.114890+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:53:45.114890+0000","last_undegraded":"2024-04-25T05:53:45.114890+0000","last_fullsized":"2024-04-25T05:53:45.114890+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.951111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.951111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:56.859438+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1358,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.119693+0000","last_change":"2024-04-25T05:53:27.936176+0000","last_active":"2024-04-25T05:53:45.119693+0000","last_peered":"2024-04-25T05:53:45.119693+0000","last_clean":"2024-04-25T05:53:45.119693+0000","last_became_active":"2024-04-25T05:52:47.115419+0000","last_became_peered":"2024-04-25T05:52:47.115419+0000","last_unstale":"2024-04-25T05:53:45.119693+0000","last_undegraded":"2024-04-25T05:53:45.119693+0000","last_fullsized":"2024-04-25T05:53:45.119693+0000","mapping_epoch":379,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:27.935927+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:53:27.935927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:57.848657+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"570'14948","reported_seq":15244,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.247132+0000","last_change":"2024-04-25T05:52:42.423083+0000","last_active":"2024-04-25T05:53:48.247132+0000","last_peered":"2024-04-25T05:53:48.247132+0000","last_clean":"2024-04-25T05:53:48.247132+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:53:48.247132+0000","last_undegraded":"2024-04-25T05:53:48.247132+0000","last_fullsized":"2024-04-25T05:53:48.247132+0000","mapping_epoch":456,"log_start":"524'10285","ondisk_log_start":"524'10285","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11106","last_scrub_stamp":"2024-04-25T05:52:42.422923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.422923+0000","objects_scrubbed":4765,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.564049+0000","scrub_duration":1509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14948,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1316,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440790+0000","last_change":"2024-04-25T05:53:31.717402+0000","last_active":"2024-04-25T05:53:48.440790+0000","last_peered":"2024-04-25T05:53:48.440790+0000","last_clean":"2024-04-25T05:53:48.440790+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:53:48.440790+0000","last_undegraded":"2024-04-25T05:53:48.440790+0000","last_fullsized":"2024-04-25T05:53:48.440790+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:53:31.717254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.717254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.453072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":766,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.115361+0000","last_change":"2024-04-25T05:52:45.015305+0000","last_active":"2024-04-25T05:53:45.115361+0000","last_peered":"2024-04-25T05:53:45.115361+0000","last_clean":"2024-04-25T05:53:45.115361+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:53:45.115361+0000","last_undegraded":"2024-04-25T05:53:45.115361+0000","last_fullsized":"2024-04-25T05:53:45.115361+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:52:45.015213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.015213+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.513334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":161,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.446627+0000","last_change":"2024-04-25T05:51:40.129777+0000","last_active":"2024-04-25T05:53:48.446627+0000","last_peered":"2024-04-25T05:53:48.446627+0000","last_clean":"2024-04-25T05:53:48.446627+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:53:48.446627+0000","last_undegraded":"2024-04-25T05:53:48.446627+0000","last_fullsized":"2024-04-25T05:53:48.446627+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":456,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.445996+0000","last_change":"2024-04-25T05:53:34.695038+0000","last_active":"2024-04-25T05:53:48.445996+0000","last_peered":"2024-04-25T05:53:48.445996+0000","last_clean":"2024-04-25T05:53:48.445996+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:53:48.445996+0000","last_undegraded":"2024-04-25T05:53:48.445996+0000","last_fullsized":"2024-04-25T05:53:48.445996+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.694925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.694925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.211727+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1326,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436185+0000","last_change":"2024-04-25T05:53:21.777034+0000","last_active":"2024-04-25T05:53:48.436185+0000","last_peered":"2024-04-25T05:53:48.436185+0000","last_clean":"2024-04-25T05:53:48.436185+0000","last_became_active":"2024-04-25T05:52:51.531591+0000","last_became_peered":"2024-04-25T05:52:51.531591+0000","last_unstale":"2024-04-25T05:53:48.436185+0000","last_undegraded":"2024-04-25T05:53:48.436185+0000","last_fullsized":"2024-04-25T05:53:48.436185+0000","mapping_epoch":387,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:21.776901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.776901+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.945686+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"570'15304","reported_seq":15632,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.016279+0000","last_change":"2024-04-25T05:52:45.545440+0000","last_active":"2024-04-25T05:53:48.016279+0000","last_peered":"2024-04-25T05:53:48.016279+0000","last_clean":"2024-04-25T05:53:48.016279+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:53:48.016279+0000","last_undegraded":"2024-04-25T05:53:48.016279+0000","last_fullsized":"2024-04-25T05:53:48.016279+0000","mapping_epoch":456,"log_start":"506'9090","ondisk_log_start":"506'9090","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'11448","last_scrub_stamp":"2024-04-25T05:52:45.545041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.545041+0000","objects_scrubbed":4796,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.959830+0000","scrub_duration":1123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965464,"num_objects":916,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15304,"num_write_kb":518977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":187,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.444011+0000","last_change":"2024-04-25T05:51:40.124005+0000","last_active":"2024-04-25T05:53:48.444011+0000","last_peered":"2024-04-25T05:53:48.444011+0000","last_clean":"2024-04-25T05:53:48.444011+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:53:48.444011+0000","last_undegraded":"2024-04-25T05:53:48.444011+0000","last_fullsized":"2024-04-25T05:53:48.444011+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":816,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440332+0000","last_change":"2024-04-25T05:51:32.949273+0000","last_active":"2024-04-25T05:53:48.440332+0000","last_peered":"2024-04-25T05:53:48.440332+0000","last_clean":"2024-04-25T05:53:48.440332+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:53:48.440332+0000","last_undegraded":"2024-04-25T05:53:48.440332+0000","last_fullsized":"2024-04-25T05:53:48.440332+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:51:32.949135+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:51:32.949135+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":406,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122750+0000","last_change":"2024-04-25T05:52:41.064075+0000","last_active":"2024-04-25T05:53:45.122750+0000","last_peered":"2024-04-25T05:53:45.122750+0000","last_clean":"2024-04-25T05:53:45.122750+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:53:45.122750+0000","last_undegraded":"2024-04-25T05:53:45.122750+0000","last_fullsized":"2024-04-25T05:53:45.122750+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:41.063933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.063933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":158,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443731+0000","last_change":"2024-04-25T05:51:40.094093+0000","last_active":"2024-04-25T05:53:48.443731+0000","last_peered":"2024-04-25T05:53:48.443731+0000","last_clean":"2024-04-25T05:53:48.443731+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:53:48.443731+0000","last_undegraded":"2024-04-25T05:53:48.443731+0000","last_fullsized":"2024-04-25T05:53:48.443731+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:09.086479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:09.086479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1308,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.427200+0000","last_change":"2024-04-25T05:52:39.109897+0000","last_active":"2024-04-25T05:53:48.427200+0000","last_peered":"2024-04-25T05:53:48.427200+0000","last_clean":"2024-04-25T05:53:48.427200+0000","last_became_active":"2024-04-25T05:52:39.109573+0000","last_became_peered":"2024-04-25T05:52:39.109573+0000","last_unstale":"2024-04-25T05:53:48.427200+0000","last_undegraded":"2024-04-25T05:53:48.427200+0000","last_fullsized":"2024-04-25T05:53:48.427200+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_clean_scrub_stamp":"2024-04-25T05:51:33.347743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"571'15651","reported_seq":15942,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.458041+0000","last_change":"2024-04-25T05:52:52.331705+0000","last_active":"2024-04-25T05:53:49.458041+0000","last_peered":"2024-04-25T05:53:49.458041+0000","last_clean":"2024-04-25T05:53:49.458041+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:53:49.458041+0000","last_undegraded":"2024-04-25T05:53:49.458041+0000","last_fullsized":"2024-04-25T05:53:49.458041+0000","mapping_epoch":456,"log_start":"527'10866","ondisk_log_start":"527'10866","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12101","last_scrub_stamp":"2024-04-25T05:52:52.331512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.331512+0000","objects_scrubbed":4384,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.199070+0000","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52232192,"num_objects":797,"num_object_clones":0,"num_object_copies":2391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15651,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":841,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426226+0000","last_change":"2024-04-25T05:53:33.265611+0000","last_active":"2024-04-25T05:53:48.426226+0000","last_peered":"2024-04-25T05:53:48.426226+0000","last_clean":"2024-04-25T05:53:48.426226+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:53:48.426226+0000","last_undegraded":"2024-04-25T05:53:48.426226+0000","last_fullsized":"2024-04-25T05:53:48.426226+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.265484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.265484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:54.263173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":186,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443910+0000","last_change":"2024-04-25T05:53:24.028014+0000","last_active":"2024-04-25T05:53:48.443910+0000","last_peered":"2024-04-25T05:53:48.443910+0000","last_clean":"2024-04-25T05:53:48.443910+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:53:48.443910+0000","last_undegraded":"2024-04-25T05:53:48.443910+0000","last_fullsized":"2024-04-25T05:53:48.443910+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:53:24.027846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.027846+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:33.305120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1104,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442656+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:53:48.442656+0000","last_peered":"2024-04-25T05:53:48.442656+0000","last_clean":"2024-04-25T05:53:48.442656+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:53:48.442656+0000","last_undegraded":"2024-04-25T05:53:48.442656+0000","last_fullsized":"2024-04-25T05:53:48.442656+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444175+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1086,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440255+0000","last_change":"2024-04-25T05:52:47.618782+0000","last_active":"2024-04-25T05:53:48.440255+0000","last_peered":"2024-04-25T05:53:48.440255+0000","last_clean":"2024-04-25T05:53:48.440255+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:53:48.440255+0000","last_undegraded":"2024-04-25T05:53:48.440255+0000","last_fullsized":"2024-04-25T05:53:48.440255+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.618569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.618569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:17.510421+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1335,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426684+0000","last_change":"2024-04-25T05:52:43.436694+0000","last_active":"2024-04-25T05:53:48.426684+0000","last_peered":"2024-04-25T05:53:48.426684+0000","last_clean":"2024-04-25T05:53:48.426684+0000","last_became_active":"2024-04-25T05:52:43.436401+0000","last_became_peered":"2024-04-25T05:52:43.436401+0000","last_unstale":"2024-04-25T05:53:48.426684+0000","last_undegraded":"2024-04-25T05:53:48.426684+0000","last_fullsized":"2024-04-25T05:53:48.426684+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:37.366472+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:52:37.366472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"571'15667","reported_seq":24657,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.402985+0000","last_change":"2024-04-25T05:53:17.142210+0000","last_active":"2024-04-25T05:53:49.402985+0000","last_peered":"2024-04-25T05:53:49.402985+0000","last_clean":"2024-04-25T05:53:49.402985+0000","last_became_active":"2024-04-25T05:53:17.107904+0000","last_became_peered":"2024-04-25T05:53:17.107904+0000","last_unstale":"2024-04-25T05:53:49.402985+0000","last_undegraded":"2024-04-25T05:53:49.402985+0000","last_fullsized":"2024-04-25T05:53:49.402985+0000","mapping_epoch":557,"log_start":"531'11127","ondisk_log_start":"531'11127","created":456,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15667,"num_write_kb":529024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4367,"num_bytes_recovered":286195712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":412,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.123006+0000","last_change":"2024-04-25T05:52:53.143666+0000","last_active":"2024-04-25T05:53:45.123006+0000","last_peered":"2024-04-25T05:53:45.123006+0000","last_clean":"2024-04-25T05:53:45.123006+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:53:45.123006+0000","last_undegraded":"2024-04-25T05:53:45.123006+0000","last_fullsized":"2024-04-25T05:53:45.123006+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.143529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.143529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.394156+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":93,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.446137+0000","last_change":"2024-04-25T05:52:43.541002+0000","last_active":"2024-04-25T05:53:48.446137+0000","last_peered":"2024-04-25T05:53:48.446137+0000","last_clean":"2024-04-25T05:53:48.446137+0000","last_became_active":"2024-04-25T05:52:43.437539+0000","last_became_peered":"2024-04-25T05:52:43.437539+0000","last_unstale":"2024-04-25T05:53:48.446137+0000","last_undegraded":"2024-04-25T05:53:48.446137+0000","last_fullsized":"2024-04-25T05:53:48.446137+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:52:38.142064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.142064+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2306,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434770+0000","last_change":"2024-04-25T05:51:40.120019+0000","last_active":"2024-04-25T05:53:48.434770+0000","last_peered":"2024-04-25T05:53:48.434770+0000","last_clean":"2024-04-25T05:53:48.434770+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:53:48.434770+0000","last_undegraded":"2024-04-25T05:53:48.434770+0000","last_fullsized":"2024-04-25T05:53:48.434770+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:51:29.129304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.129304+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1170,"num_read_kb":1165,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":166,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.446288+0000","last_change":"2024-04-25T05:53:26.049870+0000","last_active":"2024-04-25T05:53:48.446288+0000","last_peered":"2024-04-25T05:53:48.446288+0000","last_clean":"2024-04-25T05:53:48.446288+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:53:48.446288+0000","last_undegraded":"2024-04-25T05:53:48.446288+0000","last_fullsized":"2024-04-25T05:53:48.446288+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:26.049619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.049619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:36.269598+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1074,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440417+0000","last_change":"2024-04-25T05:51:22.949696+0000","last_active":"2024-04-25T05:53:48.440417+0000","last_peered":"2024-04-25T05:53:48.440417+0000","last_clean":"2024-04-25T05:53:48.440417+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:53:48.440417+0000","last_undegraded":"2024-04-25T05:53:48.440417+0000","last_fullsized":"2024-04-25T05:53:48.440417+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.949432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.949432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444472+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1322,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.445886+0000","last_change":"2024-04-25T05:52:55.110481+0000","last_active":"2024-04-25T05:53:48.445886+0000","last_peered":"2024-04-25T05:53:48.445886+0000","last_clean":"2024-04-25T05:53:48.445886+0000","last_became_active":"2024-04-25T05:52:55.109763+0000","last_became_peered":"2024-04-25T05:52:55.109763+0000","last_unstale":"2024-04-25T05:53:48.445886+0000","last_undegraded":"2024-04-25T05:53:48.445886+0000","last_fullsized":"2024-04-25T05:53:48.445886+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.463489+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.463489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.844339+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"571'15209","reported_seq":15449,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.434601+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:53:49.434601+0000","last_peered":"2024-04-25T05:53:49.434601+0000","last_clean":"2024-04-25T05:53:49.434601+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:53:49.434601+0000","last_undegraded":"2024-04-25T05:53:49.434601+0000","last_fullsized":"2024-04-25T05:53:49.434601+0000","mapping_epoch":456,"log_start":"531'10679","ondisk_log_start":"531'10679","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15209,"num_write_kb":511296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1104,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434566+0000","last_change":"2024-04-25T05:53:29.647674+0000","last_active":"2024-04-25T05:53:48.434566+0000","last_peered":"2024-04-25T05:53:48.434566+0000","last_clean":"2024-04-25T05:53:48.434566+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:53:48.434566+0000","last_undegraded":"2024-04-25T05:53:48.434566+0000","last_fullsized":"2024-04-25T05:53:48.434566+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.647523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.647523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.191587+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":1012,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426940+0000","last_change":"2024-04-25T05:52:38.405034+0000","last_active":"2024-04-25T05:53:48.426940+0000","last_peered":"2024-04-25T05:53:48.426940+0000","last_clean":"2024-04-25T05:53:48.426940+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:53:48.426940+0000","last_undegraded":"2024-04-25T05:53:48.426940+0000","last_fullsized":"2024-04-25T05:53:48.426940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:52:38.404858+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.404858+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.254310+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1083,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.441285+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:53:48.441285+0000","last_peered":"2024-04-25T05:53:48.441285+0000","last_clean":"2024-04-25T05:53:48.441285+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:53:48.441285+0000","last_undegraded":"2024-04-25T05:53:48.441285+0000","last_fullsized":"2024-04-25T05:53:48.441285+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1448,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434843+0000","last_change":"2024-04-25T05:52:35.380122+0000","last_active":"2024-04-25T05:53:48.434843+0000","last_peered":"2024-04-25T05:53:48.434843+0000","last_clean":"2024-04-25T05:53:48.434843+0000","last_became_active":"2024-04-25T05:52:35.091072+0000","last_became_peered":"2024-04-25T05:52:35.091072+0000","last_unstale":"2024-04-25T05:53:48.434843+0000","last_undegraded":"2024-04-25T05:53:48.434843+0000","last_fullsized":"2024-04-25T05:53:48.434843+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":19,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:35.380044+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.380044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:49.993838+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":373,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.450736+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:53:48.450736+0000","last_peered":"2024-04-25T05:53:48.450736+0000","last_clean":"2024-04-25T05:53:48.450736+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:53:48.450736+0000","last_undegraded":"2024-04-25T05:53:48.450736+0000","last_fullsized":"2024-04-25T05:53:48.450736+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444256+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"571'15451","reported_seq":15692,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.423565+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:53:49.423565+0000","last_peered":"2024-04-25T05:53:49.423565+0000","last_clean":"2024-04-25T05:53:49.423565+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:53:49.423565+0000","last_undegraded":"2024-04-25T05:53:49.423565+0000","last_fullsized":"2024-04-25T05:53:49.423565+0000","mapping_epoch":456,"log_start":"533'10923","ondisk_log_start":"533'10923","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950784,"num_objects":869,"num_object_clones":0,"num_object_copies":2607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15451,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":429,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121109+0000","last_change":"2024-04-25T05:50:45.550588+0000","last_active":"2024-04-25T05:53:45.121109+0000","last_peered":"2024-04-25T05:53:45.121109+0000","last_clean":"2024-04-25T05:53:45.121109+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:53:45.121109+0000","last_undegraded":"2024-04-25T05:53:45.121109+0000","last_fullsized":"2024-04-25T05:53:45.121109+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:45.550474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.550474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.473928+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1233,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.450535+0000","last_change":"2024-04-25T05:51:40.157376+0000","last_active":"2024-04-25T05:53:48.450535+0000","last_peered":"2024-04-25T05:53:48.450535+0000","last_clean":"2024-04-25T05:53:48.450535+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:53:48.450535+0000","last_undegraded":"2024-04-25T05:53:48.450535+0000","last_fullsized":"2024-04-25T05:53:48.450535+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:50:40.134604+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.134604+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444143+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":437,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.119348+0000","last_change":"2024-04-25T05:53:21.420018+0000","last_active":"2024-04-25T05:53:45.119348+0000","last_peered":"2024-04-25T05:53:45.119348+0000","last_clean":"2024-04-25T05:53:45.119348+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:53:45.119348+0000","last_undegraded":"2024-04-25T05:53:45.119348+0000","last_fullsized":"2024-04-25T05:53:45.119348+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.419893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.419893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.568518+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1110,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440119+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:53:48.440119+0000","last_peered":"2024-04-25T05:53:48.440119+0000","last_clean":"2024-04-25T05:53:48.440119+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:53:48.440119+0000","last_undegraded":"2024-04-25T05:53:48.440119+0000","last_fullsized":"2024-04-25T05:53:48.440119+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444289+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1082,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.435680+0000","last_change":"2024-04-25T05:52:51.540421+0000","last_active":"2024-04-25T05:53:48.435680+0000","last_peered":"2024-04-25T05:53:48.435680+0000","last_clean":"2024-04-25T05:53:48.435680+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:53:48.435680+0000","last_undegraded":"2024-04-25T05:53:48.435680+0000","last_fullsized":"2024-04-25T05:53:48.435680+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.540285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:53.176425+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1456,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.433759+0000","last_change":"2024-04-25T05:52:34.412036+0000","last_active":"2024-04-25T05:53:48.433759+0000","last_peered":"2024-04-25T05:53:48.433759+0000","last_clean":"2024-04-25T05:53:48.433759+0000","last_became_active":"2024-04-25T05:52:27.138282+0000","last_became_peered":"2024-04-25T05:52:27.138282+0000","last_unstale":"2024-04-25T05:53:48.433759+0000","last_undegraded":"2024-04-25T05:53:48.433759+0000","last_fullsized":"2024-04-25T05:53:48.433759+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.411897+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.411897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.588807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"571'15483","reported_seq":23768,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:49.412013+0000","last_change":"2024-04-25T05:53:16.136131+0000","last_active":"2024-04-25T05:53:49.412013+0000","last_peered":"2024-04-25T05:53:49.412013+0000","last_clean":"2024-04-25T05:53:49.412013+0000","last_became_active":"2024-04-25T05:53:16.108052+0000","last_became_peered":"2024-04-25T05:53:16.108052+0000","last_unstale":"2024-04-25T05:53:49.412013+0000","last_undegraded":"2024-04-25T05:53:49.412013+0000","last_fullsized":"2024-04-25T05:53:49.412013+0000","mapping_epoch":556,"log_start":"530'10926","ondisk_log_start":"530'10926","created":456,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15483,"num_write_kb":521344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":158,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.444198+0000","last_change":"2024-04-25T05:51:40.098006+0000","last_active":"2024-04-25T05:53:48.444198+0000","last_peered":"2024-04-25T05:53:48.444198+0000","last_clean":"2024-04-25T05:53:48.444198+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:53:48.444198+0000","last_undegraded":"2024-04-25T05:53:48.444198+0000","last_fullsized":"2024-04-25T05:53:48.444198+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:21.946445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:21.946445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1459,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442803+0000","last_change":"2024-04-25T05:53:29.429046+0000","last_active":"2024-04-25T05:53:48.442803+0000","last_peered":"2024-04-25T05:53:48.442803+0000","last_clean":"2024-04-25T05:53:48.442803+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:53:48.442803+0000","last_undegraded":"2024-04-25T05:53:48.442803+0000","last_fullsized":"2024-04-25T05:53:48.442803+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:53:29.428902+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.428902+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.894655+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":647,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.113916+0000","last_change":"2024-04-25T05:51:50.680986+0000","last_active":"2024-04-25T05:53:45.113916+0000","last_peered":"2024-04-25T05:53:45.113916+0000","last_clean":"2024-04-25T05:53:45.113916+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:53:45.113916+0000","last_undegraded":"2024-04-25T05:53:45.113916+0000","last_fullsized":"2024-04-25T05:53:45.113916+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:50.680590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:51:50.680590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.033010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"506'3","reported_seq":411,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121573+0000","last_change":"2024-04-25T05:52:01.064448+0000","last_active":"2024-04-25T05:53:45.121573+0000","last_peered":"2024-04-25T05:53:45.121573+0000","last_clean":"2024-04-25T05:53:45.121573+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:53:45.121573+0000","last_undegraded":"2024-04-25T05:53:45.121573+0000","last_fullsized":"2024-04-25T05:53:45.121573+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'3","last_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_deep_scrub":"506'3","last_deep_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_clean_scrub_stamp":"2024-04-25T05:52:01.064292+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1047,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437164+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:53:48.437164+0000","last_peered":"2024-04-25T05:53:48.437164+0000","last_clean":"2024-04-25T05:53:48.437164+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:53:48.437164+0000","last_undegraded":"2024-04-25T05:53:48.437164+0000","last_fullsized":"2024-04-25T05:53:48.437164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":1155,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121068+0000","last_change":"2024-04-25T05:52:34.205086+0000","last_active":"2024-04-25T05:53:45.121068+0000","last_peered":"2024-04-25T05:53:45.121068+0000","last_clean":"2024-04-25T05:53:45.121068+0000","last_became_active":"2024-04-25T05:52:31.177637+0000","last_became_peered":"2024-04-25T05:52:31.177637+0000","last_unstale":"2024-04-25T05:53:45.121068+0000","last_undegraded":"2024-04-25T05:53:45.121068+0000","last_fullsized":"2024-04-25T05:53:45.121068+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.204967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.204967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.850061+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"571'15406","reported_seq":15728,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.462007+0000","last_change":"2024-04-25T05:52:03.230823+0000","last_active":"2024-04-25T05:53:48.462007+0000","last_peered":"2024-04-25T05:53:48.462007+0000","last_clean":"2024-04-25T05:53:48.462007+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:53:48.462007+0000","last_undegraded":"2024-04-25T05:53:48.462007+0000","last_fullsized":"2024-04-25T05:53:48.462007+0000","mapping_epoch":456,"log_start":"527'10690","ondisk_log_start":"527'10690","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8908","last_scrub_stamp":"2024-04-25T05:52:03.230624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:03.230624+0000","objects_scrubbed":7429,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57540608,"num_objects":878,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15406,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":169,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443660+0000","last_change":"2024-04-25T05:51:40.140728+0000","last_active":"2024-04-25T05:53:48.443660+0000","last_peered":"2024-04-25T05:53:48.443660+0000","last_clean":"2024-04-25T05:53:48.443660+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:53:48.443660+0000","last_undegraded":"2024-04-25T05:53:48.443660+0000","last_fullsized":"2024-04-25T05:53:48.443660+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":406,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121762+0000","last_change":"2024-04-25T05:52:52.129720+0000","last_active":"2024-04-25T05:53:45.121762+0000","last_peered":"2024-04-25T05:53:45.121762+0000","last_clean":"2024-04-25T05:53:45.121762+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:53:45.121762+0000","last_undegraded":"2024-04-25T05:53:45.121762+0000","last_fullsized":"2024-04-25T05:53:45.121762+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:52.129566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.129566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.024201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":423,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.119621+0000","last_change":"2024-04-25T05:51:28.554329+0000","last_active":"2024-04-25T05:53:45.119621+0000","last_peered":"2024-04-25T05:53:45.119621+0000","last_clean":"2024-04-25T05:53:45.119621+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:53:45.119621+0000","last_undegraded":"2024-04-25T05:53:45.119621+0000","last_fullsized":"2024-04-25T05:53:45.119621+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.554087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.554087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.473960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.444356+0000","last_change":"2024-04-25T05:51:40.099599+0000","last_active":"2024-04-25T05:53:48.444356+0000","last_peered":"2024-04-25T05:53:48.444356+0000","last_clean":"2024-04-25T05:53:48.444356+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:53:48.444356+0000","last_undegraded":"2024-04-25T05:53:48.444356+0000","last_fullsized":"2024-04-25T05:53:48.444356+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.026628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.026628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"571'15718","reported_seq":16021,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.457056+0000","last_change":"2024-04-25T05:52:38.587166+0000","last_active":"2024-04-25T05:53:48.457056+0000","last_peered":"2024-04-25T05:53:48.457056+0000","last_clean":"2024-04-25T05:53:48.457056+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:53:48.457056+0000","last_undegraded":"2024-04-25T05:53:48.457056+0000","last_fullsized":"2024-04-25T05:53:48.457056+0000","mapping_epoch":456,"log_start":"527'10931","ondisk_log_start":"527'10931","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'11363","last_scrub_stamp":"2024-04-25T05:52:38.587034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.587034+0000","objects_scrubbed":5307,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.588841+0000","scrub_duration":1165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60555264,"num_objects":924,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15718,"num_write_kb":532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":861,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426041+0000","last_change":"2024-04-25T05:52:39.432701+0000","last_active":"2024-04-25T05:53:48.426041+0000","last_peered":"2024-04-25T05:53:48.426041+0000","last_clean":"2024-04-25T05:53:48.426041+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:53:48.426041+0000","last_undegraded":"2024-04-25T05:53:48.426041+0000","last_fullsized":"2024-04-25T05:53:48.426041+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:52:39.432428+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.432428+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:58.290428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":1128,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.435494+0000","last_change":"2024-04-25T05:52:41.458972+0000","last_active":"2024-04-25T05:53:48.435494+0000","last_peered":"2024-04-25T05:53:48.435494+0000","last_clean":"2024-04-25T05:53:48.435494+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:53:48.435494+0000","last_undegraded":"2024-04-25T05:53:48.435494+0000","last_fullsized":"2024-04-25T05:53:48.435494+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:52:41.458853+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.458853+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.006386+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":435,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.119714+0000","last_change":"2024-04-25T05:51:40.105669+0000","last_active":"2024-04-25T05:53:45.119714+0000","last_peered":"2024-04-25T05:53:45.119714+0000","last_clean":"2024-04-25T05:53:45.119714+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:53:45.119714+0000","last_undegraded":"2024-04-25T05:53:45.119714+0000","last_fullsized":"2024-04-25T05:53:45.119714+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:31.543423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:31.543423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.473991+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1082,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434678+0000","last_change":"2024-04-25T05:53:25.688414+0000","last_active":"2024-04-25T05:53:48.434678+0000","last_peered":"2024-04-25T05:53:48.434678+0000","last_clean":"2024-04-25T05:53:48.434678+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:53:48.434678+0000","last_undegraded":"2024-04-25T05:53:48.434678+0000","last_fullsized":"2024-04-25T05:53:48.434678+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.688336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.688336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.070699+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"571'15147","reported_seq":28127,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.622799+0000","last_change":"2024-04-25T05:52:45.137759+0000","last_active":"2024-04-25T05:53:48.622799+0000","last_peered":"2024-04-25T05:53:48.622799+0000","last_clean":"2024-04-25T05:53:48.622799+0000","last_became_active":"2024-04-25T05:52:45.095486+0000","last_became_peered":"2024-04-25T05:52:45.095486+0000","last_unstale":"2024-04-25T05:53:48.622799+0000","last_undegraded":"2024-04-25T05:53:48.622799+0000","last_fullsized":"2024-04-25T05:53:48.622799+0000","mapping_epoch":541,"log_start":"513'9658","ondisk_log_start":"513'9658","created":456,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":2643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15147,"num_write_kb":512896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6364,"num_bytes_recovered":417071104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":443,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.118964+0000","last_change":"2024-04-25T05:53:28.416788+0000","last_active":"2024-04-25T05:53:45.118964+0000","last_peered":"2024-04-25T05:53:45.118964+0000","last_clean":"2024-04-25T05:53:45.118964+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:53:45.118964+0000","last_undegraded":"2024-04-25T05:53:45.118964+0000","last_fullsized":"2024-04-25T05:53:45.118964+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.416661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.416661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.932624+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":496,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.115540+0000","last_change":"2024-04-25T05:53:31.936255+0000","last_active":"2024-04-25T05:53:45.115540+0000","last_peered":"2024-04-25T05:53:45.115540+0000","last_clean":"2024-04-25T05:53:45.115540+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:53:45.115540+0000","last_undegraded":"2024-04-25T05:53:45.115540+0000","last_fullsized":"2024-04-25T05:53:45.115540+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:53:31.936104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.936104+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:44.183145+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1081,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434466+0000","last_change":"2024-04-25T05:52:39.438835+0000","last_active":"2024-04-25T05:53:48.434466+0000","last_peered":"2024-04-25T05:53:48.434466+0000","last_clean":"2024-04-25T05:53:48.434466+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:53:48.434466+0000","last_undegraded":"2024-04-25T05:53:48.434466+0000","last_fullsized":"2024-04-25T05:53:48.434466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:39.438700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.438700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.055139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1058,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.441932+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:53:48.441932+0000","last_peered":"2024-04-25T05:53:48.441932+0000","last_clean":"2024-04-25T05:53:48.441932+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:53:48.441932+0000","last_undegraded":"2024-04-25T05:53:48.441932+0000","last_fullsized":"2024-04-25T05:53:48.441932+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"570'15286","reported_seq":15606,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.222689+0000","last_change":"2024-04-25T05:52:02.016861+0000","last_active":"2024-04-25T05:53:48.222689+0000","last_peered":"2024-04-25T05:53:48.222689+0000","last_clean":"2024-04-25T05:53:48.222689+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:53:48.222689+0000","last_undegraded":"2024-04-25T05:53:48.222689+0000","last_fullsized":"2024-04-25T05:53:48.222689+0000","mapping_epoch":456,"log_start":"514'9910","ondisk_log_start":"514'9910","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8841","last_scrub_stamp":"2024-04-25T05:52:02.016648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.016648+0000","objects_scrubbed":7398,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:50.980909+0000","scrub_duration":1997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15286,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1347,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.433483+0000","last_change":"2024-04-25T05:52:50.387148+0000","last_active":"2024-04-25T05:53:48.433483+0000","last_peered":"2024-04-25T05:53:48.433483+0000","last_clean":"2024-04-25T05:53:48.433483+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:53:48.433483+0000","last_undegraded":"2024-04-25T05:53:48.433483+0000","last_fullsized":"2024-04-25T05:53:48.433483+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_deep_scrub":"208'11","last_deep_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.387023+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.717658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":426,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122138+0000","last_change":"2024-04-25T05:52:06.018611+0000","last_active":"2024-04-25T05:53:45.122138+0000","last_peered":"2024-04-25T05:53:45.122138+0000","last_clean":"2024-04-25T05:53:45.122138+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:53:45.122138+0000","last_undegraded":"2024-04-25T05:53:45.122138+0000","last_fullsized":"2024-04-25T05:53:45.122138+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:52:06.018248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:06.018248+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1141,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.435300+0000","last_change":"2024-04-25T05:53:33.672727+0000","last_active":"2024-04-25T05:53:48.435300+0000","last_peered":"2024-04-25T05:53:48.435300+0000","last_clean":"2024-04-25T05:53:48.435300+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:53:48.435300+0000","last_undegraded":"2024-04-25T05:53:48.435300+0000","last_fullsized":"2024-04-25T05:53:48.435300+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:53:33.672600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.672600+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:51.282306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":404,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121669+0000","last_change":"2024-04-25T05:51:56.071814+0000","last_active":"2024-04-25T05:53:45.121669+0000","last_peered":"2024-04-25T05:53:45.121669+0000","last_clean":"2024-04-25T05:53:45.121669+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:53:45.121669+0000","last_undegraded":"2024-04-25T05:53:45.121669+0000","last_fullsized":"2024-04-25T05:53:45.121669+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:56.071706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:56.071706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":1092,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437578+0000","last_change":"2024-04-25T05:51:28.373915+0000","last_active":"2024-04-25T05:53:48.437578+0000","last_peered":"2024-04-25T05:53:48.437578+0000","last_clean":"2024-04-25T05:53:48.437578+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:53:48.437578+0000","last_undegraded":"2024-04-25T05:53:48.437578+0000","last_fullsized":"2024-04-25T05:53:48.437578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:51:28.373733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.373733+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":841,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.425349+0000","last_change":"2024-04-25T05:52:50.354986+0000","last_active":"2024-04-25T05:53:48.425349+0000","last_peered":"2024-04-25T05:53:48.425349+0000","last_clean":"2024-04-25T05:53:48.425349+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:53:48.425349+0000","last_undegraded":"2024-04-25T05:53:48.425349+0000","last_fullsized":"2024-04-25T05:53:48.425349+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:50.354868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.354868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.086443+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":405,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122591+0000","last_change":"2024-04-25T05:52:44.084490+0000","last_active":"2024-04-25T05:53:45.122591+0000","last_peered":"2024-04-25T05:53:45.122591+0000","last_clean":"2024-04-25T05:53:45.122591+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:53:45.122591+0000","last_undegraded":"2024-04-25T05:53:45.122591+0000","last_fullsized":"2024-04-25T05:53:45.122591+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.084411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.084411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.972942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1081,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434725+0000","last_change":"2024-04-25T05:51:48.376273+0000","last_active":"2024-04-25T05:53:48.434725+0000","last_peered":"2024-04-25T05:53:48.434725+0000","last_clean":"2024-04-25T05:53:48.434725+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:53:48.434725+0000","last_undegraded":"2024-04-25T05:53:48.434725+0000","last_fullsized":"2024-04-25T05:53:48.434725+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:48.375877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:48.375877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":179,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.447957+0000","last_change":"2024-04-25T05:53:32.185728+0000","last_active":"2024-04-25T05:53:48.447957+0000","last_peered":"2024-04-25T05:53:48.447957+0000","last_clean":"2024-04-25T05:53:48.447957+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:53:48.447957+0000","last_undegraded":"2024-04-25T05:53:48.447957+0000","last_fullsized":"2024-04-25T05:53:48.447957+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:53:32.185584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:32.185584+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.846897+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1099,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.433748+0000","last_change":"2024-04-25T05:52:44.356015+0000","last_active":"2024-04-25T05:53:48.433748+0000","last_peered":"2024-04-25T05:53:48.433748+0000","last_clean":"2024-04-25T05:53:48.433748+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:53:48.433748+0000","last_undegraded":"2024-04-25T05:53:48.433748+0000","last_fullsized":"2024-04-25T05:53:48.433748+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.355921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.355921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.000445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1050,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437369+0000","last_change":"2024-04-25T05:51:30.313765+0000","last_active":"2024-04-25T05:53:48.437369+0000","last_peered":"2024-04-25T05:53:48.437369+0000","last_clean":"2024-04-25T05:53:48.437369+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:53:48.437369+0000","last_undegraded":"2024-04-25T05:53:48.437369+0000","last_fullsized":"2024-04-25T05:53:48.437369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.313655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.313655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":423,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.120669+0000","last_change":"2024-04-25T05:53:23.378618+0000","last_active":"2024-04-25T05:53:45.120669+0000","last_peered":"2024-04-25T05:53:45.120669+0000","last_clean":"2024-04-25T05:53:45.120669+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:53:45.120669+0000","last_undegraded":"2024-04-25T05:53:45.120669+0000","last_fullsized":"2024-04-25T05:53:45.120669+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.378498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.378498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.114889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":417,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122280+0000","last_change":"2024-04-25T05:53:26.982110+0000","last_active":"2024-04-25T05:53:45.122280+0000","last_peered":"2024-04-25T05:53:45.122280+0000","last_clean":"2024-04-25T05:53:45.122280+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:53:45.122280+0000","last_undegraded":"2024-04-25T05:53:45.122280+0000","last_fullsized":"2024-04-25T05:53:45.122280+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:26.981810+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.981810+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.388901+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":165,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443610+0000","last_change":"2024-04-25T05:53:22.083599+0000","last_active":"2024-04-25T05:53:48.443610+0000","last_peered":"2024-04-25T05:53:48.443610+0000","last_clean":"2024-04-25T05:53:48.443610+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:53:48.443610+0000","last_undegraded":"2024-04-25T05:53:48.443610+0000","last_fullsized":"2024-04-25T05:53:48.443610+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:22.083459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.083459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.639562+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1317,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436792+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:53:48.436792+0000","last_peered":"2024-04-25T05:53:48.436792+0000","last_clean":"2024-04-25T05:53:48.436792+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:53:48.436792+0000","last_undegraded":"2024-04-25T05:53:48.436792+0000","last_fullsized":"2024-04-25T05:53:48.436792+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":618,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.117982+0000","last_change":"2024-04-25T05:52:46.212477+0000","last_active":"2024-04-25T05:53:45.117982+0000","last_peered":"2024-04-25T05:53:45.117982+0000","last_clean":"2024-04-25T05:53:45.117982+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:53:45.117982+0000","last_undegraded":"2024-04-25T05:53:45.117982+0000","last_fullsized":"2024-04-25T05:53:45.117982+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.212334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.212334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:15.795169+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":1116,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436446+0000","last_change":"2024-04-25T05:53:25.647571+0000","last_active":"2024-04-25T05:53:48.436446+0000","last_peered":"2024-04-25T05:53:48.436446+0000","last_clean":"2024-04-25T05:53:48.436446+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:53:48.436446+0000","last_undegraded":"2024-04-25T05:53:48.436446+0000","last_fullsized":"2024-04-25T05:53:48.436446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:53:25.647413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.647413+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.576268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1093,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434099+0000","last_change":"2024-04-25T05:53:30.687157+0000","last_active":"2024-04-25T05:53:48.434099+0000","last_peered":"2024-04-25T05:53:48.434099+0000","last_clean":"2024-04-25T05:53:48.434099+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:53:48.434099+0000","last_undegraded":"2024-04-25T05:53:48.434099+0000","last_fullsized":"2024-04-25T05:53:48.434099+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.687046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.687046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.184397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":497,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122553+0000","last_change":"2024-04-25T05:51:53.967723+0000","last_active":"2024-04-25T05:53:45.122553+0000","last_peered":"2024-04-25T05:53:45.122553+0000","last_clean":"2024-04-25T05:53:45.122553+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:53:45.122553+0000","last_undegraded":"2024-04-25T05:53:45.122553+0000","last_fullsized":"2024-04-25T05:53:45.122553+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'14","last_scrub_stamp":"2024-04-25T05:51:53.967616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:53.967616+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.448293+0000","last_change":"2024-04-25T05:51:40.110100+0000","last_active":"2024-04-25T05:53:48.448293+0000","last_peered":"2024-04-25T05:53:48.448293+0000","last_clean":"2024-04-25T05:53:48.448293+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:53:48.448293+0000","last_undegraded":"2024-04-25T05:53:48.448293+0000","last_fullsized":"2024-04-25T05:53:48.448293+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.972698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.972698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1218,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114252+0000","last_change":"2024-04-25T05:53:21.989605+0000","last_active":"2024-04-25T05:53:45.114252+0000","last_peered":"2024-04-25T05:53:45.114252+0000","last_clean":"2024-04-25T05:53:45.114252+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:53:45.114252+0000","last_undegraded":"2024-04-25T05:53:45.114252+0000","last_fullsized":"2024-04-25T05:53:45.114252+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:53:21.989507+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.989507+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.602540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":664,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.117258+0000","last_change":"2024-04-25T05:52:05.839366+0000","last_active":"2024-04-25T05:53:45.117258+0000","last_peered":"2024-04-25T05:53:45.117258+0000","last_clean":"2024-04-25T05:53:45.117258+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:53:45.117258+0000","last_undegraded":"2024-04-25T05:53:45.117258+0000","last_fullsized":"2024-04-25T05:53:45.117258+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:52:05.839241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:05.839241+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.033028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1080,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440214+0000","last_change":"2024-04-25T05:53:25.341898+0000","last_active":"2024-04-25T05:53:48.440214+0000","last_peered":"2024-04-25T05:53:48.440214+0000","last_clean":"2024-04-25T05:53:48.440214+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:53:48.440214+0000","last_undegraded":"2024-04-25T05:53:48.440214+0000","last_fullsized":"2024-04-25T05:53:48.440214+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.334571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.334571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.518338+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1096,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436253+0000","last_change":"2024-04-25T05:51:40.083901+0000","last_active":"2024-04-25T05:53:48.436253+0000","last_peered":"2024-04-25T05:53:48.436253+0000","last_clean":"2024-04-25T05:53:48.436253+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:53:48.436253+0000","last_undegraded":"2024-04-25T05:53:48.436253+0000","last_fullsized":"2024-04-25T05:53:48.436253+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":627,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.445186+0000","last_change":"2024-04-25T05:51:23.925876+0000","last_active":"2024-04-25T05:53:48.445186+0000","last_peered":"2024-04-25T05:53:48.445186+0000","last_clean":"2024-04-25T05:53:48.445186+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:53:48.445186+0000","last_undegraded":"2024-04-25T05:53:48.445186+0000","last_fullsized":"2024-04-25T05:53:48.445186+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:51:23.925746+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.925746+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1081,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434839+0000","last_change":"2024-04-25T05:52:45.364101+0000","last_active":"2024-04-25T05:53:48.434839+0000","last_peered":"2024-04-25T05:53:48.434839+0000","last_clean":"2024-04-25T05:53:48.434839+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:53:48.434839+0000","last_undegraded":"2024-04-25T05:53:48.434839+0000","last_fullsized":"2024-04-25T05:53:48.434839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:45.363990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.363990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.943288+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1080,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437454+0000","last_change":"2024-04-25T05:51:40.102243+0000","last_active":"2024-04-25T05:53:48.437454+0000","last_peered":"2024-04-25T05:53:48.437454+0000","last_clean":"2024-04-25T05:53:48.437454+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:53:48.437454+0000","last_undegraded":"2024-04-25T05:53:48.437454+0000","last_fullsized":"2024-04-25T05:53:48.437454+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443642+0000","last_change":"2024-04-25T05:51:40.094829+0000","last_active":"2024-04-25T05:53:48.443642+0000","last_peered":"2024-04-25T05:53:48.443642+0000","last_clean":"2024-04-25T05:53:48.443642+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:53:48.443642+0000","last_undegraded":"2024-04-25T05:53:48.443642+0000","last_fullsized":"2024-04-25T05:53:48.443642+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:02.734157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:02.734157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":1151,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436697+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:53:48.436697+0000","last_peered":"2024-04-25T05:53:48.436697+0000","last_clean":"2024-04-25T05:53:48.436697+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:53:48.436697+0000","last_undegraded":"2024-04-25T05:53:48.436697+0000","last_fullsized":"2024-04-25T05:53:48.436697+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":405,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122425+0000","last_change":"2024-04-25T05:52:42.094827+0000","last_active":"2024-04-25T05:53:45.122425+0000","last_peered":"2024-04-25T05:53:45.122425+0000","last_clean":"2024-04-25T05:53:45.122425+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:53:45.122425+0000","last_undegraded":"2024-04-25T05:53:45.122425+0000","last_fullsized":"2024-04-25T05:53:45.122425+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.094672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.094672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.738692+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1091,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.435053+0000","last_change":"2024-04-25T05:52:53.417453+0000","last_active":"2024-04-25T05:53:48.435053+0000","last_peered":"2024-04-25T05:53:48.435053+0000","last_clean":"2024-04-25T05:53:48.435053+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:53:48.435053+0000","last_undegraded":"2024-04-25T05:53:48.435053+0000","last_fullsized":"2024-04-25T05:53:48.435053+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.417346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.417346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.686711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1088,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442029+0000","last_change":"2024-04-25T05:51:40.104333+0000","last_active":"2024-04-25T05:53:48.442029+0000","last_peered":"2024-04-25T05:53:48.442029+0000","last_clean":"2024-04-25T05:53:48.442029+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:53:48.442029+0000","last_undegraded":"2024-04-25T05:53:48.442029+0000","last_fullsized":"2024-04-25T05:53:48.442029+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":1090,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.439487+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:53:48.439487+0000","last_peered":"2024-04-25T05:53:48.439487+0000","last_clean":"2024-04-25T05:53:48.439487+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:53:48.439487+0000","last_undegraded":"2024-04-25T05:53:48.439487+0000","last_fullsized":"2024-04-25T05:53:48.439487+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.444481+0000","last_change":"2024-04-25T05:51:40.098114+0000","last_active":"2024-04-25T05:53:48.444481+0000","last_peered":"2024-04-25T05:53:48.444481+0000","last_clean":"2024-04-25T05:53:48.444481+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:53:48.444481+0000","last_undegraded":"2024-04-25T05:53:48.444481+0000","last_fullsized":"2024-04-25T05:53:48.444481+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:07.763810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:07.763810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":797,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426650+0000","last_change":"2024-04-25T05:52:44.398454+0000","last_active":"2024-04-25T05:53:48.426650+0000","last_peered":"2024-04-25T05:53:48.426650+0000","last_clean":"2024-04-25T05:53:48.426650+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:53:48.426650+0000","last_undegraded":"2024-04-25T05:53:48.426650+0000","last_fullsized":"2024-04-25T05:53:48.426650+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.398379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.398379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.323065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":1082,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.433858+0000","last_change":"2024-04-25T05:51:57.236443+0000","last_active":"2024-04-25T05:53:48.433858+0000","last_peered":"2024-04-25T05:53:48.433858+0000","last_clean":"2024-04-25T05:53:48.433858+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:53:48.433858+0000","last_undegraded":"2024-04-25T05:53:48.433858+0000","last_fullsized":"2024-04-25T05:53:48.433858+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:51:57.236311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:57.236311+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":917,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.427351+0000","last_change":"2024-04-25T05:53:22.290678+0000","last_active":"2024-04-25T05:53:48.427351+0000","last_peered":"2024-04-25T05:53:48.427351+0000","last_clean":"2024-04-25T05:53:48.427351+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:53:48.427351+0000","last_undegraded":"2024-04-25T05:53:48.427351+0000","last_fullsized":"2024-04-25T05:53:48.427351+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:53:22.290465+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.290465+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.697555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1096,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.439289+0000","last_change":"2024-04-25T05:53:28.655967+0000","last_active":"2024-04-25T05:53:48.439289+0000","last_peered":"2024-04-25T05:53:48.439289+0000","last_clean":"2024-04-25T05:53:48.439289+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:53:48.439289+0000","last_undegraded":"2024-04-25T05:53:48.439289+0000","last_fullsized":"2024-04-25T05:53:48.439289+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.655843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.655843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.413792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":166,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.446371+0000","last_change":"2024-04-25T05:53:34.135232+0000","last_active":"2024-04-25T05:53:48.446371+0000","last_peered":"2024-04-25T05:53:48.446371+0000","last_clean":"2024-04-25T05:53:48.446371+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:53:48.446371+0000","last_undegraded":"2024-04-25T05:53:48.446371+0000","last_fullsized":"2024-04-25T05:53:48.446371+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.134907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.134907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.959435+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1070,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442669+0000","last_change":"2024-04-25T05:50:17.140640+0000","last_active":"2024-04-25T05:53:48.442669+0000","last_peered":"2024-04-25T05:53:48.442669+0000","last_clean":"2024-04-25T05:53:48.442669+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:53:48.442669+0000","last_undegraded":"2024-04-25T05:53:48.442669+0000","last_fullsized":"2024-04-25T05:53:48.442669+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:50:17.140516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444105+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":172,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.448188+0000","last_change":"2024-04-25T05:51:40.147090+0000","last_active":"2024-04-25T05:53:48.448188+0000","last_peered":"2024-04-25T05:53:48.448188+0000","last_clean":"2024-04-25T05:53:48.448188+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:53:48.448188+0000","last_undegraded":"2024-04-25T05:53:48.448188+0000","last_fullsized":"2024-04-25T05:53:48.448188+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1074,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440058+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:53:48.440058+0000","last_peered":"2024-04-25T05:53:48.440058+0000","last_clean":"2024-04-25T05:53:48.440058+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:53:48.440058+0000","last_undegraded":"2024-04-25T05:53:48.440058+0000","last_fullsized":"2024-04-25T05:53:48.440058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444224+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":980,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.427191+0000","last_change":"2024-04-25T05:52:53.341410+0000","last_active":"2024-04-25T05:53:48.427191+0000","last_peered":"2024-04-25T05:53:48.427191+0000","last_clean":"2024-04-25T05:53:48.427191+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:53:48.427191+0000","last_undegraded":"2024-04-25T05:53:48.427191+0000","last_fullsized":"2024-04-25T05:53:48.427191+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:52:53.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.341283+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:03.764198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"507'2","reported_seq":1110,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.434456+0000","last_change":"2024-04-25T05:52:08.192825+0000","last_active":"2024-04-25T05:53:48.434456+0000","last_peered":"2024-04-25T05:53:48.434456+0000","last_clean":"2024-04-25T05:53:48.434456+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:53:48.434456+0000","last_undegraded":"2024-04-25T05:53:48.434456+0000","last_fullsized":"2024-04-25T05:53:48.434456+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.192693+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":1097,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436690+0000","last_change":"2024-04-25T05:51:18.436527+0000","last_active":"2024-04-25T05:53:48.436690+0000","last_peered":"2024-04-25T05:53:48.436690+0000","last_clean":"2024-04-25T05:53:48.436690+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:53:48.436690+0000","last_undegraded":"2024-04-25T05:53:48.436690+0000","last_fullsized":"2024-04-25T05:53:48.436690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:18.436370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.436370+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1113,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.115116+0000","last_change":"2024-04-25T05:53:24.969033+0000","last_active":"2024-04-25T05:53:45.115116+0000","last_peered":"2024-04-25T05:53:45.115116+0000","last_clean":"2024-04-25T05:53:45.115116+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:53:45.115116+0000","last_undegraded":"2024-04-25T05:53:45.115116+0000","last_fullsized":"2024-04-25T05:53:45.115116+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:24.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:46.507723+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":165,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.426391+0000","last_change":"2024-04-25T05:52:35.374907+0000","last_active":"2024-04-25T05:53:48.426391+0000","last_peered":"2024-04-25T05:53:48.426391+0000","last_clean":"2024-04-25T05:53:48.426391+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:53:48.426391+0000","last_undegraded":"2024-04-25T05:53:48.426391+0000","last_fullsized":"2024-04-25T05:53:48.426391+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:35.374829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.374829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1071,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437454+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:53:48.437454+0000","last_peered":"2024-04-25T05:53:48.437454+0000","last_clean":"2024-04-25T05:53:48.437454+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:53:48.437454+0000","last_undegraded":"2024-04-25T05:53:48.437454+0000","last_fullsized":"2024-04-25T05:53:48.437454+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":446,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.120515+0000","last_change":"2024-04-25T05:53:25.409053+0000","last_active":"2024-04-25T05:53:45.120515+0000","last_peered":"2024-04-25T05:53:45.120515+0000","last_clean":"2024-04-25T05:53:45.120515+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:53:45.120515+0000","last_undegraded":"2024-04-25T05:53:45.120515+0000","last_fullsized":"2024-04-25T05:53:45.120515+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:25.408390+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.408390+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:25.958321+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443757+0000","last_change":"2024-04-25T05:51:40.101375+0000","last_active":"2024-04-25T05:53:48.443757+0000","last_peered":"2024-04-25T05:53:48.443757+0000","last_clean":"2024-04-25T05:53:48.443757+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:53:48.443757+0000","last_undegraded":"2024-04-25T05:53:48.443757+0000","last_fullsized":"2024-04-25T05:53:48.443757+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.835919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.835919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1094,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.442907+0000","last_change":"2024-04-25T05:51:40.109523+0000","last_active":"2024-04-25T05:53:48.442907+0000","last_peered":"2024-04-25T05:53:48.442907+0000","last_clean":"2024-04-25T05:53:48.442907+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:53:48.442907+0000","last_undegraded":"2024-04-25T05:53:48.442907+0000","last_fullsized":"2024-04-25T05:53:48.442907+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.989325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.989325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444428+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1109,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.116392+0000","last_change":"2024-04-25T05:53:23.056922+0000","last_active":"2024-04-25T05:53:45.116392+0000","last_peered":"2024-04-25T05:53:45.116392+0000","last_clean":"2024-04-25T05:53:45.116392+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:53:45.116392+0000","last_undegraded":"2024-04-25T05:53:45.116392+0000","last_fullsized":"2024-04-25T05:53:45.116392+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.056816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.056816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.493056+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":579,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114116+0000","last_change":"2024-04-25T05:51:56.835740+0000","last_active":"2024-04-25T05:53:45.114116+0000","last_peered":"2024-04-25T05:53:45.114116+0000","last_clean":"2024-04-25T05:53:45.114116+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:53:45.114116+0000","last_undegraded":"2024-04-25T05:53:45.114116+0000","last_fullsized":"2024-04-25T05:53:45.114116+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:51:56.835603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:56.835603+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.032961+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":405,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121896+0000","last_change":"2024-04-25T05:52:47.057845+0000","last_active":"2024-04-25T05:53:45.121896+0000","last_peered":"2024-04-25T05:53:45.121896+0000","last_clean":"2024-04-25T05:53:45.121896+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:53:45.121896+0000","last_undegraded":"2024-04-25T05:53:45.121896+0000","last_fullsized":"2024-04-25T05:53:45.121896+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.057728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.968343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1070,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.433671+0000","last_change":"2024-04-25T05:51:54.167227+0000","last_active":"2024-04-25T05:53:48.433671+0000","last_peered":"2024-04-25T05:53:48.433671+0000","last_clean":"2024-04-25T05:53:48.433671+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:53:48.433671+0000","last_undegraded":"2024-04-25T05:53:48.433671+0000","last_fullsized":"2024-04-25T05:53:48.433671+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:54.167105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:54.167105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":405,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121177+0000","last_change":"2024-04-25T05:53:30.164095+0000","last_active":"2024-04-25T05:53:45.121177+0000","last_peered":"2024-04-25T05:53:45.121177+0000","last_clean":"2024-04-25T05:53:45.121177+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:53:45.121177+0000","last_undegraded":"2024-04-25T05:53:45.121177+0000","last_fullsized":"2024-04-25T05:53:45.121177+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.163971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.163971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.864542+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":893,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.115029+0000","last_change":"2024-04-25T05:52:02.822759+0000","last_active":"2024-04-25T05:53:45.115029+0000","last_peered":"2024-04-25T05:53:45.115029+0000","last_clean":"2024-04-25T05:53:45.115029+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:53:45.115029+0000","last_undegraded":"2024-04-25T05:53:45.115029+0000","last_fullsized":"2024-04-25T05:53:45.115029+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:52:02.822618+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.822618+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.033045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":405,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.122400+0000","last_change":"2024-04-25T05:52:46.070334+0000","last_active":"2024-04-25T05:53:45.122400+0000","last_peered":"2024-04-25T05:53:45.122400+0000","last_clean":"2024-04-25T05:53:45.122400+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:53:45.122400+0000","last_undegraded":"2024-04-25T05:53:45.122400+0000","last_fullsized":"2024-04-25T05:53:45.122400+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.070183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.070183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.824200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":403,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.121191+0000","last_change":"2024-04-25T05:51:40.084285+0000","last_active":"2024-04-25T05:53:45.121191+0000","last_peered":"2024-04-25T05:53:45.121191+0000","last_clean":"2024-04-25T05:53:45.121191+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:53:45.121191+0000","last_undegraded":"2024-04-25T05:53:45.121191+0000","last_fullsized":"2024-04-25T05:53:45.121191+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.941496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.941496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":157,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.443538+0000","last_change":"2024-04-25T05:51:40.110384+0000","last_active":"2024-04-25T05:53:48.443538+0000","last_peered":"2024-04-25T05:53:48.443538+0000","last_clean":"2024-04-25T05:53:48.443538+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:53:48.443538+0000","last_undegraded":"2024-04-25T05:53:48.443538+0000","last_fullsized":"2024-04-25T05:53:48.443538+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1385,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436792+0000","last_change":"2024-04-25T05:51:40.151139+0000","last_active":"2024-04-25T05:53:48.436792+0000","last_peered":"2024-04-25T05:53:48.436792+0000","last_clean":"2024-04-25T05:53:48.436792+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:53:48.436792+0000","last_undegraded":"2024-04-25T05:53:48.436792+0000","last_fullsized":"2024-04-25T05:53:48.436792+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1091,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.436930+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:53:48.436930+0000","last_peered":"2024-04-25T05:53:48.436930+0000","last_clean":"2024-04-25T05:53:48.436930+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:53:48.436930+0000","last_undegraded":"2024-04-25T05:53:48.436930+0000","last_fullsized":"2024-04-25T05:53:48.436930+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":634,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114170+0000","last_change":"2024-04-25T05:52:51.042784+0000","last_active":"2024-04-25T05:53:45.114170+0000","last_peered":"2024-04-25T05:53:45.114170+0000","last_clean":"2024-04-25T05:53:45.114170+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:53:45.114170+0000","last_undegraded":"2024-04-25T05:53:45.114170+0000","last_fullsized":"2024-04-25T05:53:45.114170+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.042665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.042665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:53.131222+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1057,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.437545+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:53:48.437545+0000","last_peered":"2024-04-25T05:53:48.437545+0000","last_clean":"2024-04-25T05:53:48.437545+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:53:48.437545+0000","last_undegraded":"2024-04-25T05:53:48.437545+0000","last_fullsized":"2024-04-25T05:53:48.437545+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":1170,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.113950+0000","last_change":"2024-04-25T05:53:21.029300+0000","last_active":"2024-04-25T05:53:45.113950+0000","last_peered":"2024-04-25T05:53:45.113950+0000","last_clean":"2024-04-25T05:53:45.113950+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:53:45.113950+0000","last_undegraded":"2024-04-25T05:53:45.113950+0000","last_fullsized":"2024-04-25T05:53:45.113950+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:21.029139+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.029139+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:30.616379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":166,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.444600+0000","last_change":"2024-04-25T05:53:33.165542+0000","last_active":"2024-04-25T05:53:48.444600+0000","last_peered":"2024-04-25T05:53:48.444600+0000","last_clean":"2024-04-25T05:53:48.444600+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:53:48.444600+0000","last_undegraded":"2024-04-25T05:53:48.444600+0000","last_fullsized":"2024-04-25T05:53:48.444600+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.165409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.165409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:03.084305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1083,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-04-25T05:53:48.440165+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:53:48.440165+0000","last_peered":"2024-04-25T05:53:48.440165+0000","last_clean":"2024-04-25T05:53:48.440165+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:53:48.440165+0000","last_undegraded":"2024-04-25T05:53:48.440165+0000","last_fullsized":"2024-04-25T05:53:48.440165+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.444351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":610,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-04-25T05:53:45.114798+0000","last_change":"2024-04-25T05:51:51.705620+0000","last_active":"2024-04-25T05:53:45.114798+0000","last_peered":"2024-04-25T05:53:45.114798+0000","last_clean":"2024-04-25T05:53:45.114798+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:53:45.114798+0000","last_undegraded":"2024-04-25T05:53:45.114798+0000","last_fullsized":"2024-04-25T05:53:45.114798+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:51.705348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:51.705348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:48.032991+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":914948120,"num_objects":13962,"num_object_clones":0,"num_object_copies":41886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":246612,"num_write_kb":8338305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28230,"num_bytes_recovered":1850015744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3272880128,"data_stored":3272867912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80310,"ondisk_log_size":80310,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1170,"num_read_kb":1165,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070707,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505796,"kb_used_data":567996,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92239356,"statfs":{"total":95995035648,"available":94453100544,"internally_reserved":0,"allocated":581627904,"data_stored":580962283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":960231882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365047,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337900,"kb_used_data":467812,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92407252,"statfs":{"total":95995035648,"available":94625026048,"internally_reserved":0,"allocated":479039488,"data_stored":478397259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":890567110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575120,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514656,"kb_used_data":422676,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":92230496,"statfs":{"total":95995035648,"available":94444027904,"internally_reserved":0,"allocated":432820224,"data_stored":432135773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738646,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061240,"kb_used_data":294256,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92683912,"statfs":{"total":95995035648,"available":94908325888,"internally_reserved":0,"allocated":301318144,"data_stored":300662968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411595,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090544,"kb_used_data":369384,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92654608,"statfs":{"total":95995035648,"available":94878318592,"internally_reserved":0,"allocated":378249216,"data_stored":377602387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149957,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611800,"kb_used_data":398540,"kb_used_omap":1,"kb_used_meta":1213118,"kb_avail":92133352,"statfs":{"total":95995035648,"available":94344552448,"internally_reserved":0,"allocated":408104960,"data_stored":407428791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1242233287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986432,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1356936,"kb_used_data":549696,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92388216,"statfs":{"total":95995035648,"available":94605533184,"internally_reserved":0,"allocated":562888704,"data_stored":562217985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692092,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873904,"kb_used_data":238824,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92871248,"statfs":{"total":95995035648,"available":95100157952,"internally_reserved":0,"allocated":244555776,"data_stored":243909447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":290979840,"data_stored":290979840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":430571520,"data_stored":430571520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":363204608,"data_stored":363200536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379584512,"data_stored":379584512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":552407040,"data_stored":552402968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238485504,"data_stored":238485504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":464392192,"data_stored":464388120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":553254912,"data_stored":553254912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:53:52.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:53:52.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.590 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:52.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[27010]: 3.a scrub starts 2024-04-25T05:53:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[27010]: 3.a scrub ok 2024-04-25T05:53:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[27010]: from='client.26588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:52.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[27010]: osdmap e573: 8 total, 8 up, 8 in 2024-04-25T05:53:52.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[37492]: 3.a scrub starts 2024-04-25T05:53:52.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[37492]: 3.a scrub ok 2024-04-25T05:53:52.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[37492]: from='client.26588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:52.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:52 smithi043 ceph-mon[37492]: osdmap e573: 8 total, 8 up, 8 in 2024-04-25T05:53:52.673 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:52.690 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:52.723 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:52 smithi060 ceph-mon[30898]: 3.a scrub starts 2024-04-25T05:53:52.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:52 smithi060 ceph-mon[30898]: 3.a scrub ok 2024-04-25T05:53:52.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:52 smithi060 ceph-mon[30898]: from='client.26588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:52.724 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:52 smithi060 ceph-mon[30898]: osdmap e573: 8 total, 8 up, 8 in 2024-04-25T05:53:52.743 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:52.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:52.880 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:52.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:52.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:53.018 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:53.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-25T05:53:53.120 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:53.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:53.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:53.307 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:53.400 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:53.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:53.489 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:53.501 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:53.514 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:53:53.562 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:53:53.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:53.610 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:53:53.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[27010]: pgmap v1078: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 918 op/s 2024-04-25T05:53:53.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:53:53.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:53:53.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:53:53.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:53:53.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[37492]: pgmap v1078: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 918 op/s 2024-04-25T05:53:53.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:53:53.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:53:53.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:53:53.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:53 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:53:53.635 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:53.654 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:53:53.695 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:53:53.733 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:53:53.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:53.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-mon[30898]: pgmap v1078: 158 pgs: 158 active+clean; 873 MiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 918 op/s 2024-04-25T05:53:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:53:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:53:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:53:53.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:53:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:53.775 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:53:53.815 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:53:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:53:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:53:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:53:53.818 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:53:53.819 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:53:53.819 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:53:53.819 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:53:53.819 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:53:53.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:53.901 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:53.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:54.002 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:54.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:54.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:54.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:54.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:53:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:53:54.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:54.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:54.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:54.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:54.582 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:54.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:54.710 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:54.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:54 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:53:54.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2659155616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:54.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:54 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:53:54.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:54 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:53:54.812 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.851 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:54.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:54.874 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2659155616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2659155616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:53:54.875 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:54 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:53:54.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:54.985 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:54.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:55.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:55.114 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:55.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:55.252 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:55.354 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:55.378 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:55.480 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:55.508 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:55.511 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: pgmap v1079: 158 pgs: 158 active+clean; 656 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 4.7 scrub starts 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 4.7 scrub ok 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: osdmap e574: 8 total, 8 up, 8 in 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:53:55.512 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:53:55.513 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:53:55.609 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:55.639 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:55.741 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: pgmap v1079: 158 pgs: 158 active+clean; 656 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:55.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 4.7 scrub starts 2024-04-25T05:53:55.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 4.7 scrub ok 2024-04-25T05:53:55.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: osdmap e574: 8 total, 8 up, 8 in 2024-04-25T05:53:55.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:53:55.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:55 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:53:55.770 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:55.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:53:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:53:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:53:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: pgmap v1079: 158 pgs: 158 active+clean; 656 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-04-25T05:53:55.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 4.7 scrub starts 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 4.7 scrub ok 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: osdmap e574: 8 total, 8 up, 8 in 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:53:55.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:55 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:53:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:53:55.916 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:55.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:55.925 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:53:56.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:56.046 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:56.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:56.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:56.331 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:56.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:56.463 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:56.521 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:53:56.565 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:56.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:56.743 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:56.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:56.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:56.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:56.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: osdmap e575: 8 total, 8 up, 8 in 2024-04-25T05:53:56.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1868994891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:56.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: osdmap e575: 8 total, 8 up, 8 in 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1868994891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:53:56.871 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:56 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:53:56.879 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:56.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:56.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:56.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: osdmap e575: 8 total, 8 up, 8 in 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1868994891' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:53:57.000 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:56 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:53:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:57.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:57.145 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:57.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:53:57.274 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:53:57.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:57.424 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:57.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.563 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:57.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:57.664 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:57.693 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:57.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: from='client.26594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:57.738 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: pgmap v1082: 158 pgs: 158 active+clean; 529 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: osdmap e576: 8 total, 8 up, 8 in 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: from='client.26594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:57.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: pgmap v1082: 158 pgs: 158 active+clean; 529 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: osdmap e576: 8 total, 8 up, 8 in 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:53:57.740 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:57 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:53:57.761 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:57.793 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1081,"stamp":"2024-04-25T05:53:55.495597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":555162163,"num_objects":8693,"num_object_clones":0,"num_object_copies":26079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8693,"num_whiteouts":0,"num_read":2589,"num_read_kb":2309,"num_write":253025,"num_write_kb":8338350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28897,"num_bytes_recovered":1850560081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82003,"ondisk_log_size":82003,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9273620,"kb_used_data":2214024,"kb_used_omap":12,"kb_used_meta":7059443,"kb_avail":740687596,"statfs":{"total":767960285184,"available":758464098304,"internally_reserved":0,"allocated":2267160576,"data_stored":2261757953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7228870192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-360382464,"num_objects":-5499,"num_object_clones":0,"num_object_copies":-16497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5499,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.379846"},"pg_stats":[{"pgid":"11.3","version":"573'15802","reported_seq":16149,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.648149+0000","last_change":"2024-04-25T05:53:34.830889+0000","last_active":"2024-04-25T05:53:53.648149+0000","last_peered":"2024-04-25T05:53:53.648149+0000","last_clean":"2024-04-25T05:53:53.648149+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:53:53.648149+0000","last_undegraded":"2024-04-25T05:53:53.648149+0000","last_fullsized":"2024-04-25T05:53:53.648149+0000","mapping_epoch":456,"log_start":"525'10546","ondisk_log_start":"525'10546","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'14507","last_scrub_stamp":"2024-04-25T05:53:34.830724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.830724+0000","objects_scrubbed":1862,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:55.510615+0000","scrub_duration":573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":554,"num_object_clones":0,"num_object_copies":1662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15802,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":571,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446550+0000","last_change":"2024-04-25T05:52:35.929420+0000","last_active":"2024-04-25T05:53:51.446550+0000","last_peered":"2024-04-25T05:53:51.446550+0000","last_clean":"2024-04-25T05:53:51.446550+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:53:51.446550+0000","last_undegraded":"2024-04-25T05:53:51.446550+0000","last_fullsized":"2024-04-25T05:53:51.446550+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:52:35.929307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.929307+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1086,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454697+0000","last_change":"2024-04-25T05:51:40.101355+0000","last_active":"2024-04-25T05:53:51.454697+0000","last_peered":"2024-04-25T05:53:51.454697+0000","last_clean":"2024-04-25T05:53:51.454697+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:53:51.454697+0000","last_undegraded":"2024-04-25T05:53:51.454697+0000","last_fullsized":"2024-04-25T05:53:51.454697+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:05.117047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:05.117047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":533,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456924+0000","last_change":"2024-04-25T05:52:52.397154+0000","last_active":"2024-04-25T05:53:51.456924+0000","last_peered":"2024-04-25T05:53:51.456924+0000","last_clean":"2024-04-25T05:53:51.456924+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:53:51.456924+0000","last_undegraded":"2024-04-25T05:53:51.456924+0000","last_fullsized":"2024-04-25T05:53:51.456924+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:52:52.394206+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-04-25T05:35:37.678003+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.394206+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:54.399800+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":648,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445731+0000","last_change":"2024-04-25T05:52:08.909918+0000","last_active":"2024-04-25T05:53:51.445731+0000","last_peered":"2024-04-25T05:53:51.445731+0000","last_clean":"2024-04-25T05:53:51.445731+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:53:51.445731+0000","last_undegraded":"2024-04-25T05:53:51.445731+0000","last_fullsized":"2024-04-25T05:53:51.445731+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:08.909664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.909664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1428,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455122+0000","last_change":"2024-04-25T05:51:58.168288+0000","last_active":"2024-04-25T05:53:51.455122+0000","last_peered":"2024-04-25T05:53:51.455122+0000","last_clean":"2024-04-25T05:53:51.455122+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:53:51.455122+0000","last_undegraded":"2024-04-25T05:53:51.455122+0000","last_fullsized":"2024-04-25T05:53:51.455122+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:58.168142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:58.168142+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"573'15888","reported_seq":29628,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.041843+0000","last_change":"2024-04-25T05:52:52.524759+0000","last_active":"2024-04-25T05:53:53.041843+0000","last_peered":"2024-04-25T05:53:53.041843+0000","last_clean":"2024-04-25T05:53:53.041843+0000","last_became_active":"2024-04-25T05:52:44.443285+0000","last_became_peered":"2024-04-25T05:52:44.443285+0000","last_unstale":"2024-04-25T05:53:53.041843+0000","last_undegraded":"2024-04-25T05:53:53.041843+0000","last_fullsized":"2024-04-25T05:53:53.041843+0000","mapping_epoch":540,"log_start":"512'9724","ondisk_log_start":"512'9724","created":456,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11957","last_scrub_stamp":"2024-04-25T05:52:52.524605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.524605+0000","objects_scrubbed":4563,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.589578+0000","scrub_duration":2115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15888,"num_write_kb":527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6713,"num_bytes_recovered":439877632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":580,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445194+0000","last_change":"2024-04-25T05:53:30.906380+0000","last_active":"2024-04-25T05:53:51.445194+0000","last_peered":"2024-04-25T05:53:51.445194+0000","last_clean":"2024-04-25T05:53:51.445194+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:53:51.445194+0000","last_undegraded":"2024-04-25T05:53:51.445194+0000","last_fullsized":"2024-04-25T05:53:51.445194+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:53:30.906231+0000","last_deep_scrub":"320'39","last_deep_scrub_stamp":"2024-04-25T05:51:57.812833+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.906231+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:58.223526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":809,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455963+0000","last_change":"2024-04-25T05:53:51.336319+0000","last_active":"2024-04-25T05:53:51.455963+0000","last_peered":"2024-04-25T05:53:51.455963+0000","last_clean":"2024-04-25T05:53:51.455963+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:53:51.455963+0000","last_undegraded":"2024-04-25T05:53:51.455963+0000","last_fullsized":"2024-04-25T05:53:51.455963+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:51.336244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:53:51.336244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.399146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1099,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.451217+0000","last_change":"2024-04-25T05:51:40.109308+0000","last_active":"2024-04-25T05:53:51.451217+0000","last_peered":"2024-04-25T05:53:51.451217+0000","last_clean":"2024-04-25T05:53:51.451217+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:53:51.451217+0000","last_undegraded":"2024-04-25T05:53:51.451217+0000","last_fullsized":"2024-04-25T05:53:51.451217+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":437,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455785+0000","last_change":"2024-04-25T05:53:34.769498+0000","last_active":"2024-04-25T05:53:51.455785+0000","last_peered":"2024-04-25T05:53:51.455785+0000","last_clean":"2024-04-25T05:53:51.455785+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:53:51.455785+0000","last_undegraded":"2024-04-25T05:53:51.455785+0000","last_fullsized":"2024-04-25T05:53:51.455785+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.769373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.769373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.388264+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":1026,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452533+0000","last_change":"2024-04-25T05:52:07.018629+0000","last_active":"2024-04-25T05:53:51.452533+0000","last_peered":"2024-04-25T05:53:51.452533+0000","last_clean":"2024-04-25T05:53:51.452533+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:53:51.452533+0000","last_undegraded":"2024-04-25T05:53:51.452533+0000","last_fullsized":"2024-04-25T05:53:51.452533+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:52:07.018495+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:52:07.018495+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"573'15728","reported_seq":29610,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.695307+0000","last_change":"2024-04-25T05:52:36.116043+0000","last_active":"2024-04-25T05:53:53.695307+0000","last_peered":"2024-04-25T05:53:53.695307+0000","last_clean":"2024-04-25T05:53:53.695307+0000","last_became_active":"2024-04-25T05:52:36.097202+0000","last_became_peered":"2024-04-25T05:52:36.097202+0000","last_unstale":"2024-04-25T05:53:53.695307+0000","last_undegraded":"2024-04-25T05:53:53.695307+0000","last_fullsized":"2024-04-25T05:53:53.695307+0000","mapping_epoch":532,"log_start":"523'10465","ondisk_log_start":"523'10465","created":456,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'6932","last_scrub_stamp":"2024-04-25T05:51:30.948125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.948125+0000","objects_scrubbed":6847,"log_size":5263,"log_dups_size":0,"ondisk_log_size":5263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":546,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15728,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6773,"num_bytes_recovered":443875328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":880,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445508+0000","last_change":"2024-04-25T05:52:41.908835+0000","last_active":"2024-04-25T05:53:51.445508+0000","last_peered":"2024-04-25T05:53:51.445508+0000","last_clean":"2024-04-25T05:53:51.445508+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:53:51.445508+0000","last_undegraded":"2024-04-25T05:53:51.445508+0000","last_fullsized":"2024-04-25T05:53:51.445508+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:52:41.908693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.908693+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:56.986259+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1115,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.443837+0000","last_change":"2024-04-25T05:53:29.887090+0000","last_active":"2024-04-25T05:53:51.443837+0000","last_peered":"2024-04-25T05:53:51.443837+0000","last_clean":"2024-04-25T05:53:51.443837+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:53:51.443837+0000","last_undegraded":"2024-04-25T05:53:51.443837+0000","last_fullsized":"2024-04-25T05:53:51.443837+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.887008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.890083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453324+0000","last_change":"2024-04-25T05:53:33.021664+0000","last_active":"2024-04-25T05:53:51.453324+0000","last_peered":"2024-04-25T05:53:51.453324+0000","last_clean":"2024-04-25T05:53:51.453324+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:53:51.453324+0000","last_undegraded":"2024-04-25T05:53:51.453324+0000","last_fullsized":"2024-04-25T05:53:51.453324+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.021506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.021506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.233074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1074,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454250+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:53:51.454250+0000","last_peered":"2024-04-25T05:53:51.454250+0000","last_clean":"2024-04-25T05:53:51.454250+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:53:51.454250+0000","last_undegraded":"2024-04-25T05:53:51.454250+0000","last_fullsized":"2024-04-25T05:53:51.454250+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"573'15709","reported_seq":16008,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.450028+0000","last_change":"2024-04-25T05:52:50.663942+0000","last_active":"2024-04-25T05:53:54.450028+0000","last_peered":"2024-04-25T05:53:54.450028+0000","last_clean":"2024-04-25T05:53:54.450028+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:53:54.450028+0000","last_undegraded":"2024-04-25T05:53:54.450028+0000","last_fullsized":"2024-04-25T05:53:54.450028+0000","mapping_epoch":456,"log_start":"533'10913","ondisk_log_start":"533'10913","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'11702","last_scrub_stamp":"2024-04-25T05:52:50.663796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.663796+0000","objects_scrubbed":4552,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":518208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":1123,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448801+0000","last_change":"2024-04-25T05:53:30.416648+0000","last_active":"2024-04-25T05:53:51.448801+0000","last_peered":"2024-04-25T05:53:51.448801+0000","last_clean":"2024-04-25T05:53:51.448801+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:53:51.448801+0000","last_undegraded":"2024-04-25T05:53:51.448801+0000","last_fullsized":"2024-04-25T05:53:51.448801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:30.416481+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.416481+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.310035+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1071,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460727+0000","last_change":"2024-04-25T05:51:20.429881+0000","last_active":"2024-04-25T05:53:51.460727+0000","last_peered":"2024-04-25T05:53:51.460727+0000","last_clean":"2024-04-25T05:53:51.460727+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:53:51.460727+0000","last_undegraded":"2024-04-25T05:53:51.460727+0000","last_fullsized":"2024-04-25T05:53:51.460727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:20.429760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:20.429760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":789,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445714+0000","last_change":"2024-04-25T05:53:49.070910+0000","last_active":"2024-04-25T05:53:51.445714+0000","last_peered":"2024-04-25T05:53:51.445714+0000","last_clean":"2024-04-25T05:53:51.445714+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:53:51.445714+0000","last_undegraded":"2024-04-25T05:53:51.445714+0000","last_fullsized":"2024-04-25T05:53:51.445714+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:53:49.070607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:49.070607+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:03.074399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":418,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453242+0000","last_change":"2024-04-25T05:53:21.122301+0000","last_active":"2024-04-25T05:53:51.453242+0000","last_peered":"2024-04-25T05:53:51.453242+0000","last_clean":"2024-04-25T05:53:51.453242+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:53:51.453242+0000","last_undegraded":"2024-04-25T05:53:51.453242+0000","last_fullsized":"2024-04-25T05:53:51.453242+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.122207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.122207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:28.763652+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":990,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452767+0000","last_change":"2024-04-25T05:52:45.117231+0000","last_active":"2024-04-25T05:53:51.452767+0000","last_peered":"2024-04-25T05:53:51.452767+0000","last_clean":"2024-04-25T05:53:51.452767+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:53:51.452767+0000","last_undegraded":"2024-04-25T05:53:51.452767+0000","last_fullsized":"2024-04-25T05:53:51.452767+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.117108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.117108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.151932+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"573'15943","reported_seq":16266,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.386312+0000","last_change":"2024-04-25T05:52:09.488584+0000","last_active":"2024-04-25T05:53:53.386312+0000","last_peered":"2024-04-25T05:53:53.386312+0000","last_clean":"2024-04-25T05:53:53.386312+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:53:53.386312+0000","last_undegraded":"2024-04-25T05:53:53.386312+0000","last_fullsized":"2024-04-25T05:53:53.386312+0000","mapping_epoch":456,"log_start":"533'11169","ondisk_log_start":"533'11169","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'9496","last_scrub_stamp":"2024-04-25T05:52:09.488306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:09.488306+0000","objects_scrubbed":7109,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15943,"num_write_kb":528832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1095,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.443749+0000","last_change":"2024-04-25T05:53:33.920196+0000","last_active":"2024-04-25T05:53:51.443749+0000","last_peered":"2024-04-25T05:53:51.443749+0000","last_clean":"2024-04-25T05:53:51.443749+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:53:51.443749+0000","last_undegraded":"2024-04-25T05:53:51.443749+0000","last_fullsized":"2024-04-25T05:53:51.443749+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.920077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.920077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.068133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1253,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460807+0000","last_change":"2024-04-25T05:51:40.149035+0000","last_active":"2024-04-25T05:53:51.460807+0000","last_peered":"2024-04-25T05:53:51.460807+0000","last_clean":"2024-04-25T05:53:51.460807+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:53:51.460807+0000","last_undegraded":"2024-04-25T05:53:51.460807+0000","last_fullsized":"2024-04-25T05:53:51.460807+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:51:29.363198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.363198+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1109,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455207+0000","last_change":"2024-04-25T05:53:20.767176+0000","last_active":"2024-04-25T05:53:51.455207+0000","last_peered":"2024-04-25T05:53:51.455207+0000","last_clean":"2024-04-25T05:53:51.455207+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:53:51.455207+0000","last_undegraded":"2024-04-25T05:53:51.455207+0000","last_fullsized":"2024-04-25T05:53:51.455207+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:20.767051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:20.767051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.097716+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":642,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444197+0000","last_change":"2024-04-25T05:52:42.951239+0000","last_active":"2024-04-25T05:53:51.444197+0000","last_peered":"2024-04-25T05:53:51.444197+0000","last_clean":"2024-04-25T05:53:51.444197+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:53:51.444197+0000","last_undegraded":"2024-04-25T05:53:51.444197+0000","last_fullsized":"2024-04-25T05:53:51.444197+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.951111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.951111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:56.859438+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1364,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444701+0000","last_change":"2024-04-25T05:53:27.936176+0000","last_active":"2024-04-25T05:53:51.444701+0000","last_peered":"2024-04-25T05:53:51.444701+0000","last_clean":"2024-04-25T05:53:51.444701+0000","last_became_active":"2024-04-25T05:52:47.115419+0000","last_became_peered":"2024-04-25T05:52:47.115419+0000","last_unstale":"2024-04-25T05:53:51.444701+0000","last_undegraded":"2024-04-25T05:53:51.444701+0000","last_fullsized":"2024-04-25T05:53:51.444701+0000","mapping_epoch":379,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:27.935927+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:53:27.935927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:57.848657+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"573'15294","reported_seq":15596,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.274877+0000","last_change":"2024-04-25T05:52:42.423083+0000","last_active":"2024-04-25T05:53:53.274877+0000","last_peered":"2024-04-25T05:53:53.274877+0000","last_clean":"2024-04-25T05:53:53.274877+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:53:53.274877+0000","last_undegraded":"2024-04-25T05:53:53.274877+0000","last_fullsized":"2024-04-25T05:53:53.274877+0000","mapping_epoch":456,"log_start":"529'10585","ondisk_log_start":"529'10585","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11106","last_scrub_stamp":"2024-04-25T05:52:42.422923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.422923+0000","objects_scrubbed":4765,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.564049+0000","scrub_duration":1509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35520512,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15294,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1321,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460967+0000","last_change":"2024-04-25T05:53:31.717402+0000","last_active":"2024-04-25T05:53:51.460967+0000","last_peered":"2024-04-25T05:53:51.460967+0000","last_clean":"2024-04-25T05:53:51.460967+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:53:51.460967+0000","last_undegraded":"2024-04-25T05:53:51.460967+0000","last_fullsized":"2024-04-25T05:53:51.460967+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:53:31.717254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.717254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.453072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":772,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444037+0000","last_change":"2024-04-25T05:52:45.015305+0000","last_active":"2024-04-25T05:53:51.444037+0000","last_peered":"2024-04-25T05:53:51.444037+0000","last_clean":"2024-04-25T05:53:51.444037+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:53:51.444037+0000","last_undegraded":"2024-04-25T05:53:51.444037+0000","last_fullsized":"2024-04-25T05:53:51.444037+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:52:45.015213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.015213+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.513334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":166,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448224+0000","last_change":"2024-04-25T05:51:40.129777+0000","last_active":"2024-04-25T05:53:51.448224+0000","last_peered":"2024-04-25T05:53:51.448224+0000","last_clean":"2024-04-25T05:53:51.448224+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:53:51.448224+0000","last_undegraded":"2024-04-25T05:53:51.448224+0000","last_fullsized":"2024-04-25T05:53:51.448224+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":460,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448585+0000","last_change":"2024-04-25T05:53:34.695038+0000","last_active":"2024-04-25T05:53:51.448585+0000","last_peered":"2024-04-25T05:53:51.448585+0000","last_clean":"2024-04-25T05:53:51.448585+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:53:51.448585+0000","last_undegraded":"2024-04-25T05:53:51.448585+0000","last_fullsized":"2024-04-25T05:53:51.448585+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.694925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.694925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.211727+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1330,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454096+0000","last_change":"2024-04-25T05:53:21.777034+0000","last_active":"2024-04-25T05:53:51.454096+0000","last_peered":"2024-04-25T05:53:51.454096+0000","last_clean":"2024-04-25T05:53:51.454096+0000","last_became_active":"2024-04-25T05:52:51.531591+0000","last_became_peered":"2024-04-25T05:52:51.531591+0000","last_unstale":"2024-04-25T05:53:51.454096+0000","last_undegraded":"2024-04-25T05:53:51.454096+0000","last_fullsized":"2024-04-25T05:53:51.454096+0000","mapping_epoch":387,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:21.776901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.776901+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.945686+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"573'15658","reported_seq":15992,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.005128+0000","last_change":"2024-04-25T05:52:45.545440+0000","last_active":"2024-04-25T05:53:53.005128+0000","last_peered":"2024-04-25T05:53:53.005128+0000","last_clean":"2024-04-25T05:53:53.005128+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:53:53.005128+0000","last_undegraded":"2024-04-25T05:53:53.005128+0000","last_fullsized":"2024-04-25T05:53:53.005128+0000","mapping_epoch":456,"log_start":"512'9490","ondisk_log_start":"512'9490","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'11448","last_scrub_stamp":"2024-04-25T05:52:45.545041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.545041+0000","objects_scrubbed":4796,"log_size":6168,"log_dups_size":0,"ondisk_log_size":6168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.959830+0000","scrub_duration":1123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765720,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15658,"num_write_kb":518977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":192,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447805+0000","last_change":"2024-04-25T05:51:40.124005+0000","last_active":"2024-04-25T05:53:51.447805+0000","last_peered":"2024-04-25T05:53:51.447805+0000","last_clean":"2024-04-25T05:53:51.447805+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:53:51.447805+0000","last_undegraded":"2024-04-25T05:53:51.447805+0000","last_fullsized":"2024-04-25T05:53:51.447805+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":820,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446727+0000","last_change":"2024-04-25T05:51:32.949273+0000","last_active":"2024-04-25T05:53:51.446727+0000","last_peered":"2024-04-25T05:53:51.446727+0000","last_clean":"2024-04-25T05:53:51.446727+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:53:51.446727+0000","last_undegraded":"2024-04-25T05:53:51.446727+0000","last_fullsized":"2024-04-25T05:53:51.446727+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:51:32.949135+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:51:32.949135+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453342+0000","last_change":"2024-04-25T05:52:41.064075+0000","last_active":"2024-04-25T05:53:51.453342+0000","last_peered":"2024-04-25T05:53:51.453342+0000","last_clean":"2024-04-25T05:53:51.453342+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:53:51.453342+0000","last_undegraded":"2024-04-25T05:53:51.453342+0000","last_fullsized":"2024-04-25T05:53:51.453342+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:41.063933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.063933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":170,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447436+0000","last_change":"2024-04-25T05:53:49.262740+0000","last_active":"2024-04-25T05:53:51.447436+0000","last_peered":"2024-04-25T05:53:51.447436+0000","last_clean":"2024-04-25T05:53:51.447436+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:53:51.447436+0000","last_undegraded":"2024-04-25T05:53:51.447436+0000","last_fullsized":"2024-04-25T05:53:51.447436+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:49.262490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:49.262490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:13.639641+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1320,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455992+0000","last_change":"2024-04-25T05:53:50.288882+0000","last_active":"2024-04-25T05:53:51.455992+0000","last_peered":"2024-04-25T05:53:51.455992+0000","last_clean":"2024-04-25T05:53:51.455992+0000","last_became_active":"2024-04-25T05:52:39.109573+0000","last_became_peered":"2024-04-25T05:52:39.109573+0000","last_unstale":"2024-04-25T05:53:51.455992+0000","last_undegraded":"2024-04-25T05:53:51.455992+0000","last_fullsized":"2024-04-25T05:53:51.455992+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:50.288752+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.288752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.700218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"573'15989","reported_seq":16285,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.467408+0000","last_change":"2024-04-25T05:52:52.331705+0000","last_active":"2024-04-25T05:53:54.467408+0000","last_peered":"2024-04-25T05:53:54.467408+0000","last_clean":"2024-04-25T05:53:54.467408+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:53:54.467408+0000","last_undegraded":"2024-04-25T05:53:54.467408+0000","last_fullsized":"2024-04-25T05:53:54.467408+0000","mapping_epoch":456,"log_start":"533'11166","ondisk_log_start":"533'11166","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12101","last_scrub_stamp":"2024-04-25T05:52:52.331512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.331512+0000","objects_scrubbed":4384,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.199070+0000","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30081024,"num_objects":459,"num_object_clones":0,"num_object_copies":1377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15989,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":845,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456076+0000","last_change":"2024-04-25T05:53:33.265611+0000","last_active":"2024-04-25T05:53:51.456076+0000","last_peered":"2024-04-25T05:53:51.456076+0000","last_clean":"2024-04-25T05:53:51.456076+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:53:51.456076+0000","last_undegraded":"2024-04-25T05:53:51.456076+0000","last_fullsized":"2024-04-25T05:53:51.456076+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.265484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.265484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:54.263173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":190,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447796+0000","last_change":"2024-04-25T05:53:24.028014+0000","last_active":"2024-04-25T05:53:51.447796+0000","last_peered":"2024-04-25T05:53:51.447796+0000","last_clean":"2024-04-25T05:53:51.447796+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:53:51.447796+0000","last_undegraded":"2024-04-25T05:53:51.447796+0000","last_fullsized":"2024-04-25T05:53:51.447796+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:53:24.027846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.027846+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:33.305120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1108,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.450988+0000","last_change":"2024-04-25T05:50:39.101207+0000","last_active":"2024-04-25T05:53:51.450988+0000","last_peered":"2024-04-25T05:53:51.450988+0000","last_clean":"2024-04-25T05:53:51.450988+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:53:51.450988+0000","last_undegraded":"2024-04-25T05:53:51.450988+0000","last_fullsized":"2024-04-25T05:53:51.450988+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:39.101063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:39.101063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1090,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446464+0000","last_change":"2024-04-25T05:52:47.618782+0000","last_active":"2024-04-25T05:53:51.446464+0000","last_peered":"2024-04-25T05:53:51.446464+0000","last_clean":"2024-04-25T05:53:51.446464+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:53:51.446464+0000","last_undegraded":"2024-04-25T05:53:51.446464+0000","last_fullsized":"2024-04-25T05:53:51.446464+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.618569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.618569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:17.510421+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1339,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455293+0000","last_change":"2024-04-25T05:52:43.436694+0000","last_active":"2024-04-25T05:53:51.455293+0000","last_peered":"2024-04-25T05:53:51.455293+0000","last_clean":"2024-04-25T05:53:51.455293+0000","last_became_active":"2024-04-25T05:52:43.436401+0000","last_became_peered":"2024-04-25T05:52:43.436401+0000","last_unstale":"2024-04-25T05:53:51.455293+0000","last_undegraded":"2024-04-25T05:53:51.455293+0000","last_fullsized":"2024-04-25T05:53:51.455293+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:37.366472+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:52:37.366472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"573'16034","reported_seq":25028,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.401562+0000","last_change":"2024-04-25T05:53:17.142210+0000","last_active":"2024-04-25T05:53:54.401562+0000","last_peered":"2024-04-25T05:53:54.401562+0000","last_clean":"2024-04-25T05:53:54.401562+0000","last_became_active":"2024-04-25T05:53:17.107904+0000","last_became_peered":"2024-04-25T05:53:17.107904+0000","last_unstale":"2024-04-25T05:53:54.401562+0000","last_undegraded":"2024-04-25T05:53:54.401562+0000","last_fullsized":"2024-04-25T05:53:54.401562+0000","mapping_epoch":557,"log_start":"538'11527","ondisk_log_start":"538'11527","created":456,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16034,"num_write_kb":529024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4367,"num_bytes_recovered":286195712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":418,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453380+0000","last_change":"2024-04-25T05:52:53.143666+0000","last_active":"2024-04-25T05:53:51.453380+0000","last_peered":"2024-04-25T05:53:51.453380+0000","last_clean":"2024-04-25T05:53:51.453380+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:53:51.453380+0000","last_undegraded":"2024-04-25T05:53:51.453380+0000","last_fullsized":"2024-04-25T05:53:51.453380+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.143529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.143529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.394156+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":97,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447925+0000","last_change":"2024-04-25T05:52:43.541002+0000","last_active":"2024-04-25T05:53:51.447925+0000","last_peered":"2024-04-25T05:53:51.447925+0000","last_clean":"2024-04-25T05:53:51.447925+0000","last_became_active":"2024-04-25T05:52:43.437539+0000","last_became_peered":"2024-04-25T05:52:43.437539+0000","last_unstale":"2024-04-25T05:53:51.447925+0000","last_undegraded":"2024-04-25T05:53:51.447925+0000","last_fullsized":"2024-04-25T05:53:51.447925+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:52:38.142064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.142064+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2315,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:52.275767+0000","last_change":"2024-04-25T05:51:40.120019+0000","last_active":"2024-04-25T05:53:52.275767+0000","last_peered":"2024-04-25T05:53:52.275767+0000","last_clean":"2024-04-25T05:53:52.275767+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:53:52.275767+0000","last_undegraded":"2024-04-25T05:53:52.275767+0000","last_fullsized":"2024-04-25T05:53:52.275767+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:51:29.129304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.129304+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1175,"num_read_kb":1170,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":170,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.449025+0000","last_change":"2024-04-25T05:53:26.049870+0000","last_active":"2024-04-25T05:53:51.449025+0000","last_peered":"2024-04-25T05:53:51.449025+0000","last_clean":"2024-04-25T05:53:51.449025+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:53:51.449025+0000","last_undegraded":"2024-04-25T05:53:51.449025+0000","last_fullsized":"2024-04-25T05:53:51.449025+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:26.049619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.049619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:36.269598+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1078,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446534+0000","last_change":"2024-04-25T05:51:22.949696+0000","last_active":"2024-04-25T05:53:51.446534+0000","last_peered":"2024-04-25T05:53:51.446534+0000","last_clean":"2024-04-25T05:53:51.446534+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:53:51.446534+0000","last_undegraded":"2024-04-25T05:53:51.446534+0000","last_fullsized":"2024-04-25T05:53:51.446534+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.949432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.949432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1326,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446971+0000","last_change":"2024-04-25T05:52:55.110481+0000","last_active":"2024-04-25T05:53:51.446971+0000","last_peered":"2024-04-25T05:53:51.446971+0000","last_clean":"2024-04-25T05:53:51.446971+0000","last_became_active":"2024-04-25T05:52:55.109763+0000","last_became_peered":"2024-04-25T05:52:55.109763+0000","last_unstale":"2024-04-25T05:53:51.446971+0000","last_undegraded":"2024-04-25T05:53:51.446971+0000","last_fullsized":"2024-04-25T05:53:51.446971+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.463489+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.463489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.844339+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"573'15544","reported_seq":15789,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.436012+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:53:54.436012+0000","last_peered":"2024-04-25T05:53:54.436012+0000","last_clean":"2024-04-25T05:53:54.436012+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:53:54.436012+0000","last_undegraded":"2024-04-25T05:53:54.436012+0000","last_fullsized":"2024-04-25T05:53:54.436012+0000","mapping_epoch":456,"log_start":"537'10979","ondisk_log_start":"537'10979","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15544,"num_write_kb":511296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1108,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455119+0000","last_change":"2024-04-25T05:53:29.647674+0000","last_active":"2024-04-25T05:53:51.455119+0000","last_peered":"2024-04-25T05:53:51.455119+0000","last_clean":"2024-04-25T05:53:51.455119+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:53:51.455119+0000","last_undegraded":"2024-04-25T05:53:51.455119+0000","last_fullsized":"2024-04-25T05:53:51.455119+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.647523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.647523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.191587+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":1017,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456986+0000","last_change":"2024-04-25T05:52:38.405034+0000","last_active":"2024-04-25T05:53:51.456986+0000","last_peered":"2024-04-25T05:53:51.456986+0000","last_clean":"2024-04-25T05:53:51.456986+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:53:51.456986+0000","last_undegraded":"2024-04-25T05:53:51.456986+0000","last_fullsized":"2024-04-25T05:53:51.456986+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:52:38.404858+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.404858+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.254310+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1087,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454248+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:53:51.454248+0000","last_peered":"2024-04-25T05:53:51.454248+0000","last_clean":"2024-04-25T05:53:51.454248+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:53:51.454248+0000","last_undegraded":"2024-04-25T05:53:51.454248+0000","last_fullsized":"2024-04-25T05:53:51.454248+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1452,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454411+0000","last_change":"2024-04-25T05:52:35.380122+0000","last_active":"2024-04-25T05:53:51.454411+0000","last_peered":"2024-04-25T05:53:51.454411+0000","last_clean":"2024-04-25T05:53:51.454411+0000","last_became_active":"2024-04-25T05:52:35.091072+0000","last_became_peered":"2024-04-25T05:52:35.091072+0000","last_unstale":"2024-04-25T05:53:51.454411+0000","last_undegraded":"2024-04-25T05:53:51.454411+0000","last_fullsized":"2024-04-25T05:53:51.454411+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":19,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:35.380044+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.380044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":377,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448915+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:53:51.448915+0000","last_peered":"2024-04-25T05:53:51.448915+0000","last_clean":"2024-04-25T05:53:51.448915+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:53:51.448915+0000","last_undegraded":"2024-04-25T05:53:51.448915+0000","last_fullsized":"2024-04-25T05:53:51.448915+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"573'15783","reported_seq":16027,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.437600+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:53:54.437600+0000","last_peered":"2024-04-25T05:53:54.437600+0000","last_clean":"2024-04-25T05:53:54.437600+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:53:54.437600+0000","last_undegraded":"2024-04-25T05:53:54.437600+0000","last_fullsized":"2024-04-25T05:53:54.437600+0000","mapping_epoch":456,"log_start":"537'11223","ondisk_log_start":"537'11223","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15783,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":443,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.458241+0000","last_change":"2024-04-25T05:53:50.508173+0000","last_active":"2024-04-25T05:53:51.458241+0000","last_peered":"2024-04-25T05:53:51.458241+0000","last_clean":"2024-04-25T05:53:51.458241+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:53:51.458241+0000","last_undegraded":"2024-04-25T05:53:51.458241+0000","last_fullsized":"2024-04-25T05:53:51.458241+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:50.508053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.508053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:16.809144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1245,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.401795+0000","last_change":"2024-04-25T05:53:54.401863+0000","last_active":"2024-04-25T05:53:54.401795+0000","last_peered":"2024-04-25T05:53:54.401795+0000","last_clean":"2024-04-25T05:53:54.401795+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:53:54.401795+0000","last_undegraded":"2024-04-25T05:53:54.401795+0000","last_fullsized":"2024-04-25T05:53:54.401795+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:53:54.401716+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:53:54.401716+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:19.530444+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":443,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452907+0000","last_change":"2024-04-25T05:53:21.420018+0000","last_active":"2024-04-25T05:53:51.452907+0000","last_peered":"2024-04-25T05:53:51.452907+0000","last_clean":"2024-04-25T05:53:51.452907+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:53:51.452907+0000","last_undegraded":"2024-04-25T05:53:51.452907+0000","last_fullsized":"2024-04-25T05:53:51.452907+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.419893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.419893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.568518+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1114,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446029+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:53:51.446029+0000","last_peered":"2024-04-25T05:53:51.446029+0000","last_clean":"2024-04-25T05:53:51.446029+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:53:51.446029+0000","last_undegraded":"2024-04-25T05:53:51.446029+0000","last_fullsized":"2024-04-25T05:53:51.446029+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1086,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456413+0000","last_change":"2024-04-25T05:52:51.540421+0000","last_active":"2024-04-25T05:53:51.456413+0000","last_peered":"2024-04-25T05:53:51.456413+0000","last_clean":"2024-04-25T05:53:51.456413+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:53:51.456413+0000","last_undegraded":"2024-04-25T05:53:51.456413+0000","last_fullsized":"2024-04-25T05:53:51.456413+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.540285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:53.176425+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1460,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455507+0000","last_change":"2024-04-25T05:52:34.412036+0000","last_active":"2024-04-25T05:53:51.455507+0000","last_peered":"2024-04-25T05:53:51.455507+0000","last_clean":"2024-04-25T05:53:51.455507+0000","last_became_active":"2024-04-25T05:52:27.138282+0000","last_became_peered":"2024-04-25T05:52:27.138282+0000","last_unstale":"2024-04-25T05:53:51.455507+0000","last_undegraded":"2024-04-25T05:53:51.455507+0000","last_fullsized":"2024-04-25T05:53:51.455507+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.411897+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.411897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:51.588807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"573'15812","reported_seq":24101,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:54.443272+0000","last_change":"2024-04-25T05:53:16.136131+0000","last_active":"2024-04-25T05:53:54.443272+0000","last_peered":"2024-04-25T05:53:54.443272+0000","last_clean":"2024-04-25T05:53:54.443272+0000","last_became_active":"2024-04-25T05:53:16.108052+0000","last_became_peered":"2024-04-25T05:53:16.108052+0000","last_unstale":"2024-04-25T05:53:54.443272+0000","last_undegraded":"2024-04-25T05:53:54.443272+0000","last_fullsized":"2024-04-25T05:53:54.443272+0000","mapping_epoch":556,"log_start":"537'11326","ondisk_log_start":"537'11326","created":456,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15812,"num_write_kb":521344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":162,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445906+0000","last_change":"2024-04-25T05:51:40.098006+0000","last_active":"2024-04-25T05:53:51.445906+0000","last_peered":"2024-04-25T05:53:51.445906+0000","last_clean":"2024-04-25T05:53:51.445906+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:53:51.445906+0000","last_undegraded":"2024-04-25T05:53:51.445906+0000","last_fullsized":"2024-04-25T05:53:51.445906+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:21.946445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:21.946445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1464,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.451116+0000","last_change":"2024-04-25T05:53:29.429046+0000","last_active":"2024-04-25T05:53:51.451116+0000","last_peered":"2024-04-25T05:53:51.451116+0000","last_clean":"2024-04-25T05:53:51.451116+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:53:51.451116+0000","last_undegraded":"2024-04-25T05:53:51.451116+0000","last_fullsized":"2024-04-25T05:53:51.451116+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:53:29.428902+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.428902+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.894655+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":653,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.443808+0000","last_change":"2024-04-25T05:51:50.680986+0000","last_active":"2024-04-25T05:53:51.443808+0000","last_peered":"2024-04-25T05:53:51.443808+0000","last_clean":"2024-04-25T05:53:51.443808+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:53:51.443808+0000","last_undegraded":"2024-04-25T05:53:51.443808+0000","last_fullsized":"2024-04-25T05:53:51.443808+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:50.680590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:51:50.680590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"506'3","reported_seq":417,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452662+0000","last_change":"2024-04-25T05:52:01.064448+0000","last_active":"2024-04-25T05:53:51.452662+0000","last_peered":"2024-04-25T05:53:51.452662+0000","last_clean":"2024-04-25T05:53:51.452662+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:53:51.452662+0000","last_undegraded":"2024-04-25T05:53:51.452662+0000","last_fullsized":"2024-04-25T05:53:51.452662+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'3","last_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_deep_scrub":"506'3","last_deep_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_clean_scrub_stamp":"2024-04-25T05:52:01.064292+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1051,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455918+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:53:51.455918+0000","last_peered":"2024-04-25T05:53:51.455918+0000","last_clean":"2024-04-25T05:53:51.455918+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:53:51.455918+0000","last_undegraded":"2024-04-25T05:53:51.455918+0000","last_fullsized":"2024-04-25T05:53:51.455918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":1161,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452330+0000","last_change":"2024-04-25T05:52:34.205086+0000","last_active":"2024-04-25T05:53:51.452330+0000","last_peered":"2024-04-25T05:53:51.452330+0000","last_clean":"2024-04-25T05:53:51.452330+0000","last_became_active":"2024-04-25T05:52:31.177637+0000","last_became_peered":"2024-04-25T05:52:31.177637+0000","last_unstale":"2024-04-25T05:53:51.452330+0000","last_undegraded":"2024-04-25T05:53:51.452330+0000","last_fullsized":"2024-04-25T05:53:51.452330+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.204967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.204967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.850061+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"573'15723","reported_seq":16049,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.377717+0000","last_change":"2024-04-25T05:52:03.230823+0000","last_active":"2024-04-25T05:53:53.377717+0000","last_peered":"2024-04-25T05:53:53.377717+0000","last_clean":"2024-04-25T05:53:53.377717+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:53:53.377717+0000","last_undegraded":"2024-04-25T05:53:53.377717+0000","last_fullsized":"2024-04-25T05:53:53.377717+0000","mapping_epoch":456,"log_start":"533'10990","ondisk_log_start":"533'10990","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8908","last_scrub_stamp":"2024-04-25T05:52:03.230624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:03.230624+0000","objects_scrubbed":7429,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765696,"num_objects":561,"num_object_clones":0,"num_object_copies":1683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15723,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":173,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447699+0000","last_change":"2024-04-25T05:51:40.140728+0000","last_active":"2024-04-25T05:53:51.447699+0000","last_peered":"2024-04-25T05:53:51.447699+0000","last_clean":"2024-04-25T05:53:51.447699+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:53:51.447699+0000","last_undegraded":"2024-04-25T05:53:51.447699+0000","last_fullsized":"2024-04-25T05:53:51.447699+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:50:14.065782+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.065782+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453192+0000","last_change":"2024-04-25T05:52:52.129720+0000","last_active":"2024-04-25T05:53:51.453192+0000","last_peered":"2024-04-25T05:53:51.453192+0000","last_clean":"2024-04-25T05:53:51.453192+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:53:51.453192+0000","last_undegraded":"2024-04-25T05:53:51.453192+0000","last_fullsized":"2024-04-25T05:53:51.453192+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:52.129566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.129566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.024201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":437,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:52.487550+0000","last_change":"2024-04-25T05:53:52.487607+0000","last_active":"2024-04-25T05:53:52.487550+0000","last_peered":"2024-04-25T05:53:52.487550+0000","last_clean":"2024-04-25T05:53:52.487550+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:53:52.487550+0000","last_undegraded":"2024-04-25T05:53:52.487550+0000","last_fullsized":"2024-04-25T05:53:52.487550+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:52.487452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:53:52.487452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:02.976586+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":162,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447331+0000","last_change":"2024-04-25T05:51:40.099599+0000","last_active":"2024-04-25T05:53:51.447331+0000","last_peered":"2024-04-25T05:53:51.447331+0000","last_clean":"2024-04-25T05:53:51.447331+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:53:51.447331+0000","last_undegraded":"2024-04-25T05:53:51.447331+0000","last_fullsized":"2024-04-25T05:53:51.447331+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.026628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.026628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"573'16083","reported_seq":16390,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.369280+0000","last_change":"2024-04-25T05:52:38.587166+0000","last_active":"2024-04-25T05:53:53.369280+0000","last_peered":"2024-04-25T05:53:53.369280+0000","last_clean":"2024-04-25T05:53:53.369280+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:53:53.369280+0000","last_undegraded":"2024-04-25T05:53:53.369280+0000","last_fullsized":"2024-04-25T05:53:53.369280+0000","mapping_epoch":456,"log_start":"534'11331","ondisk_log_start":"534'11331","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'11363","last_scrub_stamp":"2024-04-25T05:52:38.587034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.587034+0000","objects_scrubbed":5307,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":559,"num_object_clones":0,"num_object_copies":1677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16083,"num_write_kb":532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":865,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455862+0000","last_change":"2024-04-25T05:52:39.432701+0000","last_active":"2024-04-25T05:53:51.455862+0000","last_peered":"2024-04-25T05:53:51.455862+0000","last_clean":"2024-04-25T05:53:51.455862+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:53:51.455862+0000","last_undegraded":"2024-04-25T05:53:51.455862+0000","last_fullsized":"2024-04-25T05:53:51.455862+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:52:39.432428+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.432428+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:58.290428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":1132,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456239+0000","last_change":"2024-04-25T05:52:41.458972+0000","last_active":"2024-04-25T05:53:51.456239+0000","last_peered":"2024-04-25T05:53:51.456239+0000","last_clean":"2024-04-25T05:53:51.456239+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:53:51.456239+0000","last_undegraded":"2024-04-25T05:53:51.456239+0000","last_fullsized":"2024-04-25T05:53:51.456239+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:52:41.458853+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.458853+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.006386+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":441,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453693+0000","last_change":"2024-04-25T05:51:40.105669+0000","last_active":"2024-04-25T05:53:51.453693+0000","last_peered":"2024-04-25T05:53:51.453693+0000","last_clean":"2024-04-25T05:53:51.453693+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:53:51.453693+0000","last_undegraded":"2024-04-25T05:53:51.453693+0000","last_fullsized":"2024-04-25T05:53:51.453693+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:31.543423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:31.543423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1086,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454295+0000","last_change":"2024-04-25T05:53:25.688414+0000","last_active":"2024-04-25T05:53:51.454295+0000","last_peered":"2024-04-25T05:53:51.454295+0000","last_clean":"2024-04-25T05:53:51.454295+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:53:51.454295+0000","last_undegraded":"2024-04-25T05:53:51.454295+0000","last_fullsized":"2024-04-25T05:53:51.454295+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.688336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.688336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.070699+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"573'15469","reported_seq":28453,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.646871+0000","last_change":"2024-04-25T05:52:45.137759+0000","last_active":"2024-04-25T05:53:53.646871+0000","last_peered":"2024-04-25T05:53:53.646871+0000","last_clean":"2024-04-25T05:53:53.646871+0000","last_became_active":"2024-04-25T05:52:45.095486+0000","last_became_peered":"2024-04-25T05:52:45.095486+0000","last_unstale":"2024-04-25T05:53:53.646871+0000","last_undegraded":"2024-04-25T05:53:53.646871+0000","last_fullsized":"2024-04-25T05:53:53.646871+0000","mapping_epoch":541,"log_start":"517'9958","ondisk_log_start":"517'9958","created":456,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":5511,"log_dups_size":0,"ondisk_log_size":5511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":559,"num_object_clones":0,"num_object_copies":1677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15469,"num_write_kb":512896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6364,"num_bytes_recovered":417071104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":449,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452759+0000","last_change":"2024-04-25T05:53:28.416788+0000","last_active":"2024-04-25T05:53:51.452759+0000","last_peered":"2024-04-25T05:53:51.452759+0000","last_clean":"2024-04-25T05:53:51.452759+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:53:51.452759+0000","last_undegraded":"2024-04-25T05:53:51.452759+0000","last_fullsized":"2024-04-25T05:53:51.452759+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.416661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.416661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.932624+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":502,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444189+0000","last_change":"2024-04-25T05:53:31.936255+0000","last_active":"2024-04-25T05:53:51.444189+0000","last_peered":"2024-04-25T05:53:51.444189+0000","last_clean":"2024-04-25T05:53:51.444189+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:53:51.444189+0000","last_undegraded":"2024-04-25T05:53:51.444189+0000","last_fullsized":"2024-04-25T05:53:51.444189+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:53:31.936104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.936104+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:44.183145+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1085,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456442+0000","last_change":"2024-04-25T05:52:39.438835+0000","last_active":"2024-04-25T05:53:51.456442+0000","last_peered":"2024-04-25T05:53:51.456442+0000","last_clean":"2024-04-25T05:53:51.456442+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:53:51.456442+0000","last_undegraded":"2024-04-25T05:53:51.456442+0000","last_fullsized":"2024-04-25T05:53:51.456442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:39.438700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.438700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.055139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1062,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454850+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:53:51.454850+0000","last_peered":"2024-04-25T05:53:51.454850+0000","last_clean":"2024-04-25T05:53:51.454850+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:53:51.454850+0000","last_undegraded":"2024-04-25T05:53:51.454850+0000","last_fullsized":"2024-04-25T05:53:51.454850+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"573'15652","reported_seq":15978,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.233653+0000","last_change":"2024-04-25T05:52:02.016861+0000","last_active":"2024-04-25T05:53:53.233653+0000","last_peered":"2024-04-25T05:53:53.233653+0000","last_clean":"2024-04-25T05:53:53.233653+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:53:53.233653+0000","last_undegraded":"2024-04-25T05:53:53.233653+0000","last_fullsized":"2024-04-25T05:53:53.233653+0000","mapping_epoch":456,"log_start":"519'10210","ondisk_log_start":"519'10210","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8841","last_scrub_stamp":"2024-04-25T05:52:02.016648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.016648+0000","objects_scrubbed":7398,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":522,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15652,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1352,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454547+0000","last_change":"2024-04-25T05:52:50.387148+0000","last_active":"2024-04-25T05:53:51.454547+0000","last_peered":"2024-04-25T05:53:51.454547+0000","last_clean":"2024-04-25T05:53:51.454547+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:53:51.454547+0000","last_undegraded":"2024-04-25T05:53:51.454547+0000","last_fullsized":"2024-04-25T05:53:51.454547+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_deep_scrub":"208'11","last_deep_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.387023+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.717658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":432,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452201+0000","last_change":"2024-04-25T05:52:06.018611+0000","last_active":"2024-04-25T05:53:51.452201+0000","last_peered":"2024-04-25T05:53:51.452201+0000","last_clean":"2024-04-25T05:53:51.452201+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:53:51.452201+0000","last_undegraded":"2024-04-25T05:53:51.452201+0000","last_fullsized":"2024-04-25T05:53:51.452201+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:52:06.018248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:06.018248+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1145,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456666+0000","last_change":"2024-04-25T05:53:33.672727+0000","last_active":"2024-04-25T05:53:51.456666+0000","last_peered":"2024-04-25T05:53:51.456666+0000","last_clean":"2024-04-25T05:53:51.456666+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:53:51.456666+0000","last_undegraded":"2024-04-25T05:53:51.456666+0000","last_fullsized":"2024-04-25T05:53:51.456666+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:53:33.672600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.672600+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:51.282306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":410,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452723+0000","last_change":"2024-04-25T05:51:56.071814+0000","last_active":"2024-04-25T05:53:51.452723+0000","last_peered":"2024-04-25T05:53:51.452723+0000","last_clean":"2024-04-25T05:53:51.452723+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:53:51.452723+0000","last_undegraded":"2024-04-25T05:53:51.452723+0000","last_fullsized":"2024-04-25T05:53:51.452723+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:56.071706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:56.071706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":1096,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455504+0000","last_change":"2024-04-25T05:51:28.373915+0000","last_active":"2024-04-25T05:53:51.455504+0000","last_peered":"2024-04-25T05:53:51.455504+0000","last_clean":"2024-04-25T05:53:51.455504+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:53:51.455504+0000","last_undegraded":"2024-04-25T05:53:51.455504+0000","last_fullsized":"2024-04-25T05:53:51.455504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:51:28.373733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.373733+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":845,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454404+0000","last_change":"2024-04-25T05:52:50.354986+0000","last_active":"2024-04-25T05:53:51.454404+0000","last_peered":"2024-04-25T05:53:51.454404+0000","last_clean":"2024-04-25T05:53:51.454404+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:53:51.454404+0000","last_undegraded":"2024-04-25T05:53:51.454404+0000","last_fullsized":"2024-04-25T05:53:51.454404+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:50.354868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.354868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.086443+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453154+0000","last_change":"2024-04-25T05:52:44.084490+0000","last_active":"2024-04-25T05:53:51.453154+0000","last_peered":"2024-04-25T05:53:51.453154+0000","last_clean":"2024-04-25T05:53:51.453154+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:53:51.453154+0000","last_undegraded":"2024-04-25T05:53:51.453154+0000","last_fullsized":"2024-04-25T05:53:51.453154+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.084411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.084411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.972942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1093,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454543+0000","last_change":"2024-04-25T05:53:48.751193+0000","last_active":"2024-04-25T05:53:51.454543+0000","last_peered":"2024-04-25T05:53:51.454543+0000","last_clean":"2024-04-25T05:53:51.454543+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:53:51.454543+0000","last_undegraded":"2024-04-25T05:53:51.454543+0000","last_fullsized":"2024-04-25T05:53:51.454543+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:48.751077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:48.751077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.118419+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":183,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448352+0000","last_change":"2024-04-25T05:53:32.185728+0000","last_active":"2024-04-25T05:53:51.448352+0000","last_peered":"2024-04-25T05:53:51.448352+0000","last_clean":"2024-04-25T05:53:51.448352+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:53:51.448352+0000","last_undegraded":"2024-04-25T05:53:51.448352+0000","last_fullsized":"2024-04-25T05:53:51.448352+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:53:32.185584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:32.185584+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.846897+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1103,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454776+0000","last_change":"2024-04-25T05:52:44.356015+0000","last_active":"2024-04-25T05:53:51.454776+0000","last_peered":"2024-04-25T05:53:51.454776+0000","last_clean":"2024-04-25T05:53:51.454776+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:53:51.454776+0000","last_undegraded":"2024-04-25T05:53:51.454776+0000","last_fullsized":"2024-04-25T05:53:51.454776+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.355921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.355921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.000445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1054,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455343+0000","last_change":"2024-04-25T05:51:30.313765+0000","last_active":"2024-04-25T05:53:51.455343+0000","last_peered":"2024-04-25T05:53:51.455343+0000","last_clean":"2024-04-25T05:53:51.455343+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:53:51.455343+0000","last_undegraded":"2024-04-25T05:53:51.455343+0000","last_fullsized":"2024-04-25T05:53:51.455343+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.313655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.313655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":429,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460875+0000","last_change":"2024-04-25T05:53:23.378618+0000","last_active":"2024-04-25T05:53:51.460875+0000","last_peered":"2024-04-25T05:53:51.460875+0000","last_clean":"2024-04-25T05:53:51.460875+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:53:51.460875+0000","last_undegraded":"2024-04-25T05:53:51.460875+0000","last_fullsized":"2024-04-25T05:53:51.460875+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.378498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.378498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.114889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":424,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453171+0000","last_change":"2024-04-25T05:53:26.982110+0000","last_active":"2024-04-25T05:53:51.453171+0000","last_peered":"2024-04-25T05:53:51.453171+0000","last_clean":"2024-04-25T05:53:51.453171+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:53:51.453171+0000","last_undegraded":"2024-04-25T05:53:51.453171+0000","last_fullsized":"2024-04-25T05:53:51.453171+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:26.981810+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.981810+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.388901+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":169,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446885+0000","last_change":"2024-04-25T05:53:22.083599+0000","last_active":"2024-04-25T05:53:51.446885+0000","last_peered":"2024-04-25T05:53:51.446885+0000","last_clean":"2024-04-25T05:53:51.446885+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:53:51.446885+0000","last_undegraded":"2024-04-25T05:53:51.446885+0000","last_fullsized":"2024-04-25T05:53:51.446885+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:22.083459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.083459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.639562+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1322,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456262+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:53:51.456262+0000","last_peered":"2024-04-25T05:53:51.456262+0000","last_clean":"2024-04-25T05:53:51.456262+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:53:51.456262+0000","last_undegraded":"2024-04-25T05:53:51.456262+0000","last_fullsized":"2024-04-25T05:53:51.456262+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":624,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445085+0000","last_change":"2024-04-25T05:52:46.212477+0000","last_active":"2024-04-25T05:53:51.445085+0000","last_peered":"2024-04-25T05:53:51.445085+0000","last_clean":"2024-04-25T05:53:51.445085+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:53:51.445085+0000","last_undegraded":"2024-04-25T05:53:51.445085+0000","last_fullsized":"2024-04-25T05:53:51.445085+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.212334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.212334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:15.795169+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":1120,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454342+0000","last_change":"2024-04-25T05:53:25.647571+0000","last_active":"2024-04-25T05:53:51.454342+0000","last_peered":"2024-04-25T05:53:51.454342+0000","last_clean":"2024-04-25T05:53:51.454342+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:53:51.454342+0000","last_undegraded":"2024-04-25T05:53:51.454342+0000","last_fullsized":"2024-04-25T05:53:51.454342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:53:25.647413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.647413+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.576268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1097,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454854+0000","last_change":"2024-04-25T05:53:30.687157+0000","last_active":"2024-04-25T05:53:51.454854+0000","last_peered":"2024-04-25T05:53:51.454854+0000","last_clean":"2024-04-25T05:53:51.454854+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:53:51.454854+0000","last_undegraded":"2024-04-25T05:53:51.454854+0000","last_fullsized":"2024-04-25T05:53:51.454854+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.687046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.687046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.184397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":505,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453294+0000","last_change":"2024-04-25T05:51:53.967723+0000","last_active":"2024-04-25T05:53:51.453294+0000","last_peered":"2024-04-25T05:53:51.453294+0000","last_clean":"2024-04-25T05:53:51.453294+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:53:51.453294+0000","last_undegraded":"2024-04-25T05:53:51.453294+0000","last_fullsized":"2024-04-25T05:53:51.453294+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'14","last_scrub_stamp":"2024-04-25T05:51:53.967616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:53.967616+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":161,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448488+0000","last_change":"2024-04-25T05:51:40.110100+0000","last_active":"2024-04-25T05:53:51.448488+0000","last_peered":"2024-04-25T05:53:51.448488+0000","last_clean":"2024-04-25T05:53:51.448488+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:53:51.448488+0000","last_undegraded":"2024-04-25T05:53:51.448488+0000","last_fullsized":"2024-04-25T05:53:51.448488+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.972698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.972698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1224,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444091+0000","last_change":"2024-04-25T05:53:21.989605+0000","last_active":"2024-04-25T05:53:51.444091+0000","last_peered":"2024-04-25T05:53:51.444091+0000","last_clean":"2024-04-25T05:53:51.444091+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:53:51.444091+0000","last_undegraded":"2024-04-25T05:53:51.444091+0000","last_fullsized":"2024-04-25T05:53:51.444091+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:53:21.989507+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.989507+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.602540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":671,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446496+0000","last_change":"2024-04-25T05:52:05.839366+0000","last_active":"2024-04-25T05:53:51.446496+0000","last_peered":"2024-04-25T05:53:51.446496+0000","last_clean":"2024-04-25T05:53:51.446496+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:53:51.446496+0000","last_undegraded":"2024-04-25T05:53:51.446496+0000","last_fullsized":"2024-04-25T05:53:51.446496+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:52:05.839241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:05.839241+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1084,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446632+0000","last_change":"2024-04-25T05:53:25.341898+0000","last_active":"2024-04-25T05:53:51.446632+0000","last_peered":"2024-04-25T05:53:51.446632+0000","last_clean":"2024-04-25T05:53:51.446632+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:53:51.446632+0000","last_undegraded":"2024-04-25T05:53:51.446632+0000","last_fullsized":"2024-04-25T05:53:51.446632+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.334571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.334571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.518338+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1100,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456068+0000","last_change":"2024-04-25T05:51:40.083901+0000","last_active":"2024-04-25T05:53:51.456068+0000","last_peered":"2024-04-25T05:53:51.456068+0000","last_clean":"2024-04-25T05:53:51.456068+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:53:51.456068+0000","last_undegraded":"2024-04-25T05:53:51.456068+0000","last_fullsized":"2024-04-25T05:53:51.456068+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":631,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448056+0000","last_change":"2024-04-25T05:51:23.925876+0000","last_active":"2024-04-25T05:53:51.448056+0000","last_peered":"2024-04-25T05:53:51.448056+0000","last_clean":"2024-04-25T05:53:51.448056+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:53:51.448056+0000","last_undegraded":"2024-04-25T05:53:51.448056+0000","last_fullsized":"2024-04-25T05:53:51.448056+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:51:23.925746+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.925746+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1085,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454640+0000","last_change":"2024-04-25T05:52:45.364101+0000","last_active":"2024-04-25T05:53:51.454640+0000","last_peered":"2024-04-25T05:53:51.454640+0000","last_clean":"2024-04-25T05:53:51.454640+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:53:51.454640+0000","last_undegraded":"2024-04-25T05:53:51.454640+0000","last_fullsized":"2024-04-25T05:53:51.454640+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:45.363990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.363990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.943288+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1084,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454757+0000","last_change":"2024-04-25T05:51:40.102243+0000","last_active":"2024-04-25T05:53:51.454757+0000","last_peered":"2024-04-25T05:53:51.454757+0000","last_clean":"2024-04-25T05:53:51.454757+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:53:51.454757+0000","last_undegraded":"2024-04-25T05:53:51.454757+0000","last_fullsized":"2024-04-25T05:53:51.454757+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":169,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:52.262087+0000","last_change":"2024-04-25T05:53:52.262152+0000","last_active":"2024-04-25T05:53:52.262087+0000","last_peered":"2024-04-25T05:53:52.262087+0000","last_clean":"2024-04-25T05:53:52.262087+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:53:52.262087+0000","last_undegraded":"2024-04-25T05:53:52.262087+0000","last_fullsized":"2024-04-25T05:53:52.262087+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:52.261997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:52.261997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.217729+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":1155,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456311+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:53:51.456311+0000","last_peered":"2024-04-25T05:53:51.456311+0000","last_clean":"2024-04-25T05:53:51.456311+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:53:51.456311+0000","last_undegraded":"2024-04-25T05:53:51.456311+0000","last_fullsized":"2024-04-25T05:53:51.456311+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453036+0000","last_change":"2024-04-25T05:52:42.094827+0000","last_active":"2024-04-25T05:53:51.453036+0000","last_peered":"2024-04-25T05:53:51.453036+0000","last_clean":"2024-04-25T05:53:51.453036+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:53:51.453036+0000","last_undegraded":"2024-04-25T05:53:51.453036+0000","last_fullsized":"2024-04-25T05:53:51.453036+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.094672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.094672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.738692+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1095,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454701+0000","last_change":"2024-04-25T05:52:53.417453+0000","last_active":"2024-04-25T05:53:51.454701+0000","last_peered":"2024-04-25T05:53:51.454701+0000","last_clean":"2024-04-25T05:53:51.454701+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:53:51.454701+0000","last_undegraded":"2024-04-25T05:53:51.454701+0000","last_fullsized":"2024-04-25T05:53:51.454701+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.417346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.417346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.686711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1092,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455031+0000","last_change":"2024-04-25T05:51:40.104333+0000","last_active":"2024-04-25T05:53:51.455031+0000","last_peered":"2024-04-25T05:53:51.455031+0000","last_clean":"2024-04-25T05:53:51.455031+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:53:51.455031+0000","last_undegraded":"2024-04-25T05:53:51.455031+0000","last_fullsized":"2024-04-25T05:53:51.455031+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":1094,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460584+0000","last_change":"2024-04-25T05:48:23.123503+0000","last_active":"2024-04-25T05:53:51.460584+0000","last_peered":"2024-04-25T05:53:51.460584+0000","last_clean":"2024-04-25T05:53:51.460584+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:53:51.460584+0000","last_undegraded":"2024-04-25T05:53:51.460584+0000","last_fullsized":"2024-04-25T05:53:51.460584+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:48:23.123371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:48:23.123371+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":169,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446251+0000","last_change":"2024-04-25T05:53:50.201657+0000","last_active":"2024-04-25T05:53:51.446251+0000","last_peered":"2024-04-25T05:53:51.446251+0000","last_clean":"2024-04-25T05:53:51.446251+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:53:51.446251+0000","last_undegraded":"2024-04-25T05:53:51.446251+0000","last_fullsized":"2024-04-25T05:53:51.446251+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:50.201525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.201525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:57.278709+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":801,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456322+0000","last_change":"2024-04-25T05:52:44.398454+0000","last_active":"2024-04-25T05:53:51.456322+0000","last_peered":"2024-04-25T05:53:51.456322+0000","last_clean":"2024-04-25T05:53:51.456322+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:53:51.456322+0000","last_undegraded":"2024-04-25T05:53:51.456322+0000","last_fullsized":"2024-04-25T05:53:51.456322+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.398379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.398379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.323065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":1086,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455607+0000","last_change":"2024-04-25T05:51:57.236443+0000","last_active":"2024-04-25T05:53:51.455607+0000","last_peered":"2024-04-25T05:53:51.455607+0000","last_clean":"2024-04-25T05:53:51.455607+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:53:51.455607+0000","last_undegraded":"2024-04-25T05:53:51.455607+0000","last_fullsized":"2024-04-25T05:53:51.455607+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:51:57.236311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:57.236311+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":921,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456157+0000","last_change":"2024-04-25T05:53:22.290678+0000","last_active":"2024-04-25T05:53:51.456157+0000","last_peered":"2024-04-25T05:53:51.456157+0000","last_clean":"2024-04-25T05:53:51.456157+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:53:51.456157+0000","last_undegraded":"2024-04-25T05:53:51.456157+0000","last_fullsized":"2024-04-25T05:53:51.456157+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:53:22.290465+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.290465+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.697555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1100,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460407+0000","last_change":"2024-04-25T05:53:28.655967+0000","last_active":"2024-04-25T05:53:51.460407+0000","last_peered":"2024-04-25T05:53:51.460407+0000","last_clean":"2024-04-25T05:53:51.460407+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:53:51.460407+0000","last_undegraded":"2024-04-25T05:53:51.460407+0000","last_fullsized":"2024-04-25T05:53:51.460407+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.655843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.655843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.413792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":170,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448094+0000","last_change":"2024-04-25T05:53:34.135232+0000","last_active":"2024-04-25T05:53:51.448094+0000","last_peered":"2024-04-25T05:53:51.448094+0000","last_clean":"2024-04-25T05:53:51.448094+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:53:51.448094+0000","last_undegraded":"2024-04-25T05:53:51.448094+0000","last_fullsized":"2024-04-25T05:53:51.448094+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.134907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.134907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.959435+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1082,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:53.649518+0000","last_change":"2024-04-25T05:53:53.649554+0000","last_active":"2024-04-25T05:53:53.649518+0000","last_peered":"2024-04-25T05:53:53.649518+0000","last_clean":"2024-04-25T05:53:53.649518+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:53:53.649518+0000","last_undegraded":"2024-04-25T05:53:53.649518+0000","last_fullsized":"2024-04-25T05:53:53.649518+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:53.649439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.649439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:22.147543+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":176,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.448367+0000","last_change":"2024-04-25T05:51:40.147090+0000","last_active":"2024-04-25T05:53:51.448367+0000","last_peered":"2024-04-25T05:53:51.448367+0000","last_clean":"2024-04-25T05:53:51.448367+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:53:51.448367+0000","last_undegraded":"2024-04-25T05:53:51.448367+0000","last_fullsized":"2024-04-25T05:53:51.448367+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1078,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446276+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:53:51.446276+0000","last_peered":"2024-04-25T05:53:51.446276+0000","last_clean":"2024-04-25T05:53:51.446276+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:53:51.446276+0000","last_undegraded":"2024-04-25T05:53:51.446276+0000","last_fullsized":"2024-04-25T05:53:51.446276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":985,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455944+0000","last_change":"2024-04-25T05:52:53.341410+0000","last_active":"2024-04-25T05:53:51.455944+0000","last_peered":"2024-04-25T05:53:51.455944+0000","last_clean":"2024-04-25T05:53:51.455944+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:53:51.455944+0000","last_undegraded":"2024-04-25T05:53:51.455944+0000","last_fullsized":"2024-04-25T05:53:51.455944+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:52:53.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.341283+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:03.764198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"507'2","reported_seq":1114,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455015+0000","last_change":"2024-04-25T05:52:08.192825+0000","last_active":"2024-04-25T05:53:51.455015+0000","last_peered":"2024-04-25T05:53:51.455015+0000","last_clean":"2024-04-25T05:53:51.455015+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:53:51.455015+0000","last_undegraded":"2024-04-25T05:53:51.455015+0000","last_fullsized":"2024-04-25T05:53:51.455015+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.192693+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":1101,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454708+0000","last_change":"2024-04-25T05:51:18.436527+0000","last_active":"2024-04-25T05:53:51.454708+0000","last_peered":"2024-04-25T05:53:51.454708+0000","last_clean":"2024-04-25T05:53:51.454708+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:53:51.454708+0000","last_undegraded":"2024-04-25T05:53:51.454708+0000","last_fullsized":"2024-04-25T05:53:51.454708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:18.436370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.436370+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1119,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444435+0000","last_change":"2024-04-25T05:53:24.969033+0000","last_active":"2024-04-25T05:53:51.444435+0000","last_peered":"2024-04-25T05:53:51.444435+0000","last_clean":"2024-04-25T05:53:51.444435+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:53:51.444435+0000","last_undegraded":"2024-04-25T05:53:51.444435+0000","last_fullsized":"2024-04-25T05:53:51.444435+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:24.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:46.507723+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":169,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455201+0000","last_change":"2024-04-25T05:52:35.374907+0000","last_active":"2024-04-25T05:53:51.455201+0000","last_peered":"2024-04-25T05:53:51.455201+0000","last_clean":"2024-04-25T05:53:51.455201+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:53:51.455201+0000","last_undegraded":"2024-04-25T05:53:51.455201+0000","last_fullsized":"2024-04-25T05:53:51.455201+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:35.374829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.374829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1075,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455421+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:53:51.455421+0000","last_peered":"2024-04-25T05:53:51.455421+0000","last_clean":"2024-04-25T05:53:51.455421+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:53:51.455421+0000","last_undegraded":"2024-04-25T05:53:51.455421+0000","last_fullsized":"2024-04-25T05:53:51.455421+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":453,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.460764+0000","last_change":"2024-04-25T05:53:25.409053+0000","last_active":"2024-04-25T05:53:51.460764+0000","last_peered":"2024-04-25T05:53:51.460764+0000","last_clean":"2024-04-25T05:53:51.460764+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:53:51.460764+0000","last_undegraded":"2024-04-25T05:53:51.460764+0000","last_fullsized":"2024-04-25T05:53:51.460764+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:25.408390+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.408390+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:25.958321+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":161,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447718+0000","last_change":"2024-04-25T05:51:40.101375+0000","last_active":"2024-04-25T05:53:51.447718+0000","last_peered":"2024-04-25T05:53:51.447718+0000","last_clean":"2024-04-25T05:53:51.447718+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:53:51.447718+0000","last_undegraded":"2024-04-25T05:53:51.447718+0000","last_fullsized":"2024-04-25T05:53:51.447718+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.835919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.835919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1098,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447693+0000","last_change":"2024-04-25T05:51:40.109523+0000","last_active":"2024-04-25T05:53:51.447693+0000","last_peered":"2024-04-25T05:53:51.447693+0000","last_clean":"2024-04-25T05:53:51.447693+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:53:51.447693+0000","last_undegraded":"2024-04-25T05:53:51.447693+0000","last_fullsized":"2024-04-25T05:53:51.447693+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.989325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.989325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1115,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.445608+0000","last_change":"2024-04-25T05:53:23.056922+0000","last_active":"2024-04-25T05:53:51.445608+0000","last_peered":"2024-04-25T05:53:51.445608+0000","last_clean":"2024-04-25T05:53:51.445608+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:53:51.445608+0000","last_undegraded":"2024-04-25T05:53:51.445608+0000","last_fullsized":"2024-04-25T05:53:51.445608+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.056816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.056816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.493056+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":593,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444189+0000","last_change":"2024-04-25T05:53:51.040346+0000","last_active":"2024-04-25T05:53:51.444189+0000","last_peered":"2024-04-25T05:53:51.444189+0000","last_clean":"2024-04-25T05:53:51.444189+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:53:51.444189+0000","last_undegraded":"2024-04-25T05:53:51.444189+0000","last_fullsized":"2024-04-25T05:53:51.444189+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:53:51.040191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:51.040191+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:13.024810+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":411,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453286+0000","last_change":"2024-04-25T05:52:47.057845+0000","last_active":"2024-04-25T05:53:51.453286+0000","last_peered":"2024-04-25T05:53:51.453286+0000","last_clean":"2024-04-25T05:53:51.453286+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:53:51.453286+0000","last_undegraded":"2024-04-25T05:53:51.453286+0000","last_fullsized":"2024-04-25T05:53:51.453286+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.057728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.968343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1074,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455415+0000","last_change":"2024-04-25T05:51:54.167227+0000","last_active":"2024-04-25T05:53:51.455415+0000","last_peered":"2024-04-25T05:53:51.455415+0000","last_clean":"2024-04-25T05:53:51.455415+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:53:51.455415+0000","last_undegraded":"2024-04-25T05:53:51.455415+0000","last_fullsized":"2024-04-25T05:53:51.455415+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:54.167105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:54.167105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":412,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452488+0000","last_change":"2024-04-25T05:53:30.164095+0000","last_active":"2024-04-25T05:53:51.452488+0000","last_peered":"2024-04-25T05:53:51.452488+0000","last_clean":"2024-04-25T05:53:51.452488+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:53:51.452488+0000","last_undegraded":"2024-04-25T05:53:51.452488+0000","last_fullsized":"2024-04-25T05:53:51.452488+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.163971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.163971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.864542+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":899,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444347+0000","last_change":"2024-04-25T05:52:02.822759+0000","last_active":"2024-04-25T05:53:51.444347+0000","last_peered":"2024-04-25T05:53:51.444347+0000","last_clean":"2024-04-25T05:53:51.444347+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:53:51.444347+0000","last_undegraded":"2024-04-25T05:53:51.444347+0000","last_fullsized":"2024-04-25T05:53:51.444347+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:52:02.822618+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.822618+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":411,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.453125+0000","last_change":"2024-04-25T05:52:46.070334+0000","last_active":"2024-04-25T05:53:51.453125+0000","last_peered":"2024-04-25T05:53:51.453125+0000","last_clean":"2024-04-25T05:53:51.453125+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:53:51.453125+0000","last_undegraded":"2024-04-25T05:53:51.453125+0000","last_fullsized":"2024-04-25T05:53:51.453125+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.070183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.070183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.824200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":409,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.452446+0000","last_change":"2024-04-25T05:51:40.084285+0000","last_active":"2024-04-25T05:53:51.452446+0000","last_peered":"2024-04-25T05:53:51.452446+0000","last_clean":"2024-04-25T05:53:51.452446+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:53:51.452446+0000","last_undegraded":"2024-04-25T05:53:51.452446+0000","last_fullsized":"2024-04-25T05:53:51.452446+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:28.941496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.941496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":161,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.447236+0000","last_change":"2024-04-25T05:51:40.110384+0000","last_active":"2024-04-25T05:53:51.447236+0000","last_peered":"2024-04-25T05:53:51.447236+0000","last_clean":"2024-04-25T05:53:51.447236+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:53:51.447236+0000","last_undegraded":"2024-04-25T05:53:51.447236+0000","last_fullsized":"2024-04-25T05:53:51.447236+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:50:08.944412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1389,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.456346+0000","last_change":"2024-04-25T05:51:40.151139+0000","last_active":"2024-04-25T05:53:51.456346+0000","last_peered":"2024-04-25T05:53:51.456346+0000","last_clean":"2024-04-25T05:53:51.456346+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:53:51.456346+0000","last_undegraded":"2024-04-25T05:53:51.456346+0000","last_fullsized":"2024-04-25T05:53:51.456346+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1095,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.455674+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:53:51.455674+0000","last_peered":"2024-04-25T05:53:51.455674+0000","last_clean":"2024-04-25T05:53:51.455674+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:53:51.455674+0000","last_undegraded":"2024-04-25T05:53:51.455674+0000","last_fullsized":"2024-04-25T05:53:51.455674+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":640,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.443260+0000","last_change":"2024-04-25T05:52:51.042784+0000","last_active":"2024-04-25T05:53:51.443260+0000","last_peered":"2024-04-25T05:53:51.443260+0000","last_clean":"2024-04-25T05:53:51.443260+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:53:51.443260+0000","last_undegraded":"2024-04-25T05:53:51.443260+0000","last_fullsized":"2024-04-25T05:53:51.443260+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.042665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.042665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:53.131222+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1061,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.454802+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:53:51.454802+0000","last_peered":"2024-04-25T05:53:51.454802+0000","last_clean":"2024-04-25T05:53:51.454802+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:53:51.454802+0000","last_undegraded":"2024-04-25T05:53:51.454802+0000","last_fullsized":"2024-04-25T05:53:51.454802+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":1176,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.442800+0000","last_change":"2024-04-25T05:53:21.029300+0000","last_active":"2024-04-25T05:53:51.442800+0000","last_peered":"2024-04-25T05:53:51.442800+0000","last_clean":"2024-04-25T05:53:51.442800+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:53:51.442800+0000","last_undegraded":"2024-04-25T05:53:51.442800+0000","last_fullsized":"2024-04-25T05:53:51.442800+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:21.029139+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.029139+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:30.616379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":170,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446335+0000","last_change":"2024-04-25T05:53:33.165542+0000","last_active":"2024-04-25T05:53:51.446335+0000","last_peered":"2024-04-25T05:53:51.446335+0000","last_clean":"2024-04-25T05:53:51.446335+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:53:51.446335+0000","last_undegraded":"2024-04-25T05:53:51.446335+0000","last_fullsized":"2024-04-25T05:53:51.446335+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.165409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.165409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:03.084305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1087,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.446380+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:53:51.446380+0000","last_peered":"2024-04-25T05:53:51.446380+0000","last_clean":"2024-04-25T05:53:51.446380+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:53:51.446380+0000","last_undegraded":"2024-04-25T05:53:51.446380+0000","last_fullsized":"2024-04-25T05:53:51.446380+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":616,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-04-25T05:53:51.444098+0000","last_change":"2024-04-25T05:51:51.705620+0000","last_active":"2024-04-25T05:53:51.444098+0000","last_peered":"2024-04-25T05:53:51.444098+0000","last_clean":"2024-04-25T05:53:51.444098+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:53:51.444098+0000","last_undegraded":"2024-04-25T05:53:51.444098+0000","last_fullsized":"2024-04-25T05:53:51.444098+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:51.705348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:51.705348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":554565656,"num_objects":8463,"num_object_clones":0,"num_object_copies":25389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8463,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252111,"num_write_kb":8338305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28230,"num_bytes_recovered":1850015744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2161848320,"data_stored":2161836104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80509,"ondisk_log_size":80509,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1175,"num_read_kb":1170,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070708,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350120,"kb_used_data":420704,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92395032,"statfs":{"total":95995035648,"available":94612512768,"internally_reserved":0,"allocated":430800896,"data_stored":430119575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":951712202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365048,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154196,"kb_used_data":288648,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92590956,"statfs":{"total":95995035648,"available":94813138944,"internally_reserved":0,"allocated":295575552,"data_stored":294917623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":886307270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575121,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350580,"kb_used_data":250348,"kb_used_omap":1,"kb_used_meta":1100158,"kb_avail":92394572,"statfs":{"total":95995035648,"available":94612041728,"internally_reserved":0,"allocated":256356352,"data_stored":255661451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1126562245},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959712,"kb_used_data":184404,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92785440,"statfs":{"total":95995035648,"available":95012290560,"internally_reserved":0,"allocated":188829696,"data_stored":188158820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411596,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966424,"kb_used_data":236940,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92778728,"statfs":{"total":95995035648,"available":95005417472,"internally_reserved":0,"allocated":242626560,"data_stored":241964031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149958,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1467180,"kb_used_data":245732,"kb_used_omap":1,"kb_used_meta":1221438,"kb_avail":92277972,"statfs":{"total":95995035648,"available":94492643328,"internally_reserved":0,"allocated":251629568,"data_stored":250942949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1250752967},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986433,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1249132,"kb_used_data":437732,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92496020,"statfs":{"total":95995035648,"available":94715924480,"internally_reserved":0,"allocated":448237568,"data_stored":447551149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692093,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776276,"kb_used_data":149516,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92968876,"statfs":{"total":95995035648,"available":95200129024,"internally_reserved":0,"allocated":153104384,"data_stored":152442355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179961856,"data_stored":179961856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250937344,"data_stored":250937344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":226365440,"data_stored":226361368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223739904,"data_stored":223739904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":439685120,"data_stored":439681048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":146735104,"data_stored":146735104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":286199808,"data_stored":286195736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":408223744,"data_stored":408223744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:53:57.799 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:53:57.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T05:53:57.800 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-04-25T05:53:57.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:53:57.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: from='client.26594 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: pgmap v1082: 158 pgs: 158 active+clean; 529 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: osdmap e576: 8 total, 8 up, 8 in 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:53:57.901 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:57 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:53:57.945 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:53:57.975 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:57.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:53:58.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T05:53:58.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:58.163 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:58.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.316 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:58.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:58.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:58.453 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:58.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:58.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:53:58.729 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:53:58.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 5.d deep-scrub starts 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 5.d deep-scrub ok 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: osdmap e577: 8 total, 8 up, 8 in 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:53:58.771 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 5.d deep-scrub starts 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 5.d deep-scrub ok 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: osdmap e577: 8 total, 8 up, 8 in 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:53:58.772 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:58 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:53:58.831 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:58.864 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:58.966 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:58.993 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:58.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 5.d deep-scrub starts 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 5.d deep-scrub ok 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: osdmap e577: 8 total, 8 up, 8 in 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:53:58.999 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:58 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:53:59.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:53:59.134 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:53:59.236 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:59.238 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:257970 11.6 11:6b72eee5:::benchmark_data_smithi043_179003_object127681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:59.251940+0000", 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.02759062, 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.027627453999999999, 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257970 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251940+0000", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251940+0000", 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251942+0000", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.809e-06 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251948+0000", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2469999999999997e-06 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251959+0000", 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0978999999999999e-05 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.251968+0000", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5590000000000006e-06 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.252036+0000", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8544e-05 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.252166+0000", 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129694 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.252295+0000", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012956799999999999 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.256804+0000", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045085300000000002 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.263646+0000", 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0068422860000000004 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:257977 11.6 11:6d959bd4:::benchmark_data_smithi043_179003_object127688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:59.259452+0000", 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.020078806000000001, 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.020214807000000001, 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257977 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259452+0000", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259452+0000", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259454+0000", 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.99e-06 2024-04-25T05:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259461+0000", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.486e-06 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259489+0000", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7691999999999998e-05 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259499+0000", 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0229e-05 2024-04-25T05:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.259717+0000", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000217376 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.260150+0000", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00043337900000000001 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.260667+0000", 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00051709499999999997 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.265413+0000", 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047460310000000004 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.265488+0000", 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4388999999999995e-05 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:257980 11.6 11:64a92b01:::benchmark_data_smithi043_179003_object127691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-04-25T05:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:59.271459+0000", 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0080714940000000002, 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0082976049999999996, 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257980 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:53:59.294 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271459+0000", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271459+0000", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271461+0000", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1169999999999998e-06 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:53:59.295 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271469+0000", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2899999999999997e-06 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271487+0000", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8555000000000001e-05 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271496+0000", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6049999999999998e-06 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:53:59.296 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271724+0000", 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000228657 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.272137+0000", 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00041213300000000002 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.272610+0000", 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00047370799999999998 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:257983 11.6 11:6c132c29:::benchmark_data_smithi043_179003_object127694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:59.271517+0000", 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 0.0080136970000000002, 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0083211239999999992, 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "waiting for sub ops", 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 257983 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:53:59.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271517+0000", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271517+0000", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271518+0000", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.13e-07 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:53:59.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271521+0000", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.281e-06 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271528+0000", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0609999999999998e-06 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.271535+0000", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4800000000000004e-06 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.275848+0000", 2024-04-25T05:53:59.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.004312766 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.276232+0000", 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00038368400000000002 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:59.276690+0000", 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00045804 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:53:59.301 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-25T05:53:59.302 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 4 2024-04-25T05:53:59.302 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:53:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:59.417 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:59.444 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:53:59.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:53:59.582 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:53:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[27010]: pgmap v1085: 158 pgs: 158 active+clean; 487 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-25T05:53:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[27010]: 11.1 scrub starts 2024-04-25T05:53:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[27010]: 11.1 scrub ok 2024-04-25T05:53:59.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[27010]: 6.12 scrub starts 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[27010]: 6.12 scrub ok 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[37492]: pgmap v1085: 158 pgs: 158 active+clean; 487 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[37492]: 11.1 scrub starts 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[37492]: 11.1 scrub ok 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[37492]: 6.12 scrub starts 2024-04-25T05:53:59.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-mon[37492]: 6.12 scrub ok 2024-04-25T05:53:59.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:53:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:53:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:53:59.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:53:59.838 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:53:59.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:53:59.966 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:59 smithi060 ceph-mon[30898]: pgmap v1085: 158 pgs: 158 active+clean; 487 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 677 op/s 2024-04-25T05:53:59.967 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:59 smithi060 ceph-mon[30898]: 11.1 scrub starts 2024-04-25T05:53:59.967 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:59 smithi060 ceph-mon[30898]: 11.1 scrub ok 2024-04-25T05:53:59.967 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:59 smithi060 ceph-mon[30898]: 6.12 scrub starts 2024-04-25T05:53:59.967 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:53:59 smithi060 ceph-mon[30898]: 6.12 scrub ok 2024-04-25T05:53:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:53:59.974 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:53:59.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:00.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:00.105 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:00.141 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:53:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:53:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:00.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:00.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:00.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:00.367 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:00.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:00.494 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:00.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.625 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:00.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:00.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:00.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-25T05:54:00.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:00.911 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:00.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:01.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:01.054 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:01.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:01.206 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:01.308 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:01.328 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 1.6 scrub starts 2024-04-25T05:54:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 1.6 scrub ok 2024-04-25T05:54:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.1e scrub starts 2024-04-25T05:54:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.1e scrub ok 2024-04-25T05:54:01.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: osdmap e578: 8 total, 8 up, 8 in 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: pgmap v1087: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 1.6 scrub starts 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 1.6 scrub ok 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.1e scrub starts 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.1e scrub ok 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:54:01.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: osdmap e578: 8 total, 8 up, 8 in 2024-04-25T05:54:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: pgmap v1087: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:54:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:54:01.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:01 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:54:01.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:01.454 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 1.6 scrub starts 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 1.6 scrub ok 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.1e scrub starts 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.1e scrub ok 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:54:01.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:54:01.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: osdmap e578: 8 total, 8 up, 8 in 2024-04-25T05:54:01.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: pgmap v1087: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-04-25T05:54:01.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:54:01.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:01 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:54:01.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:01.577 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:01.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:01.712 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:01.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:01.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:01.994 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:01.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:02.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:02.112 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Removed 130286 objects 2024-04-25T05:54:02.112 INFO:tasks.radosbench.radosbench.0.smithi043.stdout:Clean up completed and total clean up time :131.879 2024-04-25T05:54:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.122 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:02.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:02.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:02.243 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:02.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:02.366 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[27010]: 3.6 deep-scrub starts 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[27010]: 3.6 deep-scrub ok 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[27010]: 6.7 deep-scrub starts 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[37492]: 3.6 deep-scrub starts 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[37492]: 3.6 deep-scrub ok 2024-04-25T05:54:02.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:02 smithi043 ceph-mon[37492]: 6.7 deep-scrub starts 2024-04-25T05:54:02.468 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.490 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:02.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:02 smithi060 ceph-mon[30898]: 3.6 deep-scrub starts 2024-04-25T05:54:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:02 smithi060 ceph-mon[30898]: 3.6 deep-scrub ok 2024-04-25T05:54:02.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:02 smithi060 ceph-mon[30898]: 6.7 deep-scrub starts 2024-04-25T05:54:02.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:02.611 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:02.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:02.739 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:02.841 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:02.866 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:02.968 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:02.989 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:02.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:03.048 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:54:03.050 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:54:03.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:03.116 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:03.218 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:03.239 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:03.342 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:03.347 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: 6.7 deep-scrub ok 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: 4.1a scrub starts 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: 4.1a scrub ok 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: 11.6 scrub starts 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: pgmap v1088: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[27010]: 11.6 scrub ok 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: 6.7 deep-scrub ok 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: 4.1a scrub starts 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: 4.1a scrub ok 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: 11.6 scrub starts 2024-04-25T05:54:03.348 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: pgmap v1088: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s 2024-04-25T05:54:03.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:03 smithi043 ceph-mon[37492]: 11.6 scrub ok 2024-04-25T05:54:03.356 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:03.367 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:03.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-25T05:54:03.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:03.512 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: 6.7 deep-scrub ok 2024-04-25T05:54:03.512 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: 4.1a scrub starts 2024-04-25T05:54:03.512 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: 4.1a scrub ok 2024-04-25T05:54:03.513 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: 11.6 scrub starts 2024-04-25T05:54:03.513 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: pgmap v1088: 158 pgs: 158 active+clean; 192 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 900 op/s 2024-04-25T05:54:03.513 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-mon[30898]: 11.6 scrub ok 2024-04-25T05:54:03.533 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:03.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:03.655 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:03.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:03.780 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:03.881 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:03.898 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:03.904 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:03.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:04.006 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:04.028 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:04.129 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:04.135 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:04.151 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:04.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:04.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:04.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:04.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[37492]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:54:04.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:54:04.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:54:04.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:04.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[27010]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:54:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:54:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:54:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:04.381 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:04.399 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:04.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:04.406 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1088,"stamp":"2024-04-25T05:54:02.127967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":201267763,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":2594,"num_read_kb":2314,"num_write":258425,"num_write_kb":8338350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28897,"num_bytes_recovered":1850560081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81803,"ondisk_log_size":81803,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8188720,"kb_used_data":1141308,"kb_used_omap":12,"kb_used_meta":7046963,"kb_avail":741772496,"statfs":{"total":767960285184,"available":759575035904,"internally_reserved":0,"allocated":1168699392,"data_stored":1163135169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7216090672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,4,11,10,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-353894400,"num_objects":-5400,"num_object_clones":0,"num_object_copies":-16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5400,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5400,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003147"},"pg_stats":[{"pgid":"11.3","version":"577'16148","reported_seq":16503,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.662506+0000","last_change":"2024-04-25T05:53:34.830889+0000","last_active":"2024-04-25T05:53:58.662506+0000","last_peered":"2024-04-25T05:53:58.662506+0000","last_clean":"2024-04-25T05:53:58.662506+0000","last_became_active":"2024-04-25T05:50:12.040449+0000","last_became_peered":"2024-04-25T05:50:12.040449+0000","last_unstale":"2024-04-25T05:53:58.662506+0000","last_undegraded":"2024-04-25T05:53:58.662506+0000","last_fullsized":"2024-04-25T05:53:58.662506+0000","mapping_epoch":456,"log_start":"531'10946","ondisk_log_start":"531'10946","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'14507","last_scrub_stamp":"2024-04-25T05:53:34.830724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.830724+0000","objects_scrubbed":1862,"log_size":5202,"log_dups_size":0,"ondisk_log_size":5202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:55.510615+0000","scrub_duration":573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16148,"num_write_kb":523392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"320'20","reported_seq":579,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547159+0000","last_change":"2024-04-25T05:52:35.929420+0000","last_active":"2024-04-25T05:53:57.547159+0000","last_peered":"2024-04-25T05:53:57.547159+0000","last_clean":"2024-04-25T05:53:57.547159+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:53:57.547159+0000","last_undegraded":"2024-04-25T05:53:57.547159+0000","last_fullsized":"2024-04-25T05:53:57.547159+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'20","last_scrub_stamp":"2024-04-25T05:52:35.929307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.929307+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1102,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.738186+0000","last_change":"2024-04-25T05:53:57.738227+0000","last_active":"2024-04-25T05:53:57.738186+0000","last_peered":"2024-04-25T05:53:57.738186+0000","last_clean":"2024-04-25T05:53:57.738186+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:53:57.738186+0000","last_undegraded":"2024-04-25T05:53:57.738186+0000","last_fullsized":"2024-04-25T05:53:57.738186+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:57.736955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.736955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.093262+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":550,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550204+0000","last_change":"2024-04-25T05:53:57.528739+0000","last_active":"2024-04-25T05:53:57.550204+0000","last_peered":"2024-04-25T05:53:57.550204+0000","last_clean":"2024-04-25T05:53:57.550204+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:53:57.550204+0000","last_undegraded":"2024-04-25T05:53:57.550204+0000","last_fullsized":"2024-04-25T05:53:57.550204+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:53:57.528618+0000","last_deep_scrub":"388'14","last_deep_scrub_stamp":"2024-04-25T05:53:57.528618+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.528618+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.206304+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":656,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547682+0000","last_change":"2024-04-25T05:52:08.909918+0000","last_active":"2024-04-25T05:53:57.547682+0000","last_peered":"2024-04-25T05:53:57.547682+0000","last_clean":"2024-04-25T05:53:57.547682+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:53:57.547682+0000","last_undegraded":"2024-04-25T05:53:57.547682+0000","last_fullsized":"2024-04-25T05:53:57.547682+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:08.909664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.909664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1436,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533209+0000","last_change":"2024-04-25T05:51:58.168288+0000","last_active":"2024-04-25T05:53:57.533209+0000","last_peered":"2024-04-25T05:53:57.533209+0000","last_clean":"2024-04-25T05:53:57.533209+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:53:57.533209+0000","last_undegraded":"2024-04-25T05:53:57.533209+0000","last_fullsized":"2024-04-25T05:53:57.533209+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:51:58.168142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:51:58.168142+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"577'16247","reported_seq":29995,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.020354+0000","last_change":"2024-04-25T05:52:52.524759+0000","last_active":"2024-04-25T05:53:58.020354+0000","last_peered":"2024-04-25T05:53:58.020354+0000","last_clean":"2024-04-25T05:53:58.020354+0000","last_became_active":"2024-04-25T05:52:44.443285+0000","last_became_peered":"2024-04-25T05:52:44.443285+0000","last_unstale":"2024-04-25T05:53:58.020354+0000","last_undegraded":"2024-04-25T05:53:58.020354+0000","last_fullsized":"2024-04-25T05:53:58.020354+0000","mapping_epoch":540,"log_start":"517'10124","ondisk_log_start":"517'10124","created":456,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'11957","last_scrub_stamp":"2024-04-25T05:52:52.524605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.524605+0000","objects_scrubbed":4563,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.589578+0000","scrub_duration":2115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16247,"num_write_kb":527040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6713,"num_bytes_recovered":439877632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"320'39","reported_seq":588,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547774+0000","last_change":"2024-04-25T05:53:30.906380+0000","last_active":"2024-04-25T05:53:57.547774+0000","last_peered":"2024-04-25T05:53:57.547774+0000","last_clean":"2024-04-25T05:53:57.547774+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:53:57.547774+0000","last_undegraded":"2024-04-25T05:53:57.547774+0000","last_fullsized":"2024-04-25T05:53:57.547774+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:53:30.906231+0000","last_deep_scrub":"320'39","last_deep_scrub_stamp":"2024-04-25T05:51:57.812833+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.906231+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:58.223526+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":817,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537611+0000","last_change":"2024-04-25T05:53:51.336319+0000","last_active":"2024-04-25T05:53:57.537611+0000","last_peered":"2024-04-25T05:53:57.537611+0000","last_clean":"2024-04-25T05:53:57.537611+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:53:57.537611+0000","last_undegraded":"2024-04-25T05:53:57.537611+0000","last_fullsized":"2024-04-25T05:53:57.537611+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:51.336244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:02.563344+0000","last_clean_scrub_stamp":"2024-04-25T05:53:51.336244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.399146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1107,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541494+0000","last_change":"2024-04-25T05:51:40.109308+0000","last_active":"2024-04-25T05:53:57.541494+0000","last_peered":"2024-04-25T05:53:57.541494+0000","last_clean":"2024-04-25T05:53:57.541494+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:53:57.541494+0000","last_undegraded":"2024-04-25T05:53:57.541494+0000","last_fullsized":"2024-04-25T05:53:57.541494+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:55.072898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:55.072898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":445,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550496+0000","last_change":"2024-04-25T05:53:34.769498+0000","last_active":"2024-04-25T05:53:57.550496+0000","last_peered":"2024-04-25T05:53:57.550496+0000","last_clean":"2024-04-25T05:53:57.550496+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:53:57.550496+0000","last_undegraded":"2024-04-25T05:53:57.550496+0000","last_fullsized":"2024-04-25T05:53:57.550496+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.769373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.769373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.388264+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":1034,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543625+0000","last_change":"2024-04-25T05:52:07.018629+0000","last_active":"2024-04-25T05:53:57.543625+0000","last_peered":"2024-04-25T05:53:57.543625+0000","last_clean":"2024-04-25T05:53:57.543625+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:53:57.543625+0000","last_undegraded":"2024-04-25T05:53:57.543625+0000","last_fullsized":"2024-04-25T05:53:57.543625+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:52:07.018495+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:52:07.018495+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"577'16065","reported_seq":29965,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.686554+0000","last_change":"2024-04-25T05:53:58.500431+0000","last_active":"2024-04-25T05:53:58.686554+0000","last_peered":"2024-04-25T05:53:58.686554+0000","last_clean":"2024-04-25T05:53:58.686554+0000","last_became_active":"2024-04-25T05:52:36.097202+0000","last_became_peered":"2024-04-25T05:52:36.097202+0000","last_unstale":"2024-04-25T05:53:58.686554+0000","last_undegraded":"2024-04-25T05:53:58.686554+0000","last_fullsized":"2024-04-25T05:53:58.686554+0000","mapping_epoch":532,"log_start":"530'10865","ondisk_log_start":"530'10865","created":456,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'16049","last_scrub_stamp":"2024-04-25T05:53:58.500293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:53:58.500293+0000","objects_scrubbed":230,"log_size":5200,"log_dups_size":0,"ondisk_log_size":5200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:02.723816+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16065,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6773,"num_bytes_recovered":443875328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"320'25","reported_seq":888,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547496+0000","last_change":"2024-04-25T05:52:41.908835+0000","last_active":"2024-04-25T05:53:57.547496+0000","last_peered":"2024-04-25T05:53:57.547496+0000","last_clean":"2024-04-25T05:53:57.547496+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:53:57.547496+0000","last_undegraded":"2024-04-25T05:53:57.547496+0000","last_fullsized":"2024-04-25T05:53:57.547496+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'25","last_scrub_stamp":"2024-04-25T05:52:41.908693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.908693+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1123,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546571+0000","last_change":"2024-04-25T05:53:29.887090+0000","last_active":"2024-04-25T05:53:57.546571+0000","last_peered":"2024-04-25T05:53:57.546571+0000","last_clean":"2024-04-25T05:53:57.546571+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:53:57.546571+0000","last_undegraded":"2024-04-25T05:53:57.546571+0000","last_fullsized":"2024-04-25T05:53:57.546571+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.887008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.890083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544590+0000","last_change":"2024-04-25T05:53:33.021664+0000","last_active":"2024-04-25T05:53:57.544590+0000","last_peered":"2024-04-25T05:53:57.544590+0000","last_clean":"2024-04-25T05:53:57.544590+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:53:57.544590+0000","last_undegraded":"2024-04-25T05:53:57.544590+0000","last_fullsized":"2024-04-25T05:53:57.544590+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.021506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.021506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.233074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1082,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538125+0000","last_change":"2024-04-25T05:50:14.079407+0000","last_active":"2024-04-25T05:53:57.538125+0000","last_peered":"2024-04-25T05:53:57.538125+0000","last_clean":"2024-04-25T05:53:57.538125+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:53:57.538125+0000","last_undegraded":"2024-04-25T05:53:57.538125+0000","last_fullsized":"2024-04-25T05:53:57.538125+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:14.079329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:14.079329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"577'16059","reported_seq":16364,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.478417+0000","last_change":"2024-04-25T05:52:50.663942+0000","last_active":"2024-04-25T05:53:59.478417+0000","last_peered":"2024-04-25T05:53:59.478417+0000","last_clean":"2024-04-25T05:53:59.478417+0000","last_became_active":"2024-04-25T05:50:12.040951+0000","last_became_peered":"2024-04-25T05:50:12.040951+0000","last_unstale":"2024-04-25T05:53:59.478417+0000","last_undegraded":"2024-04-25T05:53:59.478417+0000","last_fullsized":"2024-04-25T05:53:59.478417+0000","mapping_epoch":456,"log_start":"538'11213","ondisk_log_start":"538'11213","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'11702","last_scrub_stamp":"2024-04-25T05:52:50.663796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.663796+0000","objects_scrubbed":4552,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":135,"num_object_clones":0,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16059,"num_write_kb":518208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"320'34","reported_seq":1131,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542902+0000","last_change":"2024-04-25T05:53:30.416648+0000","last_active":"2024-04-25T05:53:57.542902+0000","last_peered":"2024-04-25T05:53:57.542902+0000","last_clean":"2024-04-25T05:53:57.542902+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:53:57.542902+0000","last_undegraded":"2024-04-25T05:53:57.542902+0000","last_fullsized":"2024-04-25T05:53:57.542902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:30.416481+0000","last_deep_scrub":"210'22","last_deep_scrub_stamp":"2024-04-25T05:39:39.713165+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.416481+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.310035+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1079,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544341+0000","last_change":"2024-04-25T05:51:20.429881+0000","last_active":"2024-04-25T05:53:57.544341+0000","last_peered":"2024-04-25T05:53:57.544341+0000","last_clean":"2024-04-25T05:53:57.544341+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:53:57.544341+0000","last_undegraded":"2024-04-25T05:53:57.544341+0000","last_fullsized":"2024-04-25T05:53:57.544341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:20.429760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:20.429760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":798,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547616+0000","last_change":"2024-04-25T05:53:49.070910+0000","last_active":"2024-04-25T05:53:57.547616+0000","last_peered":"2024-04-25T05:53:57.547616+0000","last_clean":"2024-04-25T05:53:57.547616+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:53:57.547616+0000","last_undegraded":"2024-04-25T05:53:57.547616+0000","last_fullsized":"2024-04-25T05:53:57.547616+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:53:49.070607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:49.070607+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:03.074399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":426,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544324+0000","last_change":"2024-04-25T05:53:21.122301+0000","last_active":"2024-04-25T05:53:57.544324+0000","last_peered":"2024-04-25T05:53:57.544324+0000","last_clean":"2024-04-25T05:53:57.544324+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:53:57.544324+0000","last_undegraded":"2024-04-25T05:53:57.544324+0000","last_fullsized":"2024-04-25T05:53:57.544324+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.122207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.122207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:28.763652+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":998,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543988+0000","last_change":"2024-04-25T05:52:45.117231+0000","last_active":"2024-04-25T05:53:57.543988+0000","last_peered":"2024-04-25T05:53:57.543988+0000","last_clean":"2024-04-25T05:53:57.543988+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:53:57.543988+0000","last_undegraded":"2024-04-25T05:53:57.543988+0000","last_fullsized":"2024-04-25T05:53:57.543988+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.117108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.117108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.151932+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"577'16296","reported_seq":16627,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.459236+0000","last_change":"2024-04-25T05:52:09.488584+0000","last_active":"2024-04-25T05:53:58.459236+0000","last_peered":"2024-04-25T05:53:58.459236+0000","last_clean":"2024-04-25T05:53:58.459236+0000","last_became_active":"2024-04-25T05:50:12.041754+0000","last_became_peered":"2024-04-25T05:50:12.041754+0000","last_unstale":"2024-04-25T05:53:58.459236+0000","last_undegraded":"2024-04-25T05:53:58.459236+0000","last_fullsized":"2024-04-25T05:53:58.459236+0000","mapping_epoch":456,"log_start":"539'11569","ondisk_log_start":"539'11569","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'9496","last_scrub_stamp":"2024-04-25T05:52:09.488306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:09.488306+0000","objects_scrubbed":7109,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16296,"num_write_kb":528832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1103,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546502+0000","last_change":"2024-04-25T05:53:33.920196+0000","last_active":"2024-04-25T05:53:57.546502+0000","last_peered":"2024-04-25T05:53:57.546502+0000","last_clean":"2024-04-25T05:53:57.546502+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:53:57.546502+0000","last_undegraded":"2024-04-25T05:53:57.546502+0000","last_fullsized":"2024-04-25T05:53:57.546502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.920077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.920077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.068133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"320'42","reported_seq":1261,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544424+0000","last_change":"2024-04-25T05:51:40.149035+0000","last_active":"2024-04-25T05:53:57.544424+0000","last_peered":"2024-04-25T05:53:57.544424+0000","last_clean":"2024-04-25T05:53:57.544424+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:53:57.544424+0000","last_undegraded":"2024-04-25T05:53:57.544424+0000","last_fullsized":"2024-04-25T05:53:57.544424+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'42","last_scrub_stamp":"2024-04-25T05:51:29.363198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:29.363198+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1117,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542765+0000","last_change":"2024-04-25T05:53:20.767176+0000","last_active":"2024-04-25T05:53:57.542765+0000","last_peered":"2024-04-25T05:53:57.542765+0000","last_clean":"2024-04-25T05:53:57.542765+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:53:57.542765+0000","last_undegraded":"2024-04-25T05:53:57.542765+0000","last_fullsized":"2024-04-25T05:53:57.542765+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:20.767051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:20.767051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.097716+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":650,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546570+0000","last_change":"2024-04-25T05:52:42.951239+0000","last_active":"2024-04-25T05:53:57.546570+0000","last_peered":"2024-04-25T05:53:57.546570+0000","last_clean":"2024-04-25T05:53:57.546570+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:53:57.546570+0000","last_undegraded":"2024-04-25T05:53:57.546570+0000","last_fullsized":"2024-04-25T05:53:57.546570+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.951111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.951111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1372,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547296+0000","last_change":"2024-04-25T05:53:27.936176+0000","last_active":"2024-04-25T05:53:57.547296+0000","last_peered":"2024-04-25T05:53:57.547296+0000","last_clean":"2024-04-25T05:53:57.547296+0000","last_became_active":"2024-04-25T05:52:47.115419+0000","last_became_peered":"2024-04-25T05:52:47.115419+0000","last_unstale":"2024-04-25T05:53:57.547296+0000","last_undegraded":"2024-04-25T05:53:57.547296+0000","last_fullsized":"2024-04-25T05:53:57.547296+0000","mapping_epoch":379,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:27.935927+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:53:27.935927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:57.848657+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"577'15628","reported_seq":15938,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.279432+0000","last_change":"2024-04-25T05:52:42.423083+0000","last_active":"2024-04-25T05:53:58.279432+0000","last_peered":"2024-04-25T05:53:58.279432+0000","last_clean":"2024-04-25T05:53:58.279432+0000","last_became_active":"2024-04-25T05:50:12.041406+0000","last_became_peered":"2024-04-25T05:50:12.041406+0000","last_unstale":"2024-04-25T05:53:58.279432+0000","last_undegraded":"2024-04-25T05:53:58.279432+0000","last_fullsized":"2024-04-25T05:53:58.279432+0000","mapping_epoch":456,"log_start":"537'10985","ondisk_log_start":"537'10985","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'11106","last_scrub_stamp":"2024-04-25T05:52:42.422923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.422923+0000","objects_scrubbed":4765,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15628,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1330,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544570+0000","last_change":"2024-04-25T05:53:31.717402+0000","last_active":"2024-04-25T05:53:57.544570+0000","last_peered":"2024-04-25T05:53:57.544570+0000","last_clean":"2024-04-25T05:53:57.544570+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:53:57.544570+0000","last_undegraded":"2024-04-25T05:53:57.544570+0000","last_fullsized":"2024-04-25T05:53:57.544570+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:53:31.717254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.717254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.453072+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"320'34","reported_seq":780,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546682+0000","last_change":"2024-04-25T05:52:45.015305+0000","last_active":"2024-04-25T05:53:57.546682+0000","last_peered":"2024-04-25T05:53:57.546682+0000","last_clean":"2024-04-25T05:53:57.546682+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:53:57.546682+0000","last_undegraded":"2024-04-25T05:53:57.546682+0000","last_fullsized":"2024-04-25T05:53:57.546682+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:52:45.015213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.015213+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.513334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":174,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545734+0000","last_change":"2024-04-25T05:51:40.129777+0000","last_active":"2024-04-25T05:53:57.545734+0000","last_peered":"2024-04-25T05:53:57.545734+0000","last_clean":"2024-04-25T05:53:57.545734+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:53:57.545734+0000","last_undegraded":"2024-04-25T05:53:57.545734+0000","last_fullsized":"2024-04-25T05:53:57.545734+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:50:15.054124+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:48:04.608284+0000","last_clean_scrub_stamp":"2024-04-25T05:50:15.054124+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":468,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542537+0000","last_change":"2024-04-25T05:53:34.695038+0000","last_active":"2024-04-25T05:53:57.542537+0000","last_peered":"2024-04-25T05:53:57.542537+0000","last_clean":"2024-04-25T05:53:57.542537+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:53:57.542537+0000","last_undegraded":"2024-04-25T05:53:57.542537+0000","last_fullsized":"2024-04-25T05:53:57.542537+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.694925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.694925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.211727+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1338,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537946+0000","last_change":"2024-04-25T05:53:21.777034+0000","last_active":"2024-04-25T05:53:57.537946+0000","last_peered":"2024-04-25T05:53:57.537946+0000","last_clean":"2024-04-25T05:53:57.537946+0000","last_became_active":"2024-04-25T05:52:51.531591+0000","last_became_peered":"2024-04-25T05:52:51.531591+0000","last_unstale":"2024-04-25T05:53:57.537946+0000","last_undegraded":"2024-04-25T05:53:57.537946+0000","last_fullsized":"2024-04-25T05:53:57.537946+0000","mapping_epoch":387,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:21.776901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.776901+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.945686+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"577'15979","reported_seq":16321,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.009833+0000","last_change":"2024-04-25T05:52:45.545440+0000","last_active":"2024-04-25T05:53:58.009833+0000","last_peered":"2024-04-25T05:53:58.009833+0000","last_clean":"2024-04-25T05:53:58.009833+0000","last_became_active":"2024-04-25T05:50:12.035310+0000","last_became_peered":"2024-04-25T05:50:12.035310+0000","last_unstale":"2024-04-25T05:53:58.009833+0000","last_undegraded":"2024-04-25T05:53:58.009833+0000","last_fullsized":"2024-04-25T05:53:58.009833+0000","mapping_epoch":456,"log_start":"514'9790","ondisk_log_start":"514'9790","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'11448","last_scrub_stamp":"2024-04-25T05:52:45.545041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.545041+0000","objects_scrubbed":4796,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:14.959830+0000","scrub_duration":1123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728664,"num_objects":241,"num_object_clones":0,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15979,"num_write_kb":518977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":201,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547227+0000","last_change":"2024-04-25T05:51:40.124005+0000","last_active":"2024-04-25T05:53:57.547227+0000","last_peered":"2024-04-25T05:53:57.547227+0000","last_clean":"2024-04-25T05:53:57.547227+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:53:57.547227+0000","last_undegraded":"2024-04-25T05:53:57.547227+0000","last_fullsized":"2024-04-25T05:53:57.547227+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'16","last_scrub_stamp":"2024-04-25T05:50:45.130468+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:50:45.130468+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"320'32","reported_seq":828,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541204+0000","last_change":"2024-04-25T05:51:32.949273+0000","last_active":"2024-04-25T05:53:57.541204+0000","last_peered":"2024-04-25T05:53:57.541204+0000","last_clean":"2024-04-25T05:53:57.541204+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:53:57.541204+0000","last_undegraded":"2024-04-25T05:53:57.541204+0000","last_fullsized":"2024-04-25T05:53:57.541204+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:51:32.949135+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:51:32.949135+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544378+0000","last_change":"2024-04-25T05:52:41.064075+0000","last_active":"2024-04-25T05:53:57.544378+0000","last_peered":"2024-04-25T05:53:57.544378+0000","last_clean":"2024-04-25T05:53:57.544378+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:53:57.544378+0000","last_undegraded":"2024-04-25T05:53:57.544378+0000","last_fullsized":"2024-04-25T05:53:57.544378+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:41.063933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.063933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":178,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544981+0000","last_change":"2024-04-25T05:53:49.262740+0000","last_active":"2024-04-25T05:53:57.544981+0000","last_peered":"2024-04-25T05:53:57.544981+0000","last_clean":"2024-04-25T05:53:57.544981+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:53:57.544981+0000","last_undegraded":"2024-04-25T05:53:57.544981+0000","last_fullsized":"2024-04-25T05:53:57.544981+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:49.262490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:49.262490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:13.639641+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1328,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537922+0000","last_change":"2024-04-25T05:53:50.288882+0000","last_active":"2024-04-25T05:53:57.537922+0000","last_peered":"2024-04-25T05:53:57.537922+0000","last_clean":"2024-04-25T05:53:57.537922+0000","last_became_active":"2024-04-25T05:52:39.109573+0000","last_became_peered":"2024-04-25T05:52:39.109573+0000","last_unstale":"2024-04-25T05:53:57.537922+0000","last_undegraded":"2024-04-25T05:53:57.537922+0000","last_fullsized":"2024-04-25T05:53:57.537922+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:53:50.288752+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:33.347743+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.288752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.700218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"577'16309","reported_seq":16613,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.475616+0000","last_change":"2024-04-25T05:52:52.331705+0000","last_active":"2024-04-25T05:53:59.475616+0000","last_peered":"2024-04-25T05:53:59.475616+0000","last_clean":"2024-04-25T05:53:59.475616+0000","last_became_active":"2024-04-25T05:50:12.037786+0000","last_became_peered":"2024-04-25T05:50:12.037786+0000","last_unstale":"2024-04-25T05:53:59.475616+0000","last_undegraded":"2024-04-25T05:53:59.475616+0000","last_fullsized":"2024-04-25T05:53:59.475616+0000","mapping_epoch":456,"log_start":"537'11466","ondisk_log_start":"537'11466","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12101","last_scrub_stamp":"2024-04-25T05:52:52.331512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.331512+0000","objects_scrubbed":4384,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.199070+0000","scrub_duration":1716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":139,"num_object_clones":0,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16309,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":853,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537709+0000","last_change":"2024-04-25T05:53:33.265611+0000","last_active":"2024-04-25T05:53:57.537709+0000","last_peered":"2024-04-25T05:53:57.537709+0000","last_clean":"2024-04-25T05:53:57.537709+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:53:57.537709+0000","last_undegraded":"2024-04-25T05:53:57.537709+0000","last_fullsized":"2024-04-25T05:53:57.537709+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.265484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.265484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:54.263173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"320'35","reported_seq":198,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547141+0000","last_change":"2024-04-25T05:53:24.028014+0000","last_active":"2024-04-25T05:53:57.547141+0000","last_peered":"2024-04-25T05:53:57.547141+0000","last_clean":"2024-04-25T05:53:57.547141+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:53:57.547141+0000","last_undegraded":"2024-04-25T05:53:57.547141+0000","last_fullsized":"2024-04-25T05:53:57.547141+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'35","last_scrub_stamp":"2024-04-25T05:53:24.027846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.027846+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:33.305120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1124,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541249+0000","last_change":"2024-04-25T05:53:57.332813+0000","last_active":"2024-04-25T05:53:57.541249+0000","last_peered":"2024-04-25T05:53:57.541249+0000","last_clean":"2024-04-25T05:53:57.541249+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:53:57.541249+0000","last_undegraded":"2024-04-25T05:53:57.541249+0000","last_fullsized":"2024-04-25T05:53:57.541249+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:57.332461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.332461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:12.030643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1098,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541207+0000","last_change":"2024-04-25T05:52:47.618782+0000","last_active":"2024-04-25T05:53:57.541207+0000","last_peered":"2024-04-25T05:53:57.541207+0000","last_clean":"2024-04-25T05:53:57.541207+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:53:57.541207+0000","last_undegraded":"2024-04-25T05:53:57.541207+0000","last_fullsized":"2024-04-25T05:53:57.541207+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.618569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.618569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:17.510421+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1347,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537678+0000","last_change":"2024-04-25T05:52:43.436694+0000","last_active":"2024-04-25T05:53:57.537678+0000","last_peered":"2024-04-25T05:53:57.537678+0000","last_clean":"2024-04-25T05:53:57.537678+0000","last_became_active":"2024-04-25T05:52:43.436401+0000","last_became_peered":"2024-04-25T05:52:43.436401+0000","last_unstale":"2024-04-25T05:53:57.537678+0000","last_undegraded":"2024-04-25T05:53:57.537678+0000","last_fullsized":"2024-04-25T05:53:57.537678+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:37.366472+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:57.500683+0000","last_clean_scrub_stamp":"2024-04-25T05:52:37.366472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"577'16373","reported_seq":25374,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.442920+0000","last_change":"2024-04-25T05:53:17.142210+0000","last_active":"2024-04-25T05:53:59.442920+0000","last_peered":"2024-04-25T05:53:59.442920+0000","last_clean":"2024-04-25T05:53:59.442920+0000","last_became_active":"2024-04-25T05:53:17.107904+0000","last_became_peered":"2024-04-25T05:53:17.107904+0000","last_unstale":"2024-04-25T05:53:59.442920+0000","last_undegraded":"2024-04-25T05:53:59.442920+0000","last_fullsized":"2024-04-25T05:53:59.442920+0000","mapping_epoch":557,"log_start":"545'11827","ondisk_log_start":"545'11827","created":456,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":159,"num_object_clones":0,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16373,"num_write_kb":529024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4367,"num_bytes_recovered":286195712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":426,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544610+0000","last_change":"2024-04-25T05:52:53.143666+0000","last_active":"2024-04-25T05:53:57.544610+0000","last_peered":"2024-04-25T05:53:57.544610+0000","last_clean":"2024-04-25T05:53:57.544610+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:53:57.544610+0000","last_undegraded":"2024-04-25T05:53:57.544610+0000","last_fullsized":"2024-04-25T05:53:57.544610+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.143529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.143529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.394156+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"320'24","reported_seq":105,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545419+0000","last_change":"2024-04-25T05:52:43.541002+0000","last_active":"2024-04-25T05:53:57.545419+0000","last_peered":"2024-04-25T05:53:57.545419+0000","last_clean":"2024-04-25T05:53:57.545419+0000","last_became_active":"2024-04-25T05:52:43.437539+0000","last_became_peered":"2024-04-25T05:52:43.437539+0000","last_unstale":"2024-04-25T05:53:57.545419+0000","last_undegraded":"2024-04-25T05:53:57.545419+0000","last_fullsized":"2024-04-25T05:53:57.545419+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:52:38.142064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.142064+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2336,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532778+0000","last_change":"2024-04-25T05:53:53.683465+0000","last_active":"2024-04-25T05:53:57.532778+0000","last_peered":"2024-04-25T05:53:57.532778+0000","last_clean":"2024-04-25T05:53:57.532778+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:53:57.532778+0000","last_undegraded":"2024-04-25T05:53:57.532778+0000","last_fullsized":"2024-04-25T05:53:57.532778+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:53:53.683339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.683339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:21.597169+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1180,"num_read_kb":1175,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.549596+0000","last_change":"2024-04-25T05:53:26.049870+0000","last_active":"2024-04-25T05:53:57.549596+0000","last_peered":"2024-04-25T05:53:57.549596+0000","last_clean":"2024-04-25T05:53:57.549596+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:53:57.549596+0000","last_undegraded":"2024-04-25T05:53:57.549596+0000","last_fullsized":"2024-04-25T05:53:57.549596+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:26.049619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.049619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:36.269598+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1086,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.540932+0000","last_change":"2024-04-25T05:51:22.949696+0000","last_active":"2024-04-25T05:53:57.540932+0000","last_peered":"2024-04-25T05:53:57.540932+0000","last_clean":"2024-04-25T05:53:57.540932+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:53:57.540932+0000","last_undegraded":"2024-04-25T05:53:57.540932+0000","last_fullsized":"2024-04-25T05:53:57.540932+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.949432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.949432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1334,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542435+0000","last_change":"2024-04-25T05:52:55.110481+0000","last_active":"2024-04-25T05:53:57.542435+0000","last_peered":"2024-04-25T05:53:57.542435+0000","last_clean":"2024-04-25T05:53:57.542435+0000","last_became_active":"2024-04-25T05:52:55.109763+0000","last_became_peered":"2024-04-25T05:52:55.109763+0000","last_unstale":"2024-04-25T05:53:57.542435+0000","last_undegraded":"2024-04-25T05:53:57.542435+0000","last_fullsized":"2024-04-25T05:53:57.542435+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:45.463489+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.463489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.844339+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"577'15846","reported_seq":16099,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.423356+0000","last_change":"2024-04-25T05:50:12.038690+0000","last_active":"2024-04-25T05:53:59.423356+0000","last_peered":"2024-04-25T05:53:59.423356+0000","last_clean":"2024-04-25T05:53:59.423356+0000","last_became_active":"2024-04-25T05:50:12.038327+0000","last_became_peered":"2024-04-25T05:50:12.038327+0000","last_unstale":"2024-04-25T05:53:59.423356+0000","last_undegraded":"2024-04-25T05:53:59.423356+0000","last_fullsized":"2024-04-25T05:53:59.423356+0000","mapping_epoch":456,"log_start":"543'11279","ondisk_log_start":"543'11279","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15846,"num_write_kb":511296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1116,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534292+0000","last_change":"2024-04-25T05:53:29.647674+0000","last_active":"2024-04-25T05:53:57.534292+0000","last_peered":"2024-04-25T05:53:57.534292+0000","last_clean":"2024-04-25T05:53:57.534292+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:53:57.534292+0000","last_undegraded":"2024-04-25T05:53:57.534292+0000","last_fullsized":"2024-04-25T05:53:57.534292+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:29.647523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.647523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:53.191587+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"457'39","reported_seq":1026,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538558+0000","last_change":"2024-04-25T05:52:38.405034+0000","last_active":"2024-04-25T05:53:57.538558+0000","last_peered":"2024-04-25T05:53:57.538558+0000","last_clean":"2024-04-25T05:53:57.538558+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:53:57.538558+0000","last_undegraded":"2024-04-25T05:53:57.538558+0000","last_fullsized":"2024-04-25T05:53:57.538558+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'39","last_scrub_stamp":"2024-04-25T05:52:38.404858+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.404858+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.254310+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1095,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538912+0000","last_change":"2024-04-25T05:49:02.027900+0000","last_active":"2024-04-25T05:53:57.538912+0000","last_peered":"2024-04-25T05:53:57.538912+0000","last_clean":"2024-04-25T05:53:57.538912+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:53:57.538912+0000","last_undegraded":"2024-04-25T05:53:57.538912+0000","last_fullsized":"2024-04-25T05:53:57.538912+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:02.027759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:49:02.027759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1461,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532877+0000","last_change":"2024-04-25T05:52:35.380122+0000","last_active":"2024-04-25T05:53:57.532877+0000","last_peered":"2024-04-25T05:53:57.532877+0000","last_clean":"2024-04-25T05:53:57.532877+0000","last_became_active":"2024-04-25T05:52:35.091072+0000","last_became_peered":"2024-04-25T05:52:35.091072+0000","last_unstale":"2024-04-25T05:53:57.532877+0000","last_undegraded":"2024-04-25T05:53:57.532877+0000","last_fullsized":"2024-04-25T05:53:57.532877+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":19,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:35.380044+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.380044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":385,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543003+0000","last_change":"2024-04-25T05:50:52.428014+0000","last_active":"2024-04-25T05:53:57.543003+0000","last_peered":"2024-04-25T05:53:57.543003+0000","last_clean":"2024-04-25T05:53:57.543003+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:53:57.543003+0000","last_undegraded":"2024-04-25T05:53:57.543003+0000","last_fullsized":"2024-04-25T05:53:57.543003+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:52.427747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:09.353310+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.427747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"577'16144","reported_seq":16397,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.447005+0000","last_change":"2024-04-25T05:50:12.039212+0000","last_active":"2024-04-25T05:53:59.447005+0000","last_peered":"2024-04-25T05:53:59.447005+0000","last_clean":"2024-04-25T05:53:59.447005+0000","last_became_active":"2024-04-25T05:50:12.038922+0000","last_became_peered":"2024-04-25T05:50:12.038922+0000","last_unstale":"2024-04-25T05:53:59.447005+0000","last_undegraded":"2024-04-25T05:53:59.447005+0000","last_fullsized":"2024-04-25T05:53:59.447005+0000","mapping_epoch":456,"log_start":"544'11623","ondisk_log_start":"544'11623","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16144,"num_write_kb":522240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":451,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550049+0000","last_change":"2024-04-25T05:53:50.508173+0000","last_active":"2024-04-25T05:53:57.550049+0000","last_peered":"2024-04-25T05:53:57.550049+0000","last_clean":"2024-04-25T05:53:57.550049+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:53:57.550049+0000","last_undegraded":"2024-04-25T05:53:57.550049+0000","last_fullsized":"2024-04-25T05:53:57.550049+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:50.508053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.508053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:16.809144+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"320'29","reported_seq":1253,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542796+0000","last_change":"2024-04-25T05:53:54.401863+0000","last_active":"2024-04-25T05:53:57.542796+0000","last_peered":"2024-04-25T05:53:57.542796+0000","last_clean":"2024-04-25T05:53:57.542796+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:53:57.542796+0000","last_undegraded":"2024-04-25T05:53:57.542796+0000","last_fullsized":"2024-04-25T05:53:57.542796+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'29","last_scrub_stamp":"2024-04-25T05:53:54.401716+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:53:54.401716+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:19.530444+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":451,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550071+0000","last_change":"2024-04-25T05:53:21.420018+0000","last_active":"2024-04-25T05:53:57.550071+0000","last_peered":"2024-04-25T05:53:57.550071+0000","last_clean":"2024-04-25T05:53:57.550071+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:53:57.550071+0000","last_undegraded":"2024-04-25T05:53:57.550071+0000","last_fullsized":"2024-04-25T05:53:57.550071+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:21.419893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.419893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:39.568518+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1122,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541991+0000","last_change":"2024-04-25T05:50:48.159492+0000","last_active":"2024-04-25T05:53:57.541991+0000","last_peered":"2024-04-25T05:53:57.541991+0000","last_clean":"2024-04-25T05:53:57.541991+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:53:57.541991+0000","last_undegraded":"2024-04-25T05:53:57.541991+0000","last_fullsized":"2024-04-25T05:53:57.541991+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.159378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.159378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1094,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534402+0000","last_change":"2024-04-25T05:52:51.540421+0000","last_active":"2024-04-25T05:53:57.534402+0000","last_peered":"2024-04-25T05:53:57.534402+0000","last_clean":"2024-04-25T05:53:57.534402+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:53:57.534402+0000","last_undegraded":"2024-04-25T05:53:57.534402+0000","last_fullsized":"2024-04-25T05:53:57.534402+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.540285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1468,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533594+0000","last_change":"2024-04-25T05:52:34.412036+0000","last_active":"2024-04-25T05:53:57.533594+0000","last_peered":"2024-04-25T05:53:57.533594+0000","last_clean":"2024-04-25T05:53:57.533594+0000","last_became_active":"2024-04-25T05:52:27.138282+0000","last_became_peered":"2024-04-25T05:52:27.138282+0000","last_unstale":"2024-04-25T05:53:57.533594+0000","last_undegraded":"2024-04-25T05:53:57.533594+0000","last_fullsized":"2024-04-25T05:53:57.533594+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.411897+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.411897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"577'16126","reported_seq":24423,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:59.434226+0000","last_change":"2024-04-25T05:53:16.136131+0000","last_active":"2024-04-25T05:53:59.434226+0000","last_peered":"2024-04-25T05:53:59.434226+0000","last_clean":"2024-04-25T05:53:59.434226+0000","last_became_active":"2024-04-25T05:53:16.108052+0000","last_became_peered":"2024-04-25T05:53:16.108052+0000","last_unstale":"2024-04-25T05:53:59.434226+0000","last_undegraded":"2024-04-25T05:53:59.434226+0000","last_fullsized":"2024-04-25T05:53:59.434226+0000","mapping_epoch":556,"log_start":"543'11626","ondisk_log_start":"543'11626","created":456,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16126,"num_write_kb":521344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4013,"num_bytes_recovered":262995968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":170,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545550+0000","last_change":"2024-04-25T05:51:40.098006+0000","last_active":"2024-04-25T05:53:57.545550+0000","last_peered":"2024-04-25T05:53:57.545550+0000","last_clean":"2024-04-25T05:53:57.545550+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:53:57.545550+0000","last_undegraded":"2024-04-25T05:53:57.545550+0000","last_fullsized":"2024-04-25T05:53:57.545550+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:21.946445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:21.946445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"380'59","reported_seq":1473,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541382+0000","last_change":"2024-04-25T05:53:29.429046+0000","last_active":"2024-04-25T05:53:57.541382+0000","last_peered":"2024-04-25T05:53:57.541382+0000","last_clean":"2024-04-25T05:53:57.541382+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:53:57.541382+0000","last_undegraded":"2024-04-25T05:53:57.541382+0000","last_fullsized":"2024-04-25T05:53:57.541382+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'59","last_scrub_stamp":"2024-04-25T05:53:29.428902+0000","last_deep_scrub":"101'45","last_deep_scrub_stamp":"2024-04-25T05:35:44.336118+0000","last_clean_scrub_stamp":"2024-04-25T05:53:29.428902+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:31.894655+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":669,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.967865+0000","last_change":"2024-04-25T05:53:57.967907+0000","last_active":"2024-04-25T05:53:57.967865+0000","last_peered":"2024-04-25T05:53:57.967865+0000","last_clean":"2024-04-25T05:53:57.967865+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:53:57.967865+0000","last_undegraded":"2024-04-25T05:53:57.967865+0000","last_fullsized":"2024-04-25T05:53:57.967865+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:57.967768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.967768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:08.935713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"506'3","reported_seq":425,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543839+0000","last_change":"2024-04-25T05:52:01.064448+0000","last_active":"2024-04-25T05:53:57.543839+0000","last_peered":"2024-04-25T05:53:57.543839+0000","last_clean":"2024-04-25T05:53:57.543839+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:53:57.543839+0000","last_undegraded":"2024-04-25T05:53:57.543839+0000","last_fullsized":"2024-04-25T05:53:57.543839+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'3","last_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_deep_scrub":"506'3","last_deep_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_clean_scrub_stamp":"2024-04-25T05:52:01.064292+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1059,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.539490+0000","last_change":"2024-04-25T05:49:35.139125+0000","last_active":"2024-04-25T05:53:57.539490+0000","last_peered":"2024-04-25T05:53:57.539490+0000","last_clean":"2024-04-25T05:53:57.539490+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:53:57.539490+0000","last_undegraded":"2024-04-25T05:53:57.539490+0000","last_fullsized":"2024-04-25T05:53:57.539490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:35.139035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:49:35.139035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":1169,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543635+0000","last_change":"2024-04-25T05:52:34.205086+0000","last_active":"2024-04-25T05:53:57.543635+0000","last_peered":"2024-04-25T05:53:57.543635+0000","last_clean":"2024-04-25T05:53:57.543635+0000","last_became_active":"2024-04-25T05:52:31.177637+0000","last_became_peered":"2024-04-25T05:52:31.177637+0000","last_unstale":"2024-04-25T05:53:57.543635+0000","last_undegraded":"2024-04-25T05:53:57.543635+0000","last_fullsized":"2024-04-25T05:53:57.543635+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:52:34.204967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:52:34.204967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:57.850061+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"577'16060","reported_seq":16394,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.459152+0000","last_change":"2024-04-25T05:52:03.230823+0000","last_active":"2024-04-25T05:53:58.459152+0000","last_peered":"2024-04-25T05:53:58.459152+0000","last_clean":"2024-04-25T05:53:58.459152+0000","last_became_active":"2024-04-25T05:50:12.043555+0000","last_became_peered":"2024-04-25T05:50:12.043555+0000","last_unstale":"2024-04-25T05:53:58.459152+0000","last_undegraded":"2024-04-25T05:53:58.459152+0000","last_fullsized":"2024-04-25T05:53:58.459152+0000","mapping_epoch":456,"log_start":"538'11290","ondisk_log_start":"538'11290","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8908","last_scrub_stamp":"2024-04-25T05:52:03.230624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:03.230624+0000","objects_scrubbed":7429,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16060,"num_write_kb":521088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"320'26","reported_seq":190,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547100+0000","last_change":"2024-04-25T05:53:55.281680+0000","last_active":"2024-04-25T05:53:57.547100+0000","last_peered":"2024-04-25T05:53:57.547100+0000","last_clean":"2024-04-25T05:53:57.547100+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:53:57.547100+0000","last_undegraded":"2024-04-25T05:53:57.547100+0000","last_fullsized":"2024-04-25T05:53:57.547100+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:55.281544+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:53:55.281544+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.745055+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544204+0000","last_change":"2024-04-25T05:52:52.129720+0000","last_active":"2024-04-25T05:53:57.544204+0000","last_peered":"2024-04-25T05:53:57.544204+0000","last_clean":"2024-04-25T05:53:57.544204+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:53:57.544204+0000","last_undegraded":"2024-04-25T05:53:57.544204+0000","last_fullsized":"2024-04-25T05:53:57.544204+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:52.129566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:52:52.129566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.024201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":445,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550073+0000","last_change":"2024-04-25T05:53:52.487607+0000","last_active":"2024-04-25T05:53:57.550073+0000","last_peered":"2024-04-25T05:53:57.550073+0000","last_clean":"2024-04-25T05:53:57.550073+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:53:57.550073+0000","last_undegraded":"2024-04-25T05:53:57.550073+0000","last_fullsized":"2024-04-25T05:53:57.550073+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:52.487452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:53:52.487452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:02.976586+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":170,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544792+0000","last_change":"2024-04-25T05:51:40.099599+0000","last_active":"2024-04-25T05:53:57.544792+0000","last_peered":"2024-04-25T05:53:57.544792+0000","last_clean":"2024-04-25T05:53:57.544792+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:53:57.544792+0000","last_undegraded":"2024-04-25T05:53:57.544792+0000","last_fullsized":"2024-04-25T05:53:57.544792+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.026628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.026628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"577'16455","reported_seq":16770,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.440621+0000","last_change":"2024-04-25T05:52:38.587166+0000","last_active":"2024-04-25T05:53:58.440621+0000","last_peered":"2024-04-25T05:53:58.440621+0000","last_clean":"2024-04-25T05:53:58.440621+0000","last_became_active":"2024-04-25T05:50:12.042332+0000","last_became_peered":"2024-04-25T05:50:12.042332+0000","last_unstale":"2024-04-25T05:53:58.440621+0000","last_undegraded":"2024-04-25T05:53:58.440621+0000","last_fullsized":"2024-04-25T05:53:58.440621+0000","mapping_epoch":456,"log_start":"542'11731","ondisk_log_start":"542'11731","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'11363","last_scrub_stamp":"2024-04-25T05:52:38.587034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:38.587034+0000","objects_scrubbed":5307,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16455,"num_write_kb":532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"458'51","reported_seq":873,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537579+0000","last_change":"2024-04-25T05:52:39.432701+0000","last_active":"2024-04-25T05:53:57.537579+0000","last_peered":"2024-04-25T05:53:57.537579+0000","last_clean":"2024-04-25T05:53:57.537579+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:53:57.537579+0000","last_undegraded":"2024-04-25T05:53:57.537579+0000","last_fullsized":"2024-04-25T05:53:57.537579+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'51","last_scrub_stamp":"2024-04-25T05:52:39.432428+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.432428+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:58.290428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":1140,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534231+0000","last_change":"2024-04-25T05:52:41.458972+0000","last_active":"2024-04-25T05:53:57.534231+0000","last_peered":"2024-04-25T05:53:57.534231+0000","last_clean":"2024-04-25T05:53:57.534231+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:53:57.534231+0000","last_undegraded":"2024-04-25T05:53:57.534231+0000","last_fullsized":"2024-04-25T05:53:57.534231+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:52:41.458853+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:52:41.458853+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.006386+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":457,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550390+0000","last_change":"2024-04-25T05:53:56.505454+0000","last_active":"2024-04-25T05:53:57.550390+0000","last_peered":"2024-04-25T05:53:57.550390+0000","last_clean":"2024-04-25T05:53:57.550390+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:53:57.550390+0000","last_undegraded":"2024-04-25T05:53:57.550390+0000","last_fullsized":"2024-04-25T05:53:57.550390+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:56.505337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:56.505337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:12.871400+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1094,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532628+0000","last_change":"2024-04-25T05:53:25.688414+0000","last_active":"2024-04-25T05:53:57.532628+0000","last_peered":"2024-04-25T05:53:57.532628+0000","last_clean":"2024-04-25T05:53:57.532628+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:53:57.532628+0000","last_undegraded":"2024-04-25T05:53:57.532628+0000","last_fullsized":"2024-04-25T05:53:57.532628+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.688336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.688336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.070699+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"577'15802","reported_seq":28794,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.609703+0000","last_change":"2024-04-25T05:52:45.137759+0000","last_active":"2024-04-25T05:53:58.609703+0000","last_peered":"2024-04-25T05:53:58.609703+0000","last_clean":"2024-04-25T05:53:58.609703+0000","last_became_active":"2024-04-25T05:52:45.095486+0000","last_became_peered":"2024-04-25T05:52:45.095486+0000","last_unstale":"2024-04-25T05:53:58.609703+0000","last_undegraded":"2024-04-25T05:53:58.609703+0000","last_fullsized":"2024-04-25T05:53:58.609703+0000","mapping_epoch":541,"log_start":"522'10258","ondisk_log_start":"522'10258","created":456,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:50:11.023936+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15802,"num_write_kb":512896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6364,"num_bytes_recovered":417071104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":457,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.550505+0000","last_change":"2024-04-25T05:53:28.416788+0000","last_active":"2024-04-25T05:53:57.550505+0000","last_peered":"2024-04-25T05:53:57.550505+0000","last_clean":"2024-04-25T05:53:57.550505+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:53:57.550505+0000","last_undegraded":"2024-04-25T05:53:57.550505+0000","last_fullsized":"2024-04-25T05:53:57.550505+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.416661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.416661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.932624+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"320'30","reported_seq":510,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546781+0000","last_change":"2024-04-25T05:53:31.936255+0000","last_active":"2024-04-25T05:53:57.546781+0000","last_peered":"2024-04-25T05:53:57.546781+0000","last_clean":"2024-04-25T05:53:57.546781+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:53:57.546781+0000","last_undegraded":"2024-04-25T05:53:57.546781+0000","last_fullsized":"2024-04-25T05:53:57.546781+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'30","last_scrub_stamp":"2024-04-25T05:53:31.936104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:31.936104+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:44.183145+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534991+0000","last_change":"2024-04-25T05:52:39.438835+0000","last_active":"2024-04-25T05:53:57.534991+0000","last_peered":"2024-04-25T05:53:57.534991+0000","last_clean":"2024-04-25T05:53:57.534991+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:53:57.534991+0000","last_undegraded":"2024-04-25T05:53:57.534991+0000","last_fullsized":"2024-04-25T05:53:57.534991+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:39.438700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:52:39.438700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:06.055139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1070,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543520+0000","last_change":"2024-04-25T05:50:48.697039+0000","last_active":"2024-04-25T05:53:57.543520+0000","last_peered":"2024-04-25T05:53:57.543520+0000","last_clean":"2024-04-25T05:53:57.543520+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:53:57.543520+0000","last_undegraded":"2024-04-25T05:53:57.543520+0000","last_fullsized":"2024-04-25T05:53:57.543520+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:48.696899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.696899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"577'15974","reported_seq":16308,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:58.170944+0000","last_change":"2024-04-25T05:52:02.016861+0000","last_active":"2024-04-25T05:53:58.170944+0000","last_peered":"2024-04-25T05:53:58.170944+0000","last_clean":"2024-04-25T05:53:58.170944+0000","last_became_active":"2024-04-25T05:50:12.041624+0000","last_became_peered":"2024-04-25T05:50:12.041624+0000","last_unstale":"2024-04-25T05:53:58.170944+0000","last_undegraded":"2024-04-25T05:53:58.170944+0000","last_fullsized":"2024-04-25T05:53:58.170944+0000","mapping_epoch":456,"log_start":"525'10610","ondisk_log_start":"525'10610","created":456,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8841","last_scrub_stamp":"2024-04-25T05:52:02.016648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:11.023936+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.016648+0000","objects_scrubbed":7398,"log_size":5364,"log_dups_size":0,"ondisk_log_size":5364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15974,"num_write_kb":517568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1361,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532708+0000","last_change":"2024-04-25T05:52:50.387148+0000","last_active":"2024-04-25T05:53:57.532708+0000","last_peered":"2024-04-25T05:53:57.532708+0000","last_clean":"2024-04-25T05:53:57.532708+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:53:57.532708+0000","last_undegraded":"2024-04-25T05:53:57.532708+0000","last_fullsized":"2024-04-25T05:53:57.532708+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_deep_scrub":"208'11","last_deep_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.387023+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.717658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"320'39","reported_seq":440,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543659+0000","last_change":"2024-04-25T05:52:06.018611+0000","last_active":"2024-04-25T05:53:57.543659+0000","last_peered":"2024-04-25T05:53:57.543659+0000","last_clean":"2024-04-25T05:53:57.543659+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:53:57.543659+0000","last_undegraded":"2024-04-25T05:53:57.543659+0000","last_fullsized":"2024-04-25T05:53:57.543659+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'39","last_scrub_stamp":"2024-04-25T05:52:06.018248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:52:06.018248+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1153,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.535806+0000","last_change":"2024-04-25T05:53:33.672727+0000","last_active":"2024-04-25T05:53:57.535806+0000","last_peered":"2024-04-25T05:53:57.535806+0000","last_clean":"2024-04-25T05:53:57.535806+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:53:57.535806+0000","last_undegraded":"2024-04-25T05:53:57.535806+0000","last_fullsized":"2024-04-25T05:53:57.535806+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:53:33.672600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.672600+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:51.282306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":418,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543880+0000","last_change":"2024-04-25T05:51:56.071814+0000","last_active":"2024-04-25T05:53:57.543880+0000","last_peered":"2024-04-25T05:53:57.543880+0000","last_clean":"2024-04-25T05:53:57.543880+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:53:57.543880+0000","last_undegraded":"2024-04-25T05:53:57.543880+0000","last_fullsized":"2024-04-25T05:53:57.543880+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:56.071706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:56.071706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"320'18","reported_seq":1104,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542945+0000","last_change":"2024-04-25T05:51:28.373915+0000","last_active":"2024-04-25T05:53:57.542945+0000","last_peered":"2024-04-25T05:53:57.542945+0000","last_clean":"2024-04-25T05:53:57.542945+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:53:57.542945+0000","last_undegraded":"2024-04-25T05:53:57.542945+0000","last_fullsized":"2024-04-25T05:53:57.542945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'18","last_scrub_stamp":"2024-04-25T05:51:28.373733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:28.373733+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":853,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537874+0000","last_change":"2024-04-25T05:52:50.354986+0000","last_active":"2024-04-25T05:53:57.537874+0000","last_peered":"2024-04-25T05:53:57.537874+0000","last_clean":"2024-04-25T05:53:57.537874+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:53:57.537874+0000","last_undegraded":"2024-04-25T05:53:57.537874+0000","last_fullsized":"2024-04-25T05:53:57.537874+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:50.354868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:52:50.354868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544292+0000","last_change":"2024-04-25T05:52:44.084490+0000","last_active":"2024-04-25T05:53:57.544292+0000","last_peered":"2024-04-25T05:53:57.544292+0000","last_clean":"2024-04-25T05:53:57.544292+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:53:57.544292+0000","last_undegraded":"2024-04-25T05:53:57.544292+0000","last_fullsized":"2024-04-25T05:53:57.544292+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.084411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:24.037408+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.084411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:09.972942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1101,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533874+0000","last_change":"2024-04-25T05:53:48.751193+0000","last_active":"2024-04-25T05:53:57.533874+0000","last_peered":"2024-04-25T05:53:57.533874+0000","last_clean":"2024-04-25T05:53:57.533874+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:53:57.533874+0000","last_undegraded":"2024-04-25T05:53:57.533874+0000","last_fullsized":"2024-04-25T05:53:57.533874+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:48.751077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:48.751077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.118419+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"320'27","reported_seq":191,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545956+0000","last_change":"2024-04-25T05:53:32.185728+0000","last_active":"2024-04-25T05:53:57.545956+0000","last_peered":"2024-04-25T05:53:57.545956+0000","last_clean":"2024-04-25T05:53:57.545956+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:53:57.545956+0000","last_undegraded":"2024-04-25T05:53:57.545956+0000","last_fullsized":"2024-04-25T05:53:57.545956+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'27","last_scrub_stamp":"2024-04-25T05:53:32.185584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:32.185584+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.846897+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1111,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532923+0000","last_change":"2024-04-25T05:52:44.356015+0000","last_active":"2024-04-25T05:53:57.532923+0000","last_peered":"2024-04-25T05:53:57.532923+0000","last_clean":"2024-04-25T05:53:57.532923+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:53:57.532923+0000","last_undegraded":"2024-04-25T05:53:57.532923+0000","last_fullsized":"2024-04-25T05:53:57.532923+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.355921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.355921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:12.000445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1062,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542839+0000","last_change":"2024-04-25T05:51:30.313765+0000","last_active":"2024-04-25T05:53:57.542839+0000","last_peered":"2024-04-25T05:53:57.542839+0000","last_clean":"2024-04-25T05:53:57.542839+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:53:57.542839+0000","last_undegraded":"2024-04-25T05:53:57.542839+0000","last_fullsized":"2024-04-25T05:53:57.542839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.313655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.313655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":437,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.553799+0000","last_change":"2024-04-25T05:53:23.378618+0000","last_active":"2024-04-25T05:53:57.553799+0000","last_peered":"2024-04-25T05:53:57.553799+0000","last_clean":"2024-04-25T05:53:57.553799+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:53:57.553799+0000","last_undegraded":"2024-04-25T05:53:57.553799+0000","last_fullsized":"2024-04-25T05:53:57.553799+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.378498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.378498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:40.114889+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"320'26","reported_seq":432,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544173+0000","last_change":"2024-04-25T05:53:26.982110+0000","last_active":"2024-04-25T05:53:57.544173+0000","last_peered":"2024-04-25T05:53:57.544173+0000","last_clean":"2024-04-25T05:53:57.544173+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:53:57.544173+0000","last_undegraded":"2024-04-25T05:53:57.544173+0000","last_fullsized":"2024-04-25T05:53:57.544173+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:26.981810+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:53:26.981810+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.388901+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545322+0000","last_change":"2024-04-25T05:53:22.083599+0000","last_active":"2024-04-25T05:53:57.545322+0000","last_peered":"2024-04-25T05:53:57.545322+0000","last_clean":"2024-04-25T05:53:57.545322+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:53:57.545322+0000","last_undegraded":"2024-04-25T05:53:57.545322+0000","last_fullsized":"2024-04-25T05:53:57.545322+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:22.083459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.083459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.639562+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1331,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542509+0000","last_change":"2024-04-25T05:50:51.401770+0000","last_active":"2024-04-25T05:53:57.542509+0000","last_peered":"2024-04-25T05:53:57.542509+0000","last_clean":"2024-04-25T05:53:57.542509+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:53:57.542509+0000","last_undegraded":"2024-04-25T05:53:57.542509+0000","last_fullsized":"2024-04-25T05:53:57.542509+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:50:51.401649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.401649+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":632,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.548782+0000","last_change":"2024-04-25T05:52:46.212477+0000","last_active":"2024-04-25T05:53:57.548782+0000","last_peered":"2024-04-25T05:53:57.548782+0000","last_clean":"2024-04-25T05:53:57.548782+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:53:57.548782+0000","last_undegraded":"2024-04-25T05:53:57.548782+0000","last_fullsized":"2024-04-25T05:53:57.548782+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.212334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:52.077914+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.212334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:15.795169+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"320'32","reported_seq":1128,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538220+0000","last_change":"2024-04-25T05:53:25.647571+0000","last_active":"2024-04-25T05:53:57.538220+0000","last_peered":"2024-04-25T05:53:57.538220+0000","last_clean":"2024-04-25T05:53:57.538220+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:53:57.538220+0000","last_undegraded":"2024-04-25T05:53:57.538220+0000","last_fullsized":"2024-04-25T05:53:57.538220+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:53:25.647413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.647413+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:34.576268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1105,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.532820+0000","last_change":"2024-04-25T05:53:30.687157+0000","last_active":"2024-04-25T05:53:57.532820+0000","last_peered":"2024-04-25T05:53:57.532820+0000","last_clean":"2024-04-25T05:53:57.532820+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:53:57.532820+0000","last_undegraded":"2024-04-25T05:53:57.532820+0000","last_fullsized":"2024-04-25T05:53:57.532820+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.687046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.687046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.184397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":514,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544290+0000","last_change":"2024-04-25T05:51:53.967723+0000","last_active":"2024-04-25T05:53:57.544290+0000","last_peered":"2024-04-25T05:53:57.544290+0000","last_clean":"2024-04-25T05:53:57.544290+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:53:57.544290+0000","last_undegraded":"2024-04-25T05:53:57.544290+0000","last_fullsized":"2024-04-25T05:53:57.544290+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'14","last_scrub_stamp":"2024-04-25T05:51:53.967616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:53.967616+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":169,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546134+0000","last_change":"2024-04-25T05:51:40.110100+0000","last_active":"2024-04-25T05:53:57.546134+0000","last_peered":"2024-04-25T05:53:57.546134+0000","last_clean":"2024-04-25T05:53:57.546134+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:53:57.546134+0000","last_undegraded":"2024-04-25T05:53:57.546134+0000","last_fullsized":"2024-04-25T05:53:57.546134+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:22.972698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:22.972698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"320'24","reported_seq":1232,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547105+0000","last_change":"2024-04-25T05:53:21.989605+0000","last_active":"2024-04-25T05:53:57.547105+0000","last_peered":"2024-04-25T05:53:57.547105+0000","last_clean":"2024-04-25T05:53:57.547105+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:53:57.547105+0000","last_undegraded":"2024-04-25T05:53:57.547105+0000","last_fullsized":"2024-04-25T05:53:57.547105+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:53:21.989507+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.989507+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.602540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":679,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546980+0000","last_change":"2024-04-25T05:52:05.839366+0000","last_active":"2024-04-25T05:53:57.546980+0000","last_peered":"2024-04-25T05:53:57.546980+0000","last_clean":"2024-04-25T05:53:57.546980+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:53:57.546980+0000","last_undegraded":"2024-04-25T05:53:57.546980+0000","last_fullsized":"2024-04-25T05:53:57.546980+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:52:05.839241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:05.839241+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1092,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541074+0000","last_change":"2024-04-25T05:53:25.341898+0000","last_active":"2024-04-25T05:53:57.541074+0000","last_peered":"2024-04-25T05:53:57.541074+0000","last_clean":"2024-04-25T05:53:57.541074+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:53:57.541074+0000","last_undegraded":"2024-04-25T05:53:57.541074+0000","last_fullsized":"2024-04-25T05:53:57.541074+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:25.334571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.334571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:52.518338+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1108,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542912+0000","last_change":"2024-04-25T05:51:40.083901+0000","last_active":"2024-04-25T05:53:57.542912+0000","last_peered":"2024-04-25T05:53:57.542912+0000","last_clean":"2024-04-25T05:53:57.542912+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:53:57.542912+0000","last_undegraded":"2024-04-25T05:53:57.542912+0000","last_fullsized":"2024-04-25T05:53:57.542912+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:06.095697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:06.095697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"320'19","reported_seq":639,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541690+0000","last_change":"2024-04-25T05:51:23.925876+0000","last_active":"2024-04-25T05:53:57.541690+0000","last_peered":"2024-04-25T05:53:57.541690+0000","last_clean":"2024-04-25T05:53:57.541690+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:53:57.541690+0000","last_undegraded":"2024-04-25T05:53:57.541690+0000","last_fullsized":"2024-04-25T05:53:57.541690+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'19","last_scrub_stamp":"2024-04-25T05:51:23.925746+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:51:23.925746+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534509+0000","last_change":"2024-04-25T05:52:45.364101+0000","last_active":"2024-04-25T05:53:57.534509+0000","last_peered":"2024-04-25T05:53:57.534509+0000","last_clean":"2024-04-25T05:53:57.534509+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:53:57.534509+0000","last_undegraded":"2024-04-25T05:53:57.534509+0000","last_fullsized":"2024-04-25T05:53:57.534509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:45.363990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:52:45.363990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.943288+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1092,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543334+0000","last_change":"2024-04-25T05:51:40.102243+0000","last_active":"2024-04-25T05:53:57.543334+0000","last_peered":"2024-04-25T05:53:57.543334+0000","last_clean":"2024-04-25T05:53:57.543334+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:53:57.543334+0000","last_undegraded":"2024-04-25T05:53:57.543334+0000","last_fullsized":"2024-04-25T05:53:57.543334+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:49.427685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:49.427685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":177,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544888+0000","last_change":"2024-04-25T05:53:52.262152+0000","last_active":"2024-04-25T05:53:57.544888+0000","last_peered":"2024-04-25T05:53:57.544888+0000","last_clean":"2024-04-25T05:53:57.544888+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:53:57.544888+0000","last_undegraded":"2024-04-25T05:53:57.544888+0000","last_fullsized":"2024-04-25T05:53:57.544888+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:52.261997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:52.261997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.217729+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"406'31","reported_seq":1163,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546728+0000","last_change":"2024-04-25T05:50:52.402008+0000","last_active":"2024-04-25T05:53:57.546728+0000","last_peered":"2024-04-25T05:53:57.546728+0000","last_clean":"2024-04-25T05:53:57.546728+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:53:57.546728+0000","last_undegraded":"2024-04-25T05:53:57.546728+0000","last_fullsized":"2024-04-25T05:53:57.546728+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'31","last_scrub_stamp":"2024-04-25T05:50:52.401872+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:50:52.401872+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544368+0000","last_change":"2024-04-25T05:52:42.094827+0000","last_active":"2024-04-25T05:53:57.544368+0000","last_peered":"2024-04-25T05:53:57.544368+0000","last_clean":"2024-04-25T05:53:57.544368+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:53:57.544368+0000","last_undegraded":"2024-04-25T05:53:57.544368+0000","last_fullsized":"2024-04-25T05:53:57.544368+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:42.094672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:52:42.094672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:53:59.738692+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1103,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533689+0000","last_change":"2024-04-25T05:52:53.417453+0000","last_active":"2024-04-25T05:53:57.533689+0000","last_peered":"2024-04-25T05:53:57.533689+0000","last_clean":"2024-04-25T05:53:57.533689+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:53:57.533689+0000","last_undegraded":"2024-04-25T05:53:57.533689+0000","last_fullsized":"2024-04-25T05:53:57.533689+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:53.417346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.417346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:21.686711+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1100,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543772+0000","last_change":"2024-04-25T05:51:40.104333+0000","last_active":"2024-04-25T05:53:57.543772+0000","last_peered":"2024-04-25T05:53:57.543772+0000","last_clean":"2024-04-25T05:53:57.543772+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:53:57.543772+0000","last_undegraded":"2024-04-25T05:53:57.543772+0000","last_fullsized":"2024-04-25T05:53:57.543772+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:49:41.201897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:39:03.160752+0000","last_clean_scrub_stamp":"2024-04-25T05:49:41.201897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"320'8","reported_seq":1110,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544182+0000","last_change":"2024-04-25T05:53:56.692900+0000","last_active":"2024-04-25T05:53:57.544182+0000","last_peered":"2024-04-25T05:53:57.544182+0000","last_clean":"2024-04-25T05:53:57.544182+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:53:57.544182+0000","last_undegraded":"2024-04-25T05:53:57.544182+0000","last_fullsized":"2024-04-25T05:53:57.544182+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'8","last_scrub_stamp":"2024-04-25T05:53:56.692767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:56.692767+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:15.711610+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545769+0000","last_change":"2024-04-25T05:53:50.201657+0000","last_active":"2024-04-25T05:53:57.545769+0000","last_peered":"2024-04-25T05:53:57.545769+0000","last_clean":"2024-04-25T05:53:57.545769+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:53:57.545769+0000","last_undegraded":"2024-04-25T05:53:57.545769+0000","last_fullsized":"2024-04-25T05:53:57.545769+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:50.201525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:50.201525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:57.278709+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":809,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538814+0000","last_change":"2024-04-25T05:52:44.398454+0000","last_active":"2024-04-25T05:53:57.538814+0000","last_peered":"2024-04-25T05:53:57.538814+0000","last_clean":"2024-04-25T05:53:57.538814+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:53:57.538814+0000","last_undegraded":"2024-04-25T05:53:57.538814+0000","last_fullsized":"2024-04-25T05:53:57.538814+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:44.398379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:52:44.398379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.323065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"74'1","reported_seq":1094,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533677+0000","last_change":"2024-04-25T05:51:57.236443+0000","last_active":"2024-04-25T05:53:57.533677+0000","last_peered":"2024-04-25T05:53:57.533677+0000","last_clean":"2024-04-25T05:53:57.533677+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:53:57.533677+0000","last_undegraded":"2024-04-25T05:53:57.533677+0000","last_fullsized":"2024-04-25T05:53:57.533677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-04-25T05:51:57.236311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:51:57.236311+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"320'31","reported_seq":929,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538059+0000","last_change":"2024-04-25T05:53:22.290678+0000","last_active":"2024-04-25T05:53:57.538059+0000","last_peered":"2024-04-25T05:53:57.538059+0000","last_clean":"2024-04-25T05:53:57.538059+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:53:57.538059+0000","last_undegraded":"2024-04-25T05:53:57.538059+0000","last_fullsized":"2024-04-25T05:53:57.538059+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'31","last_scrub_stamp":"2024-04-25T05:53:22.290465+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:53:22.290465+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:47.697555+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1108,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543983+0000","last_change":"2024-04-25T05:53:28.655967+0000","last_active":"2024-04-25T05:53:57.543983+0000","last_peered":"2024-04-25T05:53:57.543983+0000","last_clean":"2024-04-25T05:53:57.543983+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:53:57.543983+0000","last_undegraded":"2024-04-25T05:53:57.543983+0000","last_fullsized":"2024-04-25T05:53:57.543983+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:28.655843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:53:28.655843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:45.413792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":178,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545645+0000","last_change":"2024-04-25T05:53:34.135232+0000","last_active":"2024-04-25T05:53:57.545645+0000","last_peered":"2024-04-25T05:53:57.545645+0000","last_clean":"2024-04-25T05:53:57.545645+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:53:57.545645+0000","last_undegraded":"2024-04-25T05:53:57.545645+0000","last_fullsized":"2024-04-25T05:53:57.545645+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:34.134907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:34.134907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:50.959435+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1090,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541575+0000","last_change":"2024-04-25T05:53:53.649554+0000","last_active":"2024-04-25T05:53:57.541575+0000","last_peered":"2024-04-25T05:53:57.541575+0000","last_clean":"2024-04-25T05:53:57.541575+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:53:57.541575+0000","last_undegraded":"2024-04-25T05:53:57.541575+0000","last_fullsized":"2024-04-25T05:53:57.541575+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:53.649439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.649439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:22.147543+0000","scrub_duration":261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"320'32","reported_seq":184,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546047+0000","last_change":"2024-04-25T05:51:40.147090+0000","last_active":"2024-04-25T05:53:57.546047+0000","last_peered":"2024-04-25T05:53:57.546047+0000","last_clean":"2024-04-25T05:53:57.546047+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:53:57.546047+0000","last_undegraded":"2024-04-25T05:53:57.546047+0000","last_fullsized":"2024-04-25T05:53:57.546047+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:50:48.098124+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":31,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1086,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541008+0000","last_change":"2024-04-25T05:50:51.112478+0000","last_active":"2024-04-25T05:53:57.541008+0000","last_peered":"2024-04-25T05:53:57.541008+0000","last_clean":"2024-04-25T05:53:57.541008+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:53:57.541008+0000","last_undegraded":"2024-04-25T05:53:57.541008+0000","last_fullsized":"2024-04-25T05:53:57.541008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:50:51.112337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:50:51.112337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":994,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.537904+0000","last_change":"2024-04-25T05:52:53.341410+0000","last_active":"2024-04-25T05:53:57.537904+0000","last_peered":"2024-04-25T05:53:57.537904+0000","last_clean":"2024-04-25T05:53:57.537904+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:53:57.537904+0000","last_undegraded":"2024-04-25T05:53:57.537904+0000","last_fullsized":"2024-04-25T05:53:57.537904+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:52:53.341283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:53.341283+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:03.764198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"507'2","reported_seq":1122,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.534187+0000","last_change":"2024-04-25T05:52:08.192825+0000","last_active":"2024-04-25T05:53:57.534187+0000","last_peered":"2024-04-25T05:53:57.534187+0000","last_clean":"2024-04-25T05:53:57.534187+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:53:57.534187+0000","last_undegraded":"2024-04-25T05:53:57.534187+0000","last_fullsized":"2024-04-25T05:53:57.534187+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_clean_scrub_stamp":"2024-04-25T05:52:08.192693+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"320'24","reported_seq":1109,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538754+0000","last_change":"2024-04-25T05:51:18.436527+0000","last_active":"2024-04-25T05:53:57.538754+0000","last_peered":"2024-04-25T05:53:57.538754+0000","last_clean":"2024-04-25T05:53:57.538754+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:53:57.538754+0000","last_undegraded":"2024-04-25T05:53:57.538754+0000","last_fullsized":"2024-04-25T05:53:57.538754+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'24","last_scrub_stamp":"2024-04-25T05:51:18.436370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:51:18.436370+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1128,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546741+0000","last_change":"2024-04-25T05:53:24.969033+0000","last_active":"2024-04-25T05:53:57.546741+0000","last_peered":"2024-04-25T05:53:57.546741+0000","last_clean":"2024-04-25T05:53:57.546741+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:53:57.546741+0000","last_undegraded":"2024-04-25T05:53:57.546741+0000","last_fullsized":"2024-04-25T05:53:57.546741+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:24.968897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:24.968897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:46.507723+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":177,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.538000+0000","last_change":"2024-04-25T05:52:35.374907+0000","last_active":"2024-04-25T05:53:57.538000+0000","last_peered":"2024-04-25T05:53:57.538000+0000","last_clean":"2024-04-25T05:53:57.538000+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:53:57.538000+0000","last_undegraded":"2024-04-25T05:53:57.538000+0000","last_fullsized":"2024-04-25T05:53:57.538000+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:35.374829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:52:35.374829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1083,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.542913+0000","last_change":"2024-04-25T05:50:05.284540+0000","last_active":"2024-04-25T05:53:57.542913+0000","last_peered":"2024-04-25T05:53:57.542913+0000","last_clean":"2024-04-25T05:53:57.542913+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:53:57.542913+0000","last_undegraded":"2024-04-25T05:53:57.542913+0000","last_fullsized":"2024-04-25T05:53:57.542913+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:05.284401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:50:05.284401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"320'34","reported_seq":461,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.553639+0000","last_change":"2024-04-25T05:53:25.409053+0000","last_active":"2024-04-25T05:53:57.553639+0000","last_peered":"2024-04-25T05:53:57.553639+0000","last_clean":"2024-04-25T05:53:57.553639+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:53:57.553639+0000","last_undegraded":"2024-04-25T05:53:57.553639+0000","last_fullsized":"2024-04-25T05:53:57.553639+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'34","last_scrub_stamp":"2024-04-25T05:53:25.408390+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:53:25.408390+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:25.958321+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":169,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547057+0000","last_change":"2024-04-25T05:51:40.101375+0000","last_active":"2024-04-25T05:53:57.547057+0000","last_peered":"2024-04-25T05:53:57.547057+0000","last_clean":"2024-04-25T05:53:57.547057+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:53:57.547057+0000","last_undegraded":"2024-04-25T05:53:57.547057+0000","last_fullsized":"2024-04-25T05:53:57.547057+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:30.835919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:51:30.835919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1106,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541802+0000","last_change":"2024-04-25T05:51:40.109523+0000","last_active":"2024-04-25T05:53:57.541802+0000","last_peered":"2024-04-25T05:53:57.541802+0000","last_clean":"2024-04-25T05:53:57.541802+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:53:57.541802+0000","last_undegraded":"2024-04-25T05:53:57.541802+0000","last_fullsized":"2024-04-25T05:53:57.541802+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:15.989325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:15.989325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1123,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547498+0000","last_change":"2024-04-25T05:53:23.056922+0000","last_active":"2024-04-25T05:53:57.547498+0000","last_peered":"2024-04-25T05:53:57.547498+0000","last_clean":"2024-04-25T05:53:57.547498+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:53:57.547498+0000","last_undegraded":"2024-04-25T05:53:57.547498+0000","last_fullsized":"2024-04-25T05:53:57.547498+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:23.056816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:53:23.056816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:43.493056+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"320'13","reported_seq":601,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.547037+0000","last_change":"2024-04-25T05:53:51.040346+0000","last_active":"2024-04-25T05:53:57.547037+0000","last_peered":"2024-04-25T05:53:57.547037+0000","last_clean":"2024-04-25T05:53:57.547037+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:53:57.547037+0000","last_undegraded":"2024-04-25T05:53:57.547037+0000","last_fullsized":"2024-04-25T05:53:57.547037+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'13","last_scrub_stamp":"2024-04-25T05:53:51.040191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:53:51.040191+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:13.024810+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":419,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544274+0000","last_change":"2024-04-25T05:52:47.057845+0000","last_active":"2024-04-25T05:53:57.544274+0000","last_peered":"2024-04-25T05:53:57.544274+0000","last_clean":"2024-04-25T05:53:57.544274+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:53:57.544274+0000","last_undegraded":"2024-04-25T05:53:57.544274+0000","last_fullsized":"2024-04-25T05:53:57.544274+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_clean_scrub_stamp":"2024-04-25T05:52:47.057728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:13.968343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1082,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.533509+0000","last_change":"2024-04-25T05:51:54.167227+0000","last_active":"2024-04-25T05:53:57.533509+0000","last_peered":"2024-04-25T05:53:57.533509+0000","last_clean":"2024-04-25T05:53:57.533509+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:53:57.533509+0000","last_undegraded":"2024-04-25T05:53:57.533509+0000","last_fullsized":"2024-04-25T05:53:57.533509+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:51:54.167105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:51:54.167105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":420,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543682+0000","last_change":"2024-04-25T05:53:30.164095+0000","last_active":"2024-04-25T05:53:57.543682+0000","last_peered":"2024-04-25T05:53:57.543682+0000","last_clean":"2024-04-25T05:53:57.543682+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:53:57.543682+0000","last_undegraded":"2024-04-25T05:53:57.543682+0000","last_fullsized":"2024-04-25T05:53:57.543682+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:30.163971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:53:30.163971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:59.864542+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"431'34","reported_seq":907,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546681+0000","last_change":"2024-04-25T05:52:02.822759+0000","last_active":"2024-04-25T05:53:57.546681+0000","last_peered":"2024-04-25T05:53:57.546681+0000","last_clean":"2024-04-25T05:53:57.546681+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:53:57.546681+0000","last_undegraded":"2024-04-25T05:53:57.546681+0000","last_fullsized":"2024-04-25T05:53:57.546681+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'34","last_scrub_stamp":"2024-04-25T05:52:02.822618+0000","last_deep_scrub":"431'34","last_deep_scrub_stamp":"2024-04-25T05:49:06.169828+0000","last_clean_scrub_stamp":"2024-04-25T05:52:02.822618+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":419,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544235+0000","last_change":"2024-04-25T05:52:46.070334+0000","last_active":"2024-04-25T05:53:57.544235+0000","last_peered":"2024-04-25T05:53:57.544235+0000","last_clean":"2024-04-25T05:53:57.544235+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:53:57.544235+0000","last_undegraded":"2024-04-25T05:53:57.544235+0000","last_fullsized":"2024-04-25T05:53:57.544235+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:46.070183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:52:46.070183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.824200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":425,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543783+0000","last_change":"2024-04-25T05:53:53.973887+0000","last_active":"2024-04-25T05:53:57.543783+0000","last_peered":"2024-04-25T05:53:57.543783+0000","last_clean":"2024-04-25T05:53:57.543783+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:53:57.543783+0000","last_undegraded":"2024-04-25T05:53:57.543783+0000","last_fullsized":"2024-04-25T05:53:57.543783+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:53.973767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.973767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:09.385007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":177,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.544796+0000","last_change":"2024-04-25T05:53:54.281982+0000","last_active":"2024-04-25T05:53:57.544796+0000","last_peered":"2024-04-25T05:53:57.544796+0000","last_clean":"2024-04-25T05:53:57.544796+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:53:57.544796+0000","last_undegraded":"2024-04-25T05:53:57.544796+0000","last_fullsized":"2024-04-25T05:53:57.544796+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:54.281861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:53:54.281861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:20.076853+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"320'51","reported_seq":1397,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546829+0000","last_change":"2024-04-25T05:51:40.151139+0000","last_active":"2024-04-25T05:53:57.546829+0000","last_peered":"2024-04-25T05:53:57.546829+0000","last_clean":"2024-04-25T05:53:57.546829+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:53:57.546829+0000","last_undegraded":"2024-04-25T05:53:57.546829+0000","last_fullsized":"2024-04-25T05:53:57.546829+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'51","last_scrub_stamp":"2024-04-25T05:50:40.381455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:50:40.381455+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1103,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.539188+0000","last_change":"2024-04-25T05:48:59.027968+0000","last_active":"2024-04-25T05:53:57.539188+0000","last_peered":"2024-04-25T05:53:57.539188+0000","last_clean":"2024-04-25T05:53:57.539188+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:53:57.539188+0000","last_undegraded":"2024-04-25T05:53:57.539188+0000","last_fullsized":"2024-04-25T05:53:57.539188+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:48:59.027887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:48:59.027887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":648,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546449+0000","last_change":"2024-04-25T05:52:51.042784+0000","last_active":"2024-04-25T05:53:57.546449+0000","last_peered":"2024-04-25T05:53:57.546449+0000","last_clean":"2024-04-25T05:53:57.546449+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:53:57.546449+0000","last_undegraded":"2024-04-25T05:53:57.546449+0000","last_fullsized":"2024-04-25T05:53:57.546449+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:52:51.042665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:52:51.042665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1069,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.543420+0000","last_change":"2024-04-25T05:50:13.195581+0000","last_active":"2024-04-25T05:53:57.543420+0000","last_peered":"2024-04-25T05:53:57.543420+0000","last_clean":"2024-04-25T05:53:57.543420+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:53:57.543420+0000","last_undegraded":"2024-04-25T05:53:57.543420+0000","last_fullsized":"2024-04-25T05:53:57.543420+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:13.195419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:50:13.195419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"320'26","reported_seq":1184,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546320+0000","last_change":"2024-04-25T05:53:21.029300+0000","last_active":"2024-04-25T05:53:57.546320+0000","last_peered":"2024-04-25T05:53:57.546320+0000","last_clean":"2024-04-25T05:53:57.546320+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:53:57.546320+0000","last_undegraded":"2024-04-25T05:53:57.546320+0000","last_fullsized":"2024-04-25T05:53:57.546320+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'26","last_scrub_stamp":"2024-04-25T05:53:21.029139+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:53:21.029139+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:30.616379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":178,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.545769+0000","last_change":"2024-04-25T05:53:33.165542+0000","last_active":"2024-04-25T05:53:57.545769+0000","last_peered":"2024-04-25T05:53:57.545769+0000","last_clean":"2024-04-25T05:53:57.545769+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:53:57.545769+0000","last_undegraded":"2024-04-25T05:53:57.545769+0000","last_fullsized":"2024-04-25T05:53:57.545769+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:33.165409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:33.165409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:03.084305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1095,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.541115+0000","last_change":"2024-04-25T05:50:56.025000+0000","last_active":"2024-04-25T05:53:57.541115+0000","last_peered":"2024-04-25T05:53:57.541115+0000","last_clean":"2024-04-25T05:53:57.541115+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:53:57.541115+0000","last_undegraded":"2024-04-25T05:53:57.541115+0000","last_fullsized":"2024-04-25T05:53:57.541115+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:50:56.024879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:50:56.024879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":632,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-25T05:53:57.546537+0000","last_change":"2024-04-25T05:53:56.092940+0000","last_active":"2024-04-25T05:53:57.546537+0000","last_peered":"2024-04-25T05:53:57.546537+0000","last_clean":"2024-04-25T05:53:57.546537+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:53:57.546537+0000","last_undegraded":"2024-04-25T05:53:57.546537+0000","last_fullsized":"2024-04-25T05:53:57.546537+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:56.092823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:53:56.092823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:17.662839+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":200671256,"num_objects":3063,"num_object_clones":0,"num_object_copies":9189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257511,"num_write_kb":8338305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28230,"num_bytes_recovered":1850015744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1066938368,"data_stored":1066926152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80309,"ondisk_log_size":80309,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1180,"num_read_kb":1175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1389,"num_read_kb":1114,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":988,"ondisk_log_size":988,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070709,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213284,"kb_used_data":275536,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92531868,"statfs":{"total":95995035648,"available":94752632832,"internally_reserved":0,"allocated":282148864,"data_stored":281446693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":960231882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365049,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969924,"kb_used_data":112696,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92775228,"statfs":{"total":95995035648,"available":95001833472,"internally_reserved":0,"allocated":115400704,"data_stored":114721925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":877787590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575122,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174700,"kb_used_data":82780,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":92570452,"statfs":{"total":95995035648,"available":94792142848,"internally_reserved":0,"allocated":84766720,"data_stored":84050969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856848,"kb_used_data":77316,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92888304,"statfs":{"total":95995035648,"available":95117623296,"internally_reserved":0,"allocated":79171584,"data_stored":78479858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411597,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822904,"kb_used_data":105904,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92922248,"statfs":{"total":95995035648,"available":95152381952,"internally_reserved":0,"allocated":108445696,"data_stored":107767517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149959,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302756,"kb_used_data":89428,"kb_used_omap":1,"kb_used_meta":1213118,"kb_avail":92442396,"statfs":{"total":95995035648,"available":94661013504,"internally_reserved":0,"allocated":91574272,"data_stored":90866803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1242233287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986434,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157156,"kb_used_data":337428,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92587996,"statfs":{"total":95995035648,"available":94810107904,"internally_reserved":0,"allocated":345526272,"data_stored":344819003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":839383493},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692094,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691148,"kb_used_data":60220,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93054004,"statfs":{"total":95995035648,"available":95287300096,"internally_reserved":0,"allocated":61665280,"data_stored":60982401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67633152,"data_stored":67633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78184448,"data_stored":78184448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":95227904,"data_stored":95223832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70057984,"data_stored":70057984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":335941632,"data_stored":335937560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54460416,"data_stored":54460416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":103813120,"data_stored":103809048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":261619712,"data_stored":261619712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:54:04.412 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-04-25T05:54:04.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:04 smithi060 ceph-mon[30898]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:54:04.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:04 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:54:04.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:04 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:54:04.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:04.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:04.501 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:04.522 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:04.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:04.646 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:04.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:04.771 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:04.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:04.894 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:04.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:04.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:05.020 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:05.034 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:05.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:05.147 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:05.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:05.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:54:05.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:54:05.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:54:05.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: pgmap v1089: 158 pgs: 158 active+clean; 57 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 992 op/s 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: osdmap e579: 8 total, 8 up, 8 in 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[37492]: osdmap e580: 8 total, 8 up, 8 in 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:54:05.273 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: pgmap v1089: 158 pgs: 158 active+clean; 57 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 992 op/s 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: osdmap e579: 8 total, 8 up, 8 in 2024-04-25T05:54:05.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:05 smithi043 ceph-mon[27010]: osdmap e580: 8 total, 8 up, 8 in 2024-04-25T05:54:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.275 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:05.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.398 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:05.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:54:05.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:54:05.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: pgmap v1089: 158 pgs: 158 active+clean; 57 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 992 op/s 2024-04-25T05:54:05.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: osdmap e579: 8 total, 8 up, 8 in 2024-04-25T05:54:05.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:05 smithi060 ceph-mon[30898]: osdmap e580: 8 total, 8 up, 8 in 2024-04-25T05:54:05.500 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:05.525 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:05.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:05.651 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:05.656 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:05.657 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:05.657 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:54:05.657 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:54:05.657 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:05.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:05.776 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:05.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:05.901 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:05.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:06.003 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:06.024 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:06.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:06.157 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is set 2024-04-25T05:54:06.253 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:06.275 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:06.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:06.397 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:06.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:54:06.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:54:06.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1611016646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:54:06.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:54:06.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1611016646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:06 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:54:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:06 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:54:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:06 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1611016646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-25T05:54:06.499 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:06.521 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:06.531 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:54:06.550 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:54:06.570 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:54:06.590 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:54:06.611 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:54:06.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:06.633 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:54:06.649 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:06.654 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:54:06.674 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:54:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:54:06.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:06.772 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:06.874 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:06.897 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:06.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:06.999 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:07.022 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:07.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:07.148 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:07.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:07.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.395 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:07.496 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 11.4 scrub starts 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 11.4 scrub ok 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 6.13 deep-scrub starts 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 6.13 deep-scrub ok 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: pgmap v1092: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 510 op/s 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 4.1e scrub starts 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: 4.1e scrub ok 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:07.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:54:07.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: osdmap e581: 8 total, 8 up, 8 in 2024-04-25T05:54:07.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:54:07.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:54:07.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:07 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4274928028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:07.619 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 11.4 scrub starts 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 11.4 scrub ok 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 6.13 deep-scrub starts 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 6.13 deep-scrub ok 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: pgmap v1092: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 510 op/s 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 4.1e scrub starts 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: 4.1e scrub ok 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:54:07.636 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: osdmap e581: 8 total, 8 up, 8 in 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4274928028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 11.4 scrub starts 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 11.4 scrub ok 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 6.13 deep-scrub starts 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 6.13 deep-scrub ok 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: pgmap v1092: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 510 op/s 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 4.1e scrub starts 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: 4.1e scrub ok 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-25T05:54:07.637 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: osdmap e581: 8 total, 8 up, 8 in 2024-04-25T05:54:07.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:54:07.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:54:07.638 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:07 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4274928028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:07.640 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:07.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:07.763 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:07.865 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:07.886 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:07.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:07.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:08.013 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:08.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:08.141 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-25T05:54:08.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:08.244 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:08.265 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:08.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:08.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:08 smithi043 ceph-mon[27010]: osdmap e582: 8 total, 8 up, 8 in 2024-04-25T05:54:08.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:08 smithi043 ceph-mon[37492]: osdmap e582: 8 total, 8 up, 8 in 2024-04-25T05:54:08.392 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:08.495 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:08.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:08 smithi060 ceph-mon[30898]: osdmap e582: 8 total, 8 up, 8 in 2024-04-25T05:54:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:08.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:08.647 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:08.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:08.774 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:08.876 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:08.902 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:08.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:09.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:09.029 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:09.097 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:09.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:09.152 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:09.164 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 11 pgp_num to 16 2024-04-25T05:54:09.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[37492]: pgmap v1095: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 234 op/s 2024-04-25T05:54:09.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3748803741' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.220 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.220 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[27010]: pgmap v1095: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 234 op/s 2024-04-25T05:54:09.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3748803741' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:09.278 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:09.380 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.403 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:09.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:09 smithi060 ceph-mon[30898]: pgmap v1095: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 0 B/s wr, 234 op/s 2024-04-25T05:54:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:09 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3748803741' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-25T05:54:09.504 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:09.524 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:09.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:09.664 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T05:54:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:09.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:09.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:09.803 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:09.883 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:09.907 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:09.910 INFO:teuthology.orchestra.run.smithi043.stdout:ERROR: (22) Invalid argument 2024-04-25T05:54:09.910 INFO:teuthology.orchestra.run.smithi043.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T05:54:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:09.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:10.031 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:10.051 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:10.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:10.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:10.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:10.298 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:10.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:10.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:54:10.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[27010]: osdmap e583: 8 total, 8 up, 8 in 2024-04-25T05:54:10.421 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[27010]: osdmap e584: 8 total, 8 up, 8 in 2024-04-25T05:54:10.421 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:54:10.421 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[37492]: osdmap e583: 8 total, 8 up, 8 in 2024-04-25T05:54:10.422 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:10 smithi043 ceph-mon[37492]: osdmap e584: 8 total, 8 up, 8 in 2024-04-25T05:54:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:10.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:10.448 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-25T05:54:10.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-25T05:54:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:10 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-25T05:54:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:10 smithi060 ceph-mon[30898]: osdmap e583: 8 total, 8 up, 8 in 2024-04-25T05:54:10.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:10 smithi060 ceph-mon[30898]: osdmap e584: 8 total, 8 up, 8 in 2024-04-25T05:54:10.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:10.555 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:10.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:10.678 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:10.730 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:10.780 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:10.800 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:10.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:10.926 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:10.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:10.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:11.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:11.054 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:11.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-25T05:54:11.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.178 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:11.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:11.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:11.301 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:11.360 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0: {} 2024-04-25T05:54:11.380 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1: {} 2024-04-25T05:54:11.380 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:11 smithi043 ceph-mon[37492]: pgmap v1098: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s 2024-04-25T05:54:11.380 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:11 smithi043 ceph-mon[27010]: pgmap v1098: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s 2024-04-25T05:54:11.401 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2: {} 2024-04-25T05:54:11.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:11.421 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3: {} 2024-04-25T05:54:11.428 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:11.442 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4: {} 2024-04-25T05:54:11.462 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5: {} 2024-04-25T05:54:11.483 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6: {} 2024-04-25T05:54:11.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:11 smithi060 ceph-mon[30898]: pgmap v1098: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s 2024-04-25T05:54:11.503 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7: {} 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T05:54:11.505 INFO:teuthology.orchestra.run.smithi043.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T05:54:11.530 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:11.552 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:11.654 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:11.675 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:11.777 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:11.803 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:11.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2 2024-04-25T05:54:11.927 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:11.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.2' 2024-04-25T05:54:11.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-25T05:54:12.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:12.054 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:12.156 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:12.180 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:12.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:12.306 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:12.332 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3838466223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:12.333 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3838466223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:12.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:12.430 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:12.498 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3838466223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-25T05:54:12.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:12.557 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:12.659 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.682 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:12.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:12.783 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1 2024-04-25T05:54:12.806 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.1' 2024-04-25T05:54:12.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6 2024-04-25T05:54:12.933 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:12.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.6' 2024-04-25T05:54:13.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:13.060 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:13.162 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:13.186 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'unique_pool_4' removed 2024-04-25T05:54:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:13.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:13.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[27010]: pgmap v1099: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-25T05:54:13.256 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4189342585' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.257 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[37492]: pgmap v1099: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-25T05:54:13.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4189342585' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.257 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:13 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:13.318 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:13.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4 2024-04-25T05:54:13.445 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.4' 2024-04-25T05:54:13.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:13 smithi060 ceph-mon[30898]: pgmap v1099: 158 pgs: 158 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-25T05:54:13.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:13 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4189342585' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:13 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T05:54:13.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7 2024-04-25T05:54:13.571 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.7' 2024-04-25T05:54:13.576 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:13.577 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:13.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0 2024-04-25T05:54:13.694 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.0' 2024-04-25T05:54:13.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:13.817 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:13.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3 2024-04-25T05:54:13.941 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: docker: command not found 2024-04-25T05:54:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:13.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi043 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.3' 2024-04-25T05:54:14.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5 2024-04-25T05:54:14.065 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: docker: command not found 2024-04-25T05:54:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:54:14.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi060 with status 1: 'sudo docker kill -s 1 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd.5' 2024-04-25T05:54:14.076 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-25T05:54:14.087 INFO:tasks.thrashosds:joining thrashosds 2024-04-25T05:54:14.177 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:14.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:14 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:54:14.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:14 smithi043 ceph-mon[37492]: osdmap e585: 8 total, 8 up, 8 in 2024-04-25T05:54:14.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:14 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:54:14.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:14 smithi043 ceph-mon[27010]: osdmap e585: 8 total, 8 up, 8 in 2024-04-25T05:54:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:14 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T05:54:14.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:14 smithi060 ceph-mon[30898]: osdmap e585: 8 total, 8 up, 8 in 2024-04-25T05:54:14.721 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:14.721 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:14.721 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:54:14.721 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:15.449 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-25T05:54:15.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-25T05:54:15.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:15.473 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[27010]: pgmap v1101: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 70 KiB/s rd, 69 op/s 2024-04-25T05:54:15.474 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[27010]: osdmap e586: 8 total, 8 up, 8 in 2024-04-25T05:54:15.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:15.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[37492]: pgmap v1101: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 70 KiB/s rd, 69 op/s 2024-04-25T05:54:15.474 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:15 smithi043 ceph-mon[37492]: osdmap e586: 8 total, 8 up, 8 in 2024-04-25T05:54:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:15 smithi060 ceph-mon[30898]: pgmap v1101: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 70 KiB/s rd, 69 op/s 2024-04-25T05:54:15.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:15 smithi060 ceph-mon[30898]: osdmap e586: 8 total, 8 up, 8 in 2024-04-25T05:54:15.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-25T05:54:16.033 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:16.543 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:17.090 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:17.090 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:17.090 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:54:17.090 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:54:17.090 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:17.191 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is set 2024-04-25T05:54:17.192 INFO:teuthology.orchestra.run.smithi043.stderr:set pool 1 pgp_num to 11 2024-04-25T05:54:17.382 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[27010]: pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-25T05:54:17.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:17.385 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-25T05:54:17.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[37492]: pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-25T05:54:17.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:17.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:17 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-25T05:54:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:17 smithi060 ceph-mon[30898]: pgmap v1103: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-04-25T05:54:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:17.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:17 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-25T05:54:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[27010]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-25T05:54:18.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[27010]: osdmap e587: 8 total, 8 up, 8 in 2024-04-25T05:54:18.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[37492]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-25T05:54:18.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:18 smithi043 ceph-mon[37492]: osdmap e587: 8 total, 8 up, 8 in 2024-04-25T05:54:18.473 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-25T05:54:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:18 smithi060 ceph-mon[30898]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1881556892' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:18 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3742203721' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-25T05:54:18.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:18 smithi060 ceph-mon[30898]: osdmap e587: 8 total, 8 up, 8 in 2024-04-25T05:54:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:19 smithi060 ceph-mon[30898]: pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:54:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:19.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:19.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[27010]: pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:54:19.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:19.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[37492]: pgmap v1105: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-25T05:54:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:19.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:19.695 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:19.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-25T05:54:19.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155160 9.6 9:68737ed0:::benchmark_data_smithi043_129820_object33846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.24096901500002, 2024-04-25T05:54:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4685459839999999, 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "reached pg", 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155160 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054725+0000", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054726+0000", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4890000000000001e-06 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054732+0000", 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0260000000000004e-06 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054747+0000", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4762e-05 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.054756+0000", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2070000000000001e-06 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.056987+0000", 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0022311950000000001 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.057103+0000", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000115751 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6,7", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.062978+0000", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0058742639999999997 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.065122+0000", 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021449189999999999 2024-04-25T05:54:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.065931+0000", 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00080820200000000005 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.081687+0000", 2024-04-25T05:54:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015756503000000002 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.081698+0000", 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0569e-05 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519331+0000", 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4376332760000001 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for missing object", 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519417+0000", 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5554000000000001e-05 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523215+0000", 2024-04-25T05:54:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037980840000000002 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523271+0000", 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6183000000000001e-05 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.307 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155182 9.6 9:64768de4:::benchmark_data_smithi043_129820_object33868:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.21967494199998, 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.44742573, 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155182 2024-04-25T05:54:20.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076019+0000", 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076020+0000", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3370000000000001e-06 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076026+0000", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1090000000000003e-06 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076038+0000", 2024-04-25T05:54:20.310 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1559999999999999e-05 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.076046+0000", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7530000000000001e-06 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083212+0000", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071663600000000001 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083220+0000", 2024-04-25T05:54:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3010000000000007e-06 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519718+0000", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4364973889999999 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519802+0000", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4505000000000005e-05 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.519898+0000", 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5798999999999995e-05 2024-04-25T05:54:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523005+0000", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031065419999999999 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523303+0000", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000297977 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523339+0000", 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6943999999999997e-05 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523369+0000", 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9410000000000001e-05 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523445+0000", 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5743999999999999e-05 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155187 9.6 9:62aab14c:::benchmark_data_smithi043_129820_object33873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.21477734400003, 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.445533349, 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155187 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080916+0000", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080917+0000", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080922+0000", 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.989e-06 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080933+0000", 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0903e-05 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.317 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.080941+0000", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.96e-06 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083245+0000", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023036889999999998 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.083252+0000", 2024-04-25T05:54:20.318 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8340000000000001e-06 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521237+0000", 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4379850830000001 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521310+0000", 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3540000000000004e-05 2024-04-25T05:54:20.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.521382+0000", 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1266000000000005e-05 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523269+0000", 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001887122 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.321 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.322 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526171+0000", 2024-04-25T05:54:20.322 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029016379999999998 2024-04-25T05:54:20.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.322 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526300+0000", 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012934099999999999 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526353+0000", 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3245999999999998e-05 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.323 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526450+0000", 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6738999999999997e-05 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.324 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155197 9.6 9:60d19fde:::benchmark_data_smithi043_129820_object33883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.20869447500002, 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.439703379, 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.325 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155197 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:54:20.326 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.086999+0000", 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.327 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087000+0000", 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4300000000000003e-07 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087003+0000", 2024-04-25T05:54:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0079999999999998e-06 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087010+0000", 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.793e-06 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087016+0000", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7549999999999997e-06 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087222+0000", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000205167 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.087231+0000", 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0329999999999997e-06 2024-04-25T05:54:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522178+0000", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4349474170000001 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522246+0000", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7964000000000002e-05 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.522316+0000", 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0087999999999997e-05 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523459+0000", 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011426279999999999 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526260+0000", 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002801455 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526528+0000", 2024-04-25T05:54:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00026746199999999999 2024-04-25T05:54:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526589+0000", 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1754999999999994e-05 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526703+0000", 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113311 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155207 9.6 9:6f7e51c0:::benchmark_data_smithi043_129820_object33893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.19717903900005, 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4315835130000001, 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155207 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098515+0000", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098516+0000", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.423e-06 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.336 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098521+0000", 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.809e-06 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098530+0000", 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3079999999999997e-06 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098538+0000", 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3000000000000004e-06 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098698+0000", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000159975 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098707+0000", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3749999999999992e-06 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523520+0000", 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4248128410000001 2024-04-25T05:54:20.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523674+0000", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000154444 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.523838+0000", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000164067 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526225+0000", 2024-04-25T05:54:20.339 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0023871679999999998 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526772+0000", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00054641399999999995 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.529943+0000", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031711209999999998 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.529996+0000", 2024-04-25T05:54:20.340 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3053000000000002e-05 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530098+0000", 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000102215 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155209 9.6 9:69e43cc6:::benchmark_data_smithi043_129820_object33895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.19712544900005, 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.432188107, 2024-04-25T05:54:20.341 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155209 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098568+0000", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098569+0000", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1900000000000002e-07 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098572+0000", 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1999999999999999e-06 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098579+0000", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8499999999999996e-06 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098585+0000", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4659999999999998e-06 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098738+0000", 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015333 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.344 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098745+0000", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1080000000000004e-06 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525155+0000", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.42640999 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525248+0000", 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3037999999999999e-05 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.345 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.525316+0000", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8225999999999994e-05 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526464+0000", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0011481219999999999 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530363+0000", 2024-04-25T05:54:20.346 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038985299999999999 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530405+0000", 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1757e-05 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530499+0000", 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4128000000000004e-05 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.347 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530756+0000", 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000257743 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155210 9.6 9:62a79c7f:::benchmark_data_smithi043_129820_object33896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.19710090700005, 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4385105970000001, 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.348 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155210 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098593+0000", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3300000000000003e-07 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098596+0000", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1619999999999998e-06 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098601+0000", 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6149999999999996e-06 2024-04-25T05:54:20.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098606+0000", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1189999999999996e-06 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098767+0000", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016068300000000001 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.098773+0000", 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4409999999999998e-06 2024-04-25T05:54:20.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526824+0000", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4280502369999999 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.526982+0000", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015850099999999999 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.527143+0000", 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016125800000000001 2024-04-25T05:54:20.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530317+0000", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031736619999999998 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.533960+0000", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036430019999999998 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536942+0000", 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029821040000000002 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537000+0000", 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7896999999999998e-05 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537104+0000", 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000103483 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.354 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155231 9.6 9:6c10c19d:::benchmark_data_smithi043_129820_object33917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.17661160299997, 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4224660419999999, 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155231 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.355 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119082+0000", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119084+0000", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4279999999999999e-06 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.356 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119090+0000", 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2899999999999999e-06 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119101+0000", 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1277999999999999e-05 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.357 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119110+0000", 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1130000000000007e-06 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.358 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119205+0000", 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4975000000000005e-05 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.119216+0000", 2024-04-25T05:54:20.359 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0268000000000001e-05 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528403+0000", 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4091876059999999 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528524+0000", 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012039799999999999 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.528655+0000", 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013160099999999999 2024-04-25T05:54:20.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530322+0000", 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001667167 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536752+0000", 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0064295860000000002 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.363 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541312+0000", 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045599179999999996 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541439+0000", 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012706499999999999 2024-04-25T05:54:20.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541548+0000", 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109349 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155286 9.6 9:62058b0b:::benchmark_data_smithi043_129820_object33972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.070053282, 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3162657389999999, 2024-04-25T05:54:20.365 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155286 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.367 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225641+0000", 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.75e-07 2024-04-25T05:54:20.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225646+0000", 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4710000000000001e-06 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225654+0000", 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7190000000000007e-06 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.369 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225662+0000", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6510000000000001e-06 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225716+0000", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4165000000000001e-05 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.225725+0000", 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3580000000000003e-06 2024-04-25T05:54:20.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530799+0000", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.305074265 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.530941+0000", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014229499999999999 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.531083+0000", 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014207999999999999 2024-04-25T05:54:20.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.533984+0000", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029003129999999998 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536825+0000", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002841784 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541749+0000", 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049237810000000003 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541805+0000", 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6110000000000003e-05 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541906+0000", 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101072 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155290 9.6 9:6b6bab7a:::benchmark_data_smithi043_129820_object33976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.373 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.06739866500004, 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.317358917, 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155290 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.374 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228295+0000", 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228297+0000", 2024-04-25T05:54:20.375 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.359e-06 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228301+0000", 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0860000000000004e-06 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228309+0000", 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9799999999999998e-06 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.376 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228315+0000", 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8519999999999998e-06 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228486+0000", 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017009600000000001 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.228494+0000", 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.123e-06 2024-04-25T05:54:20.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532331+0000", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3038373779999999 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532444+0000", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000113113 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.532608+0000", 2024-04-25T05:54:20.378 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000163707 2024-04-25T05:54:20.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.379 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.379 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:20.380 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.536773+0000", 2024-04-25T05:54:20.380 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041653539999999996 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537161+0000", 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00038807199999999999 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545511+0000", 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0083501449999999998 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.381 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545562+0000", 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0943999999999999e-05 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.545654+0000", 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1707999999999997e-05 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155323 9.6 9:60343a60:::benchmark_data_smithi043_129820_object34009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.382 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.04659083499996, 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2987439169999999, 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155323 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249103+0000", 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.384 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249104+0000", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5000000000000002e-07 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249108+0000", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3080000000000001e-06 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249117+0000", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5180000000000004e-06 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.385 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249124+0000", 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2119999999999997e-06 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249210+0000", 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6537000000000002e-05 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.249220+0000", 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8220000000000002e-06 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.386 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534015+0000", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2847948680000001 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534132+0000", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117316 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.534248+0000", 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011565699999999999 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.387 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537116+0000", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028682479999999999 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537205+0000", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8677999999999994e-05 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547761+0000", 2024-04-25T05:54:20.388 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010556204 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547809+0000", 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7682000000000002e-05 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547847+0000", 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8117000000000001e-05 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:155332 9.6 9:63fa8193:::benchmark_data_smithi043_129820_object34018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 544.03881117399999, 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2911138019999999, 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.390 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 155332 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256883+0000", 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256884+0000", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.6099999999999999e-07 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256888+0000", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5789999999999997e-06 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256899+0000", 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1071e-05 2024-04-25T05:54:20.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256907+0000", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7109999999999995e-06 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256960+0000", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2632e-05 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:16.256965+0000", 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8270000000000003e-06 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535461+0000", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.278495739 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535548+0000", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7292000000000004e-05 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 6,7", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.535700+0000", 2024-04-25T05:54:20.394 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015115699999999999 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.537122+0000", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014224750000000001 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.541968+0000", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0048458190000000003 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547897+0000", 2024-04-25T05:54:20.395 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059294289999999999 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547940+0000", 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3050000000000003e-05 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:45:17.547996+0000", 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6060000000000002e-05 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94216 11.0 11:04369c26:::benchmark_data_smithi043_179003_object94215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.81180900499999, 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.239603538, 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94216 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483885+0000", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483886+0000", 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.68e-06 2024-04-25T05:54:20.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483911+0000", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.4275000000000001e-05 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483927+0000", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6209e-05 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483936+0000", 2024-04-25T05:54:20.399 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6270000000000006e-06 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.483995+0000", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9590999999999999e-05 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.484129+0000", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000134184 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.484266+0000", 2024-04-25T05:54:20.400 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000136938 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.487255+0000", 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002988465 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.489356+0000", 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002100874 2024-04-25T05:54:20.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723380+0000", 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.23402442500000001 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723437+0000", 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6498e-05 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.402 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.723488+0000", 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1771999999999998e-05 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94230 11.0 11:0c3febce:::benchmark_data_smithi043_179003_object94229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.761555741, 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.193247635, 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.403 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94230 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534138+0000", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.404 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534140+0000", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3350000000000001e-06 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534168+0000", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7818999999999999e-05 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534190+0000", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.1359e-05 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534203+0000", 2024-04-25T05:54:20.405 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3040000000000001e-05 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534285+0000", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2291000000000002e-05 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534464+0000", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017857600000000001 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.534640+0000", 2024-04-25T05:54:20.406 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017681899999999999 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.539282+0000", 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046420999999999997 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.539676+0000", 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00039339799999999999 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.407 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727295+0000", 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.18761967500000001 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727360+0000", 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4937000000000005e-05 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.727386+0000", 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5286000000000002e-05 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94244 11.0 11:09c229f5:::benchmark_data_smithi043_179003_object94243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.744201514, 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.17984397599999999, 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94244 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551492+0000", 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551494+0000", 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6390000000000001e-06 2024-04-25T05:54:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551516+0000", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.2483000000000001e-05 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551546+0000", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9730000000000002e-05 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.551784+0000", 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000238142 2024-04-25T05:54:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552058+0000", 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027341399999999998 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552330+0000", 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027213399999999999 2024-04-25T05:54:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.552513+0000", 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018327 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.555341+0000", 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0028275739999999998 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.415 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562661+0000", 2024-04-25T05:54:20.416 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0073205220000000003 2024-04-25T05:54:20.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.416 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.416 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731226+0000", 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.16856490299999999 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731286+0000", 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9462000000000001e-05 2024-04-25T05:54:20.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731336+0000", 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0702999999999999e-05 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94258 11.0 11:0e232a3d:::benchmark_data_smithi043_179003_object94257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.733303009, 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.169532664, 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.419 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94258 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562391+0000", 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3e-07 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562445+0000", 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3516000000000001e-05 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562453+0000", 2024-04-25T05:54:20.422 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.3319999999999992e-06 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562461+0000", 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5270000000000001e-06 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.423 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562694+0000", 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000233327 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.562833+0000", 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138596 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.563001+0000", 2024-04-25T05:54:20.424 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016854399999999999 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.565736+0000", 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002735038 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.570245+0000", 2024-04-25T05:54:20.425 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045089120000000003 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731812+0000", 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.161567138 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731883+0000", 2024-04-25T05:54:20.426 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0382000000000005e-05 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.731923+0000", 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0722000000000001e-05 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94295 11.0 11:00ae873d:::benchmark_data_smithi043_179003_object94294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.70491418500001, 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14333959800000001, 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.428 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94295 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.429 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590780+0000", 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590781+0000", 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.265e-06 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.430 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590801+0000", 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.9935000000000001e-05 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590811+0000", 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0329999999999999e-05 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.431 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590817+0000", 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.964e-06 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.590891+0000", 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.3915e-05 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.432 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.591038+0000", 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014694799999999999 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.591208+0000", 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000170464 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.433 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.593800+0000", 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025914570000000001 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598539+0000", 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047392720000000001 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.434 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734007+0000", 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.135468218 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734077+0000", 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9395999999999998e-05 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.435 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.734119+0000", 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2434000000000001e-05 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:94309 11.0 11:02925c71:::benchmark_data_smithi043_179003_object94308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e484)", 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 183.69678268800001, 2024-04-25T05:54:20.436 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.14004730400000001, 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 94309 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598911+0000", 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598913+0000", 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3999999999999999e-06 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598970+0000", 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7605999999999997e-05 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598981+0000", 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1052e-05 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.598989+0000", 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599043+0000", 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3981999999999999e-05 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599210+0000", 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000166799 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.441 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.599359+0000", 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000149071 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.602209+0000", 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002850024 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.442 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.609505+0000", 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0072964589999999999 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738834+0000", 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.129328478 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.443 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738911+0000", 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7008999999999994e-05 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:51:16.738958+0000", 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7487999999999997e-05 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:211906 11.0 11:0c3d86ed:::benchmark_data_smithi043_179003_object81617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:14.502381+0000", 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 65.793313177000002, 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.129414325, 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.445 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 211906 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502381+0000", 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502381+0000", 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502382+0000", 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2139999999999999e-06 2024-04-25T05:54:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502387+0000", 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0889999999999999e-06 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502397+0000", 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0275000000000001e-05 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502405+0000", 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7970000000000001e-06 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502458+0000", 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3408000000000001e-05 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502575+0000", 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116839 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 3,7", 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.502688+0000", 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112984 2024-04-25T05:54:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.602681+0000", 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.099993125000000002 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.602748+0000", 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6750999999999998e-05 2024-04-25T05:54:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.631686+0000", 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.028937724000000001 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.631748+0000", 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2477000000000005e-05 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:14.631795+0000", 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6641999999999998e-05 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:260117 11.c 11:3e5771df:::benchmark_data_smithi043_179003_object129828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:20.453 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:54:01.424638+0000", 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 18.871055468000002, 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.221910526, 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 260117 2024-04-25T05:54:20.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424638+0000", 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424638+0000", 2024-04-25T05:54:20.455 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424639+0000", 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4399999999999998e-07 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424645+0000", 2024-04-25T05:54:20.456 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4439999999999999e-06 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424655+0000", 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0519999999999999e-05 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424664+0000", 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8920000000000006e-06 2024-04-25T05:54:20.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424730+0000", 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5547000000000001e-05 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424835+0000", 2024-04-25T05:54:20.458 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000105577 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 2,4", 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.424944+0000", 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109189 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.427573+0000", 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026283420000000001 2024-04-25T05:54:20.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.428281+0000", 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00070776400000000005 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:20.461 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.646448+0000", 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.218167429 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.646504+0000", 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6266e-05 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:20.462 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.646549+0000", 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4611999999999999e-05 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:20.463 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:20.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-25T05:54:21.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-25T05:54:21.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:21 smithi043 ceph-mon[27010]: osdmap e588: 8 total, 8 up, 8 in 2024-04-25T05:54:21.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:21 smithi043 ceph-mon[27010]: pgmap v1107: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:21.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:21 smithi043 ceph-mon[37492]: osdmap e588: 8 total, 8 up, 8 in 2024-04-25T05:54:21.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:21 smithi043 ceph-mon[37492]: pgmap v1107: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:21 smithi060 ceph-mon[30898]: osdmap e588: 8 total, 8 up, 8 in 2024-04-25T05:54:21.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:21 smithi060 ceph-mon[30898]: pgmap v1107: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:21.747 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a: {} 2024-04-25T05:54:21.754 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b: {} 2024-04-25T05:54:21.765 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c: {} 2024-04-25T05:54:21.767 INFO:teuthology.orchestra.run.smithi043.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:54:21.767 INFO:teuthology.orchestra.run.smithi043.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:54:21.767 INFO:teuthology.orchestra.run.smithi043.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T05:54:22.018 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:22.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:22 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/897416357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:54:22.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:22 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/897416357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:54:22.464 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:22.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-25T05:54:22.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:22 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/897416357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T05:54:23.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:23 smithi043 ceph-mon[27010]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:23.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:23 smithi043 ceph-mon[37492]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:23.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:23 smithi060 ceph-mon[30898]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:23.845 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:23.845 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:23.845 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:54:23.845 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:24.148 INFO:teuthology.orchestra.run.smithi043.stderr:noscrub is unset 2024-04-25T05:54:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:24 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3062413081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:24 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:24 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3062413081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:24 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:24 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3062413081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:24 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-25T05:54:24.729 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-25T05:54:24.996 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:54:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: osdmap e589: 8 total, 8 up, 8 in 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[27010]: osdmap e590: 8 total, 8 up, 8 in 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: osdmap e589: 8 total, 8 up, 8 in 2024-04-25T05:54:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:25 smithi043 ceph-mon[37492]: osdmap e590: 8 total, 8 up, 8 in 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: pgmap v1109: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: osdmap e589: 8 total, 8 up, 8 in 2024-04-25T05:54:25.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:25 smithi060 ceph-mon[30898]: osdmap e590: 8 total, 8 up, 8 in 2024-04-25T05:54:25.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-25T05:54:26.192 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:26.195 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:26.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 3.10 scrub starts 2024-04-25T05:54:26.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 3.10 scrub ok 2024-04-25T05:54:26.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:54:26.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:54:26.320 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 6.9 scrub starts 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[27010]: 6.9 scrub ok 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 3.10 scrub starts 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 3.10 scrub ok 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 6.9 scrub starts 2024-04-25T05:54:26.321 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:26 smithi043 ceph-mon[37492]: 6.9 scrub ok 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 3.10 scrub starts 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 3.10 scrub ok 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 6.9 scrub starts 2024-04-25T05:54:26.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:26 smithi060 ceph-mon[30898]: 6.9 scrub ok 2024-04-25T05:54:26.740 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:26.740 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:26.740 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:54:26.740 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:54:26.741 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:27.154 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.0 to 1 (10000) 2024-04-25T05:54:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:54:27.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: pgmap v1112: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.c scrub starts 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.c scrub ok 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 5.5 scrub starts 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 5.5 scrub ok 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2659501220' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: pgmap v1112: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-04-25T05:54:27.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.c scrub starts 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.c scrub ok 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 5.5 scrub starts 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 5.5 scrub ok 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2659501220' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:54:27.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:27 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:54:27.416 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: pgmap v1112: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.c scrub starts 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.c scrub ok 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 5.5 scrub starts 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 5.5 scrub ok 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2659501220' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:54:27.417 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:27 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:54:27.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-25T05:54:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 4.5 scrub starts 2024-04-25T05:54:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 4.5 scrub ok 2024-04-25T05:54:28.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: osdmap e591: 8 total, 8 up, 8 in 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 3.0 scrub starts 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[37492]: 3.0 scrub ok 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 4.5 scrub starts 2024-04-25T05:54:28.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 4.5 scrub ok 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: osdmap e591: 8 total, 8 up, 8 in 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 3.0 scrub starts 2024-04-25T05:54:28.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:28 smithi043 ceph-mon[27010]: 3.0 scrub ok 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 4.5 scrub starts 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 4.5 scrub ok 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: osdmap e591: 8 total, 8 up, 8 in 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 3.0 scrub starts 2024-04-25T05:54:28.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:28 smithi060 ceph-mon[30898]: 3.0 scrub ok 2024-04-25T05:54:28.965 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-25T05:54:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:nodeep-scrub is unset 2024-04-25T05:54:29.195 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:54:29.195 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:54:29.195 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:29.195 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 6.2 scrub starts 2024-04-25T05:54:29.195 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 6.2 scrub ok 2024-04-25T05:54:29.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/729922670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 5.c scrub starts 2024-04-25T05:54:29.196 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[27010]: 5.c scrub ok 2024-04-25T05:54:29.196 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 6.2 scrub starts 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 6.2 scrub ok 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/729922670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.197 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.198 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 5.c scrub starts 2024-04-25T05:54:29.198 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-mon[37492]: 5.c scrub ok 2024-04-25T05:54:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:54:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:54:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: pgmap v1114: 142 pgs: 142 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 6.2 scrub starts 2024-04-25T05:54:29.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 6.2 scrub ok 2024-04-25T05:54:29.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/729922670' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-25T05:54:29.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 5.c scrub starts 2024-04-25T05:54:29.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:29 smithi060 ceph-mon[30898]: 5.c scrub ok 2024-04-25T05:54:30.140 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:54:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:54:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:54:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: Cluster is now healthy 2024-04-25T05:54:30.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: osdmap e592: 8 total, 8 up, 8 in 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 6.16 scrub starts 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 6.16 scrub ok 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 5.19 scrub starts 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[27010]: 5.19 scrub ok 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: Cluster is now healthy 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: osdmap e592: 8 total, 8 up, 8 in 2024-04-25T05:54:30.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 6.16 scrub starts 2024-04-25T05:54:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 6.16 scrub ok 2024-04-25T05:54:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 5.19 scrub starts 2024-04-25T05:54:30.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:30 smithi043 ceph-mon[37492]: 5.19 scrub ok 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: Cluster is now healthy 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: osdmap e592: 8 total, 8 up, 8 in 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 6.16 scrub starts 2024-04-25T05:54:30.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 6.16 scrub ok 2024-04-25T05:54:30.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 5.19 scrub starts 2024-04-25T05:54:30.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:30 smithi060 ceph-mon[30898]: 5.19 scrub ok 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232177 10.18 10:1a382fa4:::benchmark_data_smithi043_152558_object103827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.18145898900002, 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95601884800000003, 2024-04-25T05:54:30.538 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232177 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355126+0000", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355127+0000", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4000000000000001e-07 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355130+0000", 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.811e-06 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355139+0000", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.687e-06 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355146+0000", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9190000000000002e-06 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355285+0000", 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013905100000000001 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.541 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.355296+0000", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0637e-05 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307193+0000", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.95189739500000004 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307333+0000", 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013948199999999999 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.542 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.307609+0000", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00027630100000000001 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.310940+0000", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003331493 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311007+0000", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6475000000000003e-05 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.543 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311024+0000", 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7076999999999999e-05 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311094+0000", 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9638e-05 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311145+0000", 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1242000000000002e-05 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232184 10.18 10:1f984730:::benchmark_data_smithi043_152558_object103834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.173267936, 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94951154000000004, 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232184 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363317+0000", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363318+0000", 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.181e-06 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363324+0000", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6590000000000004e-06 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363333+0000", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7019999999999996e-06 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363341+0000", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4930000000000001e-06 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363402+0000", 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1212999999999998e-05 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.363416+0000", 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4008e-05 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309166+0000", 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94575008000000005 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309307+0000", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00014065700000000001 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.309544+0000", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00023717000000000001 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312676+0000", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0031320359999999999 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312690+0000", 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3619e-05 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312725+0000", 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5490000000000001e-05 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312787+0000", 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2051999999999995e-05 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312828+0000", 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1180000000000002e-05 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232190 10.18 10:1c12a50d:::benchmark_data_smithi043_152558_object103840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.16672163800001, 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94767507500000003, 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232190 2024-04-25T05:54:30.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369863+0000", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369865+0000", 2024-04-25T05:54:30.553 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4619999999999999e-06 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369871+0000", 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3749999999999999e-06 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369882+0000", 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0515e-05 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369889+0000", 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.481e-06 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369947+0000", 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7445999999999999e-05 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.369956+0000", 2024-04-25T05:54:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.3000000000000007e-06 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311176+0000", 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.941220371 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311289+0000", 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112652 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.556 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.311488+0000", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00019917399999999999 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315758+0000", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0042704539999999999 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317049+0000", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012905200000000001 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.557 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317411+0000", 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036197200000000001 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317495+0000", 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4339999999999995e-05 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317538+0000", 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3013000000000003e-05 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232212 10.18 10:1d7e6992:::benchmark_data_smithi043_152558_object103862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.14373773599999, 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93768464399999996, 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232212 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392847+0000", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392848+0000", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.265e-06 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.560 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392862+0000", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4076999999999999e-05 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392874+0000", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1736000000000001e-05 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392883+0000", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5909999999999996e-06 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392949+0000", 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5753999999999994e-05 2024-04-25T05:54:30.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.392958+0000", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7180000000000008e-06 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.312940+0000", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.919981771 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.313071+0000", 2024-04-25T05:54:30.562 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013091099999999999 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.313274+0000", 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020262899999999999 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317004+0000", 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0037301209999999999 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.563 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317372+0000", 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00036868600000000001 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330370+0000", 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.012997140000000001 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330481+0000", 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000111634 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.564 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330532+0000", 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0611e-05 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232280 10.18 10:1f825152:::benchmark_data_smithi043_152558_object103930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.07499472199999, 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.87212884099999999, 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.565 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232280 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.566 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461590+0000", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461591+0000", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.24e-06 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461597+0000", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4500000000000003e-06 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.567 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461607+0000", 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9990000000000003e-06 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461615+0000", 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8429999999999993e-06 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461674+0000", 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9651999999999997e-05 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.568 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.461683+0000", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.9549999999999998e-06 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314427+0000", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.85274352399999997 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314544+0000", 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117234 2024-04-25T05:54:30.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.314753+0000", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00020859700000000001 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317062+0000", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002308994 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317596+0000", 2024-04-25T05:54:30.570 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00053388299999999997 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333567+0000", 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.015971346000000001 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333670+0000", 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000102645 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.571 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333719+0000", 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9478999999999998e-05 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232299 10.18 10:1cc40e50:::benchmark_data_smithi043_152558_object103949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.05942197000002, 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.856721332, 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.572 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232299 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477163+0000", 2024-04-25T05:54:30.573 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477164+0000", 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4190000000000001e-06 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477171+0000", 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1929999999999998e-06 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.574 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477184+0000", 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3501e-05 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477193+0000", 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7329999999999992e-06 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477262+0000", 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9710999999999997e-05 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.575 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.477269+0000", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2580000000000001e-06 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315793+0000", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.83852447299999999 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.315900+0000", 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001072 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.576 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.316075+0000", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017479799999999999 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320733+0000", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0046573889999999996 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323425+0000", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026927019999999999 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.577 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333779+0000", 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010353968 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333832+0000", 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2717999999999999e-05 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333884+0000", 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2268999999999997e-05 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232324 10.18 10:1aeaad8b:::benchmark_data_smithi043_152558_object103974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.03692180500002, 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.83436993100000001, 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232324 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499663+0000", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499665+0000", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5820000000000001e-06 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.580 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499670+0000", 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7880000000000003e-06 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499682+0000", 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1754e-05 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499697+0000", 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4946999999999999e-05 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499757+0000", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9710999999999998e-05 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.499766+0000", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8470000000000002e-06 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.317631+0000", 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.81786536399999998 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.582 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.318088+0000", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000456828 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.318348+0000", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000260294 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320775+0000", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024265430000000002 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.583 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324784+0000", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040088809999999997 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333952+0000", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0091684090000000006 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333997+0000", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5451999999999998e-05 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.584 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334033+0000", 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5531000000000003e-05 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232342 10.18 10:1cb472a9:::benchmark_data_smithi043_152558_object103992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.022648473, 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.82022194299999995, 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.585 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232342 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513936+0000", 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513937+0000", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9400000000000004e-07 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513942+0000", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8919999999999999e-06 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513951+0000", 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6090000000000001e-06 2024-04-25T05:54:30.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.513958+0000", 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5399999999999998e-06 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.515008+0000", 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001050193 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.588 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.515016+0000", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0369999999999995e-06 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319428+0000", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.80441196400000003 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319549+0000", 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012084200000000001 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.589 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.319732+0000", 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000182597 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324789+0000", 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050572589999999997 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.590 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324835+0000", 2024-04-25T05:54:30.591 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5748000000000003e-05 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334075+0000", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0092403490000000001 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334122+0000", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6983000000000001e-05 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334158+0000", 2024-04-25T05:54:30.592 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6136000000000001e-05 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232354 10.18 10:1d2e927d:::benchmark_data_smithi043_152558_object104004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.00769995600001, 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.80545024600000004, 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.593 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232354 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528885+0000", 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.594 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528887+0000", 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.792e-06 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528892+0000", 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.271e-06 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528906+0000", 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3696999999999999e-05 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.595 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528912+0000", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9519999999999999e-06 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528967+0000", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5707e-05 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528977+0000", 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.009e-05 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320806+0000", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.79182896999999997 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.320924+0000", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011754100000000001 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.321095+0000", 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000171349 2024-04-25T05:54:30.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324794+0000", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0036986520000000002 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327302+0000", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025076109999999999 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334244+0000", 2024-04-25T05:54:30.598 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0069425119999999996 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334295+0000", 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0688e-05 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334335+0000", 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0414000000000003e-05 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232355 10.18 10:1899febd:::benchmark_data_smithi043_152558_object104005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 289.00765611899999, 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.805527047, 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232355 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.600 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528929+0000", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.06e-07 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.601 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528932+0000", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9289999999999998e-06 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528939+0000", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4899999999999997e-06 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.528945+0000", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7349999999999999e-06 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.529003+0000", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7460999999999998e-05 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.529010+0000", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6320000000000002e-06 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322040+0000", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.79303036000000005 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322147+0000", 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000106777 2024-04-25T05:54:30.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.322304+0000", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000157009 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327251+0000", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049475370000000001 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327352+0000", 2024-04-25T05:54:30.604 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010056999999999999 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334372+0000", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0070206349999999999 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334420+0000", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7917999999999998e-05 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334456+0000", 2024-04-25T05:54:30.605 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5488000000000003e-05 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232443 10.18 10:1a37d6af:::benchmark_data_smithi043_152558_object104093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.91441965899998, 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.712427268, 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.606 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232443 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622165+0000", 2024-04-25T05:54:30.607 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622166+0000", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1209999999999999e-06 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622171+0000", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1220000000000003e-06 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622190+0000", 2024-04-25T05:54:30.608 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8165e-05 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622197+0000", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7640000000000005e-06 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622280+0000", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2244999999999996e-05 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.622289+0000", 2024-04-25T05:54:30.609 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0820000000000005e-06 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323472+0000", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.70118327999999996 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323581+0000", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000109454 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.323751+0000", 2024-04-25T05:54:30.610 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000169116 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327257+0000", 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003506412 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330319+0000", 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030615999999999998 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.611 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334494+0000", 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0041753980000000003 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334539+0000", 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4817999999999997e-05 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334593+0000", 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3690999999999998e-05 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232545 10.18 10:1949a72c:::benchmark_data_smithi043_152558_object104195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.77208921599998, 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.57027235399999998, 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232545 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764496+0000", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764497+0000", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.176e-06 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.614 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764502+0000", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8420000000000004e-06 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764512+0000", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0516999999999999e-05 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764518+0000", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8989999999999998e-06 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.615 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764575+0000", 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6471999999999997e-05 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.764586+0000", 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1054e-05 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324873+0000", 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.56028686800000005 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.616 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.324974+0000", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000101086 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.325137+0000", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00016339300000000001 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327314+0000", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0021773650000000001 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.617 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330579+0000", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003264922 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334633+0000", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040539970000000002 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334729+0000", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5847000000000006e-05 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.618 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334768+0000", 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8915999999999998e-05 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232637 10.18 10:1e18b6bd:::benchmark_data_smithi043_152558_object104287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.6694584, 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.46779147700000001, 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.619 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232637 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867126+0000", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867128+0000", 2024-04-25T05:54:30.620 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.46e-06 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867134+0000", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0689999999999998e-06 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867149+0000", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4642e-05 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867158+0000", 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1460000000000002e-06 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867240+0000", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1972999999999996e-05 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.867249+0000", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.5070000000000006e-06 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.325969+0000", 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.45871979299999999 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.326081+0000", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112388 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.326255+0000", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017321900000000001 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330277+0000", 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0040226740000000004 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.623 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333354+0000", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030765520000000002 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334830+0000", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0014759689999999999 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334881+0000", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1400999999999998e-05 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.624 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334918+0000", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6684000000000003e-05 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232658 10.18 10:1ec2bb35:::benchmark_data_smithi043_152558_object104308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.64566634699997, 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.44412062099999999, 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.625 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232658 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890919+0000", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890920+0000", 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.294e-06 2024-04-25T05:54:30.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890926+0000", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3069999999999999e-06 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890939+0000", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2842e-05 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.890948+0000", 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8300000000000002e-06 2024-04-25T05:54:30.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.891010+0000", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2336999999999996e-05 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.891022+0000", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1616e-05 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327388+0000", 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.43636574500000003 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327501+0000", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011370900000000001 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.327676+0000", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017499599999999999 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.330331+0000", 2024-04-25T05:54:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026544569999999998 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333412+0000", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0030809420000000001 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.334958+0000", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001546635 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335003+0000", 2024-04-25T05:54:30.630 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4320000000000003e-05 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335039+0000", 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6591000000000003e-05 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232694 10.18 10:19db009e:::benchmark_data_smithi043_152558_object104344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.58291823299999, 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.38153700099999999, 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232694 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953667+0000", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953668+0000", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2610000000000001e-06 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953673+0000", 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6829999999999999e-06 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.633 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953683+0000", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0689e-05 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953692+0000", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7800000000000006e-06 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953744+0000", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1499000000000003e-05 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.634 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:41.953751+0000", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6709999999999999e-06 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.328723+0000", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.37497164700000002 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.328841+0000", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117623 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.329030+0000", 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000189676 2024-04-25T05:54:30.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333431+0000", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0044009139999999997 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.333492+0000", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1301000000000001e-05 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335119+0000", 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001626901 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335167+0000", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7777e-05 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335204+0000", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6579000000000003e-05 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.15975.0:232721 10.18 10:1e695aa9:::benchmark_data_smithi043_152558_object104371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 288.50524275200002, 2024-04-25T05:54:30.637 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.31091695600000002, 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.15975", 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 232721 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:54:30.638 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031342+0000", 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031344+0000", 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.609e-06 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031350+0000", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3180000000000003e-06 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031366+0000", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6240999999999999e-05 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031376+0000", 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.4760000000000004e-06 2024-04-25T05:54:30.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031440+0000", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3822e-05 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.031447+0000", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6490000000000008e-06 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331065+0000", 2024-04-25T05:54:30.641 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.29961802999999998 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331187+0000", 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012126299999999999 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.331525+0000", 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00033875700000000002 2024-04-25T05:54:30.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.335248+0000", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.003722235 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.337203+0000", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0019557929999999999 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342156+0000", 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0049530540000000001 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342213+0000", 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6521000000000003e-05 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:49:42.342259+0000", 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6187999999999999e-05 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.644 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:260085 11.4 e578/456 11:282765cf:::benchmark_data_smithi043_179003_object129796:head v 578'16254, mlcod=578'16250)", 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:54:01.370482+0000", 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.166102801000001, 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27024745500000003, 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370482+0000", 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370482+0000", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370483+0000", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0900000000000001e-07 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370489+0000", 2024-04-25T05:54:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9100000000000002e-06 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370497+0000", 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9049999999999997e-06 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.370503+0000", 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0240000000000003e-06 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.647 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.374397+0000", 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0038944230000000002 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.631944+0000", 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25754709599999998 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640690+0000", 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0087462519999999995 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.648 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640730+0000", 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9236000000000002e-05 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:260101 11.2 11:494d2f86:::benchmark_data_smithi043_179003_object129812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:54:01.371770+0000", 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.164814696000001, 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.268744645, 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.649 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 260101 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371770+0000", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371770+0000", 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371771+0000", 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0500000000000002e-07 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371776+0000", 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0679999999999996e-06 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.651 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371787+0000", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0485e-05 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.371794+0000", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8169999999999999e-06 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.632132+0000", 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26033721900000001 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.632311+0000", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00017954900000000001 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.632450+0000", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013845400000000001 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640371+0000", 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0079215429999999996 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640411+0000", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9977e-05 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640424+0000", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3145e-05 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640473+0000", 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8164999999999997e-05 2024-04-25T05:54:30.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.640515+0000", 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2317999999999997e-05 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:260089 11.4 e578/456 11:28ca0e53:::benchmark_data_smithi043_179003_object129800:head v 578'16255, mlcod=578'16250)", 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:54:01.372959+0000", 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.163625855999999, 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.27058087800000002, 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372959+0000", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372959+0000", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372960+0000", 2024-04-25T05:54:30.656 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4000000000000001e-07 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372965+0000", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9389999999999996e-06 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372973+0000", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.8789999999999992e-06 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.372980+0000", 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7909999999999999e-06 2024-04-25T05:54:30.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.634411+0000", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26143107199999999 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.634456+0000", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4583999999999999e-05 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.643471+0000", 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0090145639999999992 2024-04-25T05:54:30.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.643540+0000", 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9408999999999999e-05 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_repop(client.16413.0:260098 11.f e578/456 11:f76527f5:::benchmark_data_smithi043_179003_object129809:head v 578'16494, mlcod=578'16489)", 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:54:01.373939+0000", 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 29.162645875999999, 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.26878551499999997, 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:30.659 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.373939+0000", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.373939+0000", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.373940+0000", 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7800000000000001e-07 2024-04-25T05:54:30.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.373945+0000", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.288e-06 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.373996+0000", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0815000000000001e-05 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.374003+0000", 2024-04-25T05:54:30.661 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.2169999999999997e-06 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.374393+0000", 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000390565 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.631988+0000", 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.25759478600000002 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.662 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.642693+0000", 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.010705007000000001 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:54:01.642725+0000", 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1259e-05 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:30.663 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:30.664 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 5.1f scrub starts 2024-04-25T05:54:31.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 5.1f scrub ok 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 4.12 scrub starts 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[27010]: 4.12 scrub ok 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 5.1f scrub starts 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 5.1f scrub ok 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:54:31.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 4.12 scrub starts 2024-04-25T05:54:31.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:31 smithi043 ceph-mon[37492]: 4.12 scrub ok 2024-04-25T05:54:31.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 5.1f scrub starts 2024-04-25T05:54:31.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 5.1f scrub ok 2024-04-25T05:54:31.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: pgmap v1116: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-25T05:54:31.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:54:31.501 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:54:31.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 4.12 scrub starts 2024-04-25T05:54:31.502 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:31 smithi060 ceph-mon[30898]: 4.12 scrub ok 2024-04-25T05:54:31.773 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-25T05:54:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:32 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:54:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:32 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:54:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:32 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:54:32.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:32 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:54:32.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:54:32.561 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:32 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:54:33.111 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:33.111 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:33.111 INFO:teuthology.orchestra.run.smithi043.stdout: "num_ops": 0 2024-04-25T05:54:33.111 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:33.172 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-25T05:54:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: 5.11 scrub starts 2024-04-25T05:54:33.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: 5.11 scrub ok 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: 4.1 scrub starts 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: 4.1 scrub ok 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: 5.11 scrub starts 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: 5.11 scrub ok 2024-04-25T05:54:33.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: 4.1 scrub starts 2024-04-25T05:54:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: 4.1 scrub ok 2024-04-25T05:54:33.393 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: 5.11 scrub starts 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: 5.11 scrub ok 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: pgmap v1117: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: 4.1 scrub starts 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: 4.1 scrub ok 2024-04-25T05:54:33.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-25T05:54:34.173 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:34.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: 5.15 scrub starts 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: 5.15 scrub ok 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: 6.6 deep-scrub starts 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: 6.6 deep-scrub ok 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: osdmap e593: 8 total, 8 up, 8 in 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:34.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: 5.15 scrub starts 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: 5.15 scrub ok 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: 6.6 deep-scrub starts 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: 6.6 deep-scrub ok 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: osdmap e593: 8 total, 8 up, 8 in 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: 5.15 scrub starts 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: 5.15 scrub ok 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: 6.6 deep-scrub starts 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: 6.6 deep-scrub ok 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1510658145' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: osdmap e593: 8 total, 8 up, 8 in 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:34.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-25T05:54:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 6.1c scrub starts 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 6.1c scrub ok 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 6.1b scrub starts 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 6.1b scrub ok 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 3.1b scrub starts 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 3.1b scrub ok 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 5.1d scrub starts 2024-04-25T05:54:35.348 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: 5.1d scrub ok 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[27010]: osdmap e594: 8 total, 8 up, 8 in 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 6.1c scrub starts 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 6.1c scrub ok 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 6.1b scrub starts 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 6.1b scrub ok 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 3.1b scrub starts 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 3.1b scrub ok 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 5.1d scrub starts 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: 5.1d scrub ok 2024-04-25T05:54:35.349 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:35 smithi043 ceph-mon[37492]: osdmap e594: 8 total, 8 up, 8 in 2024-04-25T05:54:35.491 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 6.1c scrub starts 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 6.1c scrub ok 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 6.1b scrub starts 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 6.1b scrub ok 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 3.1b scrub starts 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 3.1b scrub ok 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 5.1d scrub starts 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: 5.1d scrub ok 2024-04-25T05:54:35.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:35 smithi060 ceph-mon[30898]: osdmap e594: 8 total, 8 up, 8 in 2024-04-25T05:54:36.044 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:36.044 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [], 2024-04-25T05:54:36.044 INFO:teuthology.orchestra.run.smithi043.stdout: "complaint_time": 30, 2024-04-25T05:54:36.045 INFO:teuthology.orchestra.run.smithi043.stdout: "num_blocked_ops": 0 2024-04-25T05:54:36.045 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:36.181 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T05:54:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:36 smithi060 ceph-mon[30898]: 3.5 scrub starts 2024-04-25T05:54:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:36 smithi060 ceph-mon[30898]: 3.5 scrub ok 2024-04-25T05:54:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:36 smithi060 ceph-mon[30898]: 6.4 scrub starts 2024-04-25T05:54:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:36 smithi060 ceph-mon[30898]: 6.4 scrub ok 2024-04-25T05:54:36.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:36 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:36.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[27010]: 3.5 scrub starts 2024-04-25T05:54:36.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[27010]: 3.5 scrub ok 2024-04-25T05:54:36.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[27010]: 6.4 scrub starts 2024-04-25T05:54:36.607 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[27010]: 6.4 scrub ok 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[37492]: 3.5 scrub starts 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[37492]: 3.5 scrub ok 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[37492]: 6.4 scrub starts 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[37492]: 6.4 scrub ok 2024-04-25T05:54:36.608 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:36 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:37.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-25T05:54:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: 3.16 scrub starts 2024-04-25T05:54:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: 3.16 scrub ok 2024-04-25T05:54:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-04-25T05:54:37.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: osdmap e595: 8 total, 8 up, 8 in 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: 4.2 scrub starts 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: 4.2 scrub ok 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: 3.16 scrub starts 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: 3.16 scrub ok 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: osdmap e595: 8 total, 8 up, 8 in 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: 4.2 scrub starts 2024-04-25T05:54:37.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: 4.2 scrub ok 2024-04-25T05:54:37.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:54:37.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:37 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: 3.16 scrub starts 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: 3.16 scrub ok 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3449979092' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: osdmap e595: 8 total, 8 up, 8 in 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: 4.2 scrub starts 2024-04-25T05:54:37.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: 4.2 scrub ok 2024-04-25T05:54:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:54:37.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:37 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:54:37.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-25T05:54:37.702 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:38.309 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:38.436 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:38 smithi043 ceph-mon[27010]: osdmap e596: 8 total, 8 up, 8 in 2024-04-25T05:54:38.436 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:38 smithi043 ceph-mon[37492]: osdmap e596: 8 total, 8 up, 8 in 2024-04-25T05:54:38.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:38 smithi060 ceph-mon[30898]: osdmap e596: 8 total, 8 up, 8 in 2024-04-25T05:54:38.857 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-25T05:54:38.857 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 20, 2024-04-25T05:54:38.857 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 600, 2024-04-25T05:54:38.857 INFO:teuthology.orchestra.run.smithi043.stdout: "ops": [ 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.25487.0:234264 9.19 9:9fa2e610:::benchmark_data_smithi043_129820_object112950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:46:13.420539+0000", 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 505.43495193699999, 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.037135972, 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.25487", 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 234264 2024-04-25T05:54:38.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420539+0000", 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420539+0000", 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.859 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420539+0000", 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2699999999999999e-07 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420543+0000", 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.653e-06 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420552+0000", 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1770000000000004e-06 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.860 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420560+0000", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.5229999999999998e-06 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420607+0000", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7577000000000002e-05 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:13.420615+0000", 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8539999999999997e-06 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.861 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.451121+0000", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0305059350000001 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.451199+0000", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7892000000000006e-05 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 4,6,7", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.451298+0000", 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.9124999999999997e-05 2024-04-25T05:54:38.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.451844+0000", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00054553200000000005 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.452866+0000", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.001022352 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.452914+0000", 2024-04-25T05:54:38.863 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8288000000000003e-05 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.457473+0000", 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045583400000000001 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.457535+0000", 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2281999999999998e-05 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.864 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:46:14.457675+0000", 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000139815 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212486 11.b 11:d8017b5c:::benchmark_data_smithi043_179003_object82197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.124679+0000", 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.730811669999994, 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0167925250000001, 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.865 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212486 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124679+0000", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124679+0000", 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124680+0000", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.331e-06 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124685+0000", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1900000000000003e-06 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124701+0000", 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5194999999999999e-05 2024-04-25T05:54:38.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124709+0000", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1559999999999995e-06 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124763+0000", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4713999999999998e-05 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.124772+0000", 2024-04-25T05:54:38.868 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.5140000000000001e-06 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.133383+0000", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.008611181 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.133488+0000", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001048 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.133579+0000", 2024-04-25T05:54:38.869 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1206999999999996e-05 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.136279+0000", 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026995690000000002 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141341+0000", 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0050620459999999997 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.870 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141409+0000", 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7713999999999996e-05 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141443+0000", 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4656999999999997e-05 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141471+0000", 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8251000000000001e-05 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212487 11.b 11:df25fdaa:::benchmark_data_smithi043_179003_object82198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.126208+0000", 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.729282900000001, 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0154058619999999, 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.872 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212487 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126208+0000", 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126208+0000", 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.873 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126209+0000", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0550000000000001e-06 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126213+0000", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1710000000000004e-06 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126222+0000", 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6980000000000004e-06 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.874 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126229+0000", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.9720000000000003e-06 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126276+0000", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.7404000000000002e-05 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.126284+0000", 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2099999999999993e-06 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.136311+0000", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0100267780000001 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.136466+0000", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015515200000000001 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.136584+0000", 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000117756 2024-04-25T05:54:38.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141360+0000", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047765919999999996 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141520+0000", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015949700000000001 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141552+0000", 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1727999999999998e-05 2024-04-25T05:54:38.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141580+0000", 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8479999999999998e-05 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141614+0000", 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3368999999999998e-05 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212498 11.b 11:d3bdb0d4:::benchmark_data_smithi043_179003_object82209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.134392+0000", 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.721098338999994, 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.007360751, 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212498 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.879 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134392+0000", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134392+0000", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134393+0000", 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9599999999999999e-07 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.880 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134396+0000", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9160000000000001e-06 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134401+0000", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7599999999999999e-06 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134407+0000", 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5969999999999999e-06 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134532+0000", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012513800000000001 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.134546+0000", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3835000000000001e-05 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.138652+0000", 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.004106011 2024-04-25T05:54:38.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.138782+0000", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000129927 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.138921+0000", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013957299999999999 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141529+0000", 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026078590000000001 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.884 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.884 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141668+0000", 2024-04-25T05:54:38.884 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000138546 2024-04-25T05:54:38.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141699+0000", 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1009000000000001e-05 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141726+0000", 2024-04-25T05:54:38.885 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.7240999999999999e-05 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141753+0000", 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6843e-05 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212505 11.b 11:d6bc364f:::benchmark_data_smithi043_179003_object82216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.139921+0000", 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.715569821000003, 2024-04-25T05:54:38.886 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.009422281, 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212505 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139921+0000", 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139921+0000", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139922+0000", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.353e-06 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139927+0000", 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2939999999999996e-06 2024-04-25T05:54:38.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139943+0000", 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5985999999999999e-05 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.139951+0000", 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7190000000000001e-06 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.889 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.140005+0000", 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4144000000000002e-05 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.140013+0000", 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.1480000000000006e-06 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141787+0000", 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.001773834 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.890 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141904+0000", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011715299999999999 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.142008+0000", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000103477 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149150+0000", 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0071424970000000003 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149219+0000", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8244999999999995e-05 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149275+0000", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6858999999999999e-05 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149310+0000", 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.4196000000000003e-05 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149343+0000", 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3376e-05 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212506 11.b 11:da7d03ec:::benchmark_data_smithi043_179003_object82217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.142473+0000", 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.713017993999998, 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.012645206, 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212506 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142473+0000", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.894 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142473+0000", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142474+0000", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5090000000000001e-06 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142479+0000", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2270000000000001e-06 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.895 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142490+0000", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0723999999999999e-05 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142498+0000", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.588e-06 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142554+0000", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6458000000000001e-05 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.896 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.142563+0000", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6819999999999992e-06 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.144588+0000", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0020255090000001 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.144724+0000", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00013583000000000001 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.897 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.144851+0000", 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012701300000000001 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149363+0000", 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0045116840000000002 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149408+0000", 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5317e-05 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.898 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155016+0000", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0056078810000000003 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155073+0000", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7151000000000002e-05 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155118+0000", 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4632999999999998e-05 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.899 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212527 11.b 11:d9954519:::benchmark_data_smithi043_179003_object82238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.159280+0000", 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.696210340999997, 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99599159100000001, 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212527 2024-04-25T05:54:38.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159280+0000", 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159280+0000", 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.901 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159282+0000", 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.5120000000000001e-06 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159288+0000", 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.806e-06 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159299+0000", 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1433e-05 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.902 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159307+0000", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.8290000000000008e-06 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159367+0000", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0266000000000002e-05 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.159375+0000", 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.4060000000000005e-06 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.147039+0000", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98766329200000003 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.147165+0000", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012598399999999999 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.147276+0000", 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011093499999999999 2024-04-25T05:54:38.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149420+0000", 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002144417 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.154937+0000", 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005516534 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.905 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155187+0000", 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00025038300000000002 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155228+0000", 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1245000000000002e-05 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155272+0000", 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3548999999999999e-05 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212539 11.b 11:da793cbc:::benchmark_data_smithi043_179003_object82250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.168427+0000", 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.687064000999996, 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98706317600000004, 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212539 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.907 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168427+0000", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168427+0000", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168427+0000", 2024-04-25T05:54:38.908 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5400000000000002e-07 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168430+0000", 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.6120000000000001e-06 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168436+0000", 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0549999999999996e-06 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.909 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168440+0000", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.3850000000000004e-06 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168517+0000", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6422999999999997e-05 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.168553+0000", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.6795999999999998e-05 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.910 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149451+0000", 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98089782400000003 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149588+0000", 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000137238 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.149711+0000", 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012284399999999999 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.911 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155335+0000", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.005624169 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155349+0000", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.3625000000000001e-05 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155415+0000", 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5920999999999994e-05 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.912 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155449+0000", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.3680000000000003e-05 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155490+0000", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1149999999999997e-05 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212548 11.b 11:da7ebe16:::benchmark_data_smithi043_179003_object82259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.226125+0000", 2024-04-25T05:54:38.913 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.629366043000005, 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92956858600000003, 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212548 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.914 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226125+0000", 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226125+0000", 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226125+0000", 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.03e-07 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.915 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226127+0000", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5210000000000001e-06 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226133+0000", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1819999999999997e-06 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226139+0000", 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9599999999999997e-06 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226197+0000", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8187999999999999e-05 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.226206+0000", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.0219999999999993e-06 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.152182+0000", 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92597605699999996 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.152292+0000", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010966100000000001 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.152407+0000", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011573099999999999 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155355+0000", 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002947912 2024-04-25T05:54:38.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155541+0000", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00018620900000000001 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155609+0000", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.7282999999999999e-05 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155648+0000", 2024-04-25T05:54:38.919 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.8860999999999999e-05 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155693+0000", 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.5596000000000002e-05 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212562 11.b 11:dc267ec5:::benchmark_data_smithi043_179003_object82273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.240742+0000", 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.614748903999995, 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.92467090500000004, 2024-04-25T05:54:38.920 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212562 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240742+0000", 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.921 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240742+0000", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240743+0000", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1510000000000001e-06 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240747+0000", 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.4490000000000001e-06 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240757+0000", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.2259999999999994e-06 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240764+0000", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1010000000000003e-06 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240816+0000", 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.2298e-05 2024-04-25T05:54:38.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.240826+0000", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7240000000000004e-06 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155740+0000", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91491477899999996 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155850+0000", 2024-04-25T05:54:38.924 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010919100000000001 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155978+0000", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000128474 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165246+0000", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0092675359999999998 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165305+0000", 2024-04-25T05:54:38.925 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.9624999999999999e-05 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165348+0000", 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2794999999999999e-05 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165381+0000", 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.2857000000000001e-05 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.926 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165413+0000", 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.1699000000000003e-05 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212586 11.b 11:d5d0b5dd:::benchmark_data_smithi043_179003_object82297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.261068+0000", 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.594422863000005, 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.90451488899999999, 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.927 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212586 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261068+0000", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261068+0000", 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261070+0000", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.8670000000000001e-06 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261077+0000", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.1999999999999997e-06 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261094+0000", 2024-04-25T05:54:38.929 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.7567999999999998e-05 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261105+0000", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0699999999999999e-05 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261171+0000", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5473000000000001e-05 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.261180+0000", 2024-04-25T05:54:38.930 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8859999999999999e-06 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.158720+0000", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.89753934000000002 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.159385+0000", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00066509499999999999 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.159514+0000", 2024-04-25T05:54:38.931 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012933399999999999 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165454+0000", 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0059394340000000004 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165464+0000", 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0397999999999999e-05 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.932 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165506+0000", 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.1766e-05 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165544+0000", 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.7811000000000001e-05 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.165583+0000", 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9017000000000003e-05 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212605 11.b 11:dc779934:::benchmark_data_smithi043_179003_object82316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.278727+0000", 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.576763634000002, 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.89533603299999998, 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212605 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.934 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278727+0000", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278727+0000", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278728+0000", 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.6199999999999997e-07 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278732+0000", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278742+0000", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.048e-05 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278748+0000", 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8320000000000002e-06 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278799+0000", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0445999999999998e-05 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.278805+0000", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8479999999999997e-06 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.162621+0000", 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.883816029 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.937 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.162731+0000", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000110783 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.162845+0000", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011370300000000001 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.171176+0000", 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0083307099999999999 2024-04-25T05:54:38.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173977+0000", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.002800931 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.174006+0000", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.9420999999999998e-05 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.174037+0000", 2024-04-25T05:54:38.939 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.0859999999999999e-05 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.174063+0000", 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5987e-05 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212616 11.b 11:d5520cbd:::benchmark_data_smithi043_179003_object82327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.288940+0000", 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.566551012000005, 2024-04-25T05:54:38.940 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.94834570600000001, 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212616 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288940+0000", 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288940+0000", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288941+0000", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2470000000000001e-06 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288946+0000", 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3369999999999999e-06 2024-04-25T05:54:38.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288961+0000", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4361e-05 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.288968+0000", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4359999999999996e-06 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.289019+0000", 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.0865000000000002e-05 2024-04-25T05:54:38.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.289027+0000", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.6570000000000003e-06 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.171219+0000", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.88219108899999998 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.171331+0000", 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000112526 2024-04-25T05:54:38.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.171448+0000", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000116431 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234514+0000", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.063066575999999999 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237102+0000", 2024-04-25T05:54:38.945 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0025879700000000002 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237173+0000", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.0483999999999998e-05 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237230+0000", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7565000000000002e-05 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237285+0000", 2024-04-25T05:54:38.946 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5161999999999998e-05 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212646 11.b 11:d7e93894:::benchmark_data_smithi043_179003_object82357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.329649+0000", 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.525841940000007, 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91048335700000005, 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212646 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329649+0000", 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329649+0000", 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329650+0000", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329655+0000", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4879999999999998e-06 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329668+0000", 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.2255999999999999e-05 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.949 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329678+0000", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.8949999999999993e-06 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329740+0000", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2014000000000006e-05 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.329746+0000", 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2260000000000002e-06 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.951 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.951 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.174237+0000", 2024-04-25T05:54:38.951 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.84449151300000003 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.174378+0000", 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0001405 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.231959+0000", 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.05758133 2024-04-25T05:54:38.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237343+0000", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0053837340000000003 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.237354+0000", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1545e-05 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.240024+0000", 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0026695 2024-04-25T05:54:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.240078+0000", 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4032e-05 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.240132+0000", 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.4048999999999997e-05 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212664 11.b 11:d1aa086f:::benchmark_data_smithi043_179003_object82375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:15.361760+0000", 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 83.493730623000005, 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91149090600000005, 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212664 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361760+0000", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361760+0000", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361761+0000", 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.78e-07 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361766+0000", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.6299999999999997e-06 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361776+0000", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.044e-05 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361782+0000", 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.4289999999999998e-06 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361831+0000", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8637999999999997e-05 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:15.361837+0000", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1340000000000001e-06 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234555+0000", 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.87271765800000001 2024-04-25T05:54:38.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234681+0000", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00012613899999999999 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,1", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234793+0000", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011183000000000001 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.239981+0000", 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0051882259999999998 2024-04-25T05:54:38.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.271885+0000", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.031903617000000002 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.273146+0000", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0012610360000000001 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.273199+0000", 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3222e-05 2024-04-25T05:54:38.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.273251+0000", 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.1928999999999997e-05 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212688 11.9 11:95d7a9a9:::benchmark_data_smithi043_179003_object82399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:16.141920+0000", 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.713570087999997, 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.002225659, 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212688 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141920+0000", 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.962 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141920+0000", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141922+0000", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1370000000000001e-06 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141927+0000", 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.9570000000000001e-06 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.963 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141936+0000", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.1530000000000003e-06 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141942+0000", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.8409999999999996e-06 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.141995+0000", 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.3356999999999998e-05 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.964 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.142001+0000", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.0929999999999999e-06 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.139046+0000", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99704504400000005 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.139150+0000", 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.000103504 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.965 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.139247+0000", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 9.7027999999999993e-05 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144043+0000", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0047965560000000004 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144053+0000", 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0322999999999999e-05 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144077+0000", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.3951000000000002e-05 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144117+0000", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.9978999999999998e-05 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144146+0000", 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8736000000000001e-05 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.967 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212693 11.9 11:920a87f2:::benchmark_data_smithi043_179003_object82404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:16.153743+0000", 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.701747376, 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99682211799999998, 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212693 2024-04-25T05:54:38.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153743+0000", 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153743+0000", 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.969 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153745+0000", 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.6589999999999999e-06 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153751+0000", 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1040000000000003e-06 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153763+0000", 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.165e-05 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.970 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153771+0000", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2039999999999997e-06 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153838+0000", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.6841000000000006e-05 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.153844+0000", 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5610000000000004e-06 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.971 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.142414+0000", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98856934100000005 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.142502+0000", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7960000000000005e-05 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.142581+0000", 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9927999999999998e-05 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150315+0000", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0077332290000000003 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150377+0000", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2695999999999997e-05 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150446+0000", 2024-04-25T05:54:38.973 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.8403000000000002e-05 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150508+0000", 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.2688999999999996e-05 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150565+0000", 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.6852999999999999e-05 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212698 11.9 11:9cdd75ce:::benchmark_data_smithi043_179003_object82409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:16.155866+0000", 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.699624806000003, 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.99488332300000004, 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212698 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155866+0000", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155866+0000", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155867+0000", 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.071e-06 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.976 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155873+0000", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.1970000000000001e-06 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155884+0000", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.118e-05 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155893+0000", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7290000000000006e-06 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155957+0000", 2024-04-25T05:54:38.977 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3616999999999998e-05 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.155967+0000", 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0701e-05 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144179+0000", 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.98821179100000001 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.978 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144253+0000", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.4010999999999993e-05 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.144331+0000", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.7793999999999994e-05 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150580+0000", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0062489709999999999 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150637+0000", 2024-04-25T05:54:38.979 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7340000000000003e-05 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150686+0000", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8585000000000003e-05 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150721+0000", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 3.5240999999999997e-05 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150749+0000", 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.8095000000000001e-05 2024-04-25T05:54:38.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212710 11.9 11:9156e9d9:::benchmark_data_smithi043_179003_object82421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:16.173133+0000", 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.682357496999998, 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97779670900000004, 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212710 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.981 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173133+0000", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173133+0000", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173135+0000", 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.466e-06 2024-04-25T05:54:38.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173140+0000", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.5330000000000002e-06 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173152+0000", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.1965e-05 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173160+0000", 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.2910000000000007e-06 2024-04-25T05:54:38.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173209+0000", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8223000000000002e-05 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.173215+0000", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.3509999999999998e-06 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.147489+0000", 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.97427418200000004 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.147733+0000", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00024390499999999999 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.147850+0000", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00011715200000000001 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150792+0000", 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0029415299999999999 2024-04-25T05:54:38.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150807+0000", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.4904e-05 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150847+0000", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.0133999999999999e-05 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150904+0000", 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 5.7640999999999999e-05 2024-04-25T05:54:38.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150930+0000", 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 2.5432e-05 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "description": "osd_op(client.16413.0:212716 11.9 11:9b2f0a13:::benchmark_data_smithi043_179003_object82427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "initiated_at": "2024-04-25T05:53:16.232138+0000", 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "age": 82.623352749999995, 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.93623121899999995, 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "continuous": false, 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type_data": { 2024-04-25T05:54:38.987 INFO:teuthology.orchestra.run.smithi043.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client_info": { 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client": "client.16413", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "tid": 212716 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "events": [ 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "initiated", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232138+0000", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "header_read", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232138+0000", 2024-04-25T05:54:38.988 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "throttled", 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232139+0000", 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.579e-06 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "all_read", 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232146+0000", 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.207e-06 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "dispatched", 2024-04-25T05:54:38.989 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232156+0000", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 1.0711e-05 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "queued_for_pg", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.232164+0000", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.0919999999999998e-06 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234623+0000", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0024590049999999998 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for peered", 2024-04-25T05:54:38.990 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:16.234632+0000", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 8.7460000000000006e-06 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "reached_pg", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.150967+0000", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.91633480099999998 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "started", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.151047+0000", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 7.9732000000000002e-05 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "waiting for subops from 0,6", 2024-04-25T05:54:38.991 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.151148+0000", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00010177499999999999 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.158373+0000", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0072248030000000001 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.168100+0000", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.0097269109999999995 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "op_commit", 2024-04-25T05:54:38.992 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.168255+0000", 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 0.00015521299999999999 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "commit_sent", 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.168304+0000", 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 4.8313000000000003e-05 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "event": "done", 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "time": "2024-04-25T05:53:17.168369+0000", 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: "duration": 6.5331000000000002e-05 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-25T05:54:38.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-25T05:54:38.994 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-25T05:54:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: 1.2 scrub starts 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: 1.2 scrub ok 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: 1.3 scrub starts 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: 1.3 scrub ok 2024-04-25T05:54:39.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: 1.2 scrub starts 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: 1.2 scrub ok 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: 1.3 scrub starts 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: 1.3 scrub ok 2024-04-25T05:54:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: 1.2 scrub starts 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: 1.2 scrub ok 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: 1.3 scrub starts 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: 1.3 scrub ok 2024-04-25T05:54:39.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-25T05:54:40.131 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:40.169 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: 1.0 scrub starts 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: 1.0 scrub ok 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: osdmap e597: 8 total, 8 up, 8 in 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: 6.17 deep-scrub starts 2024-04-25T05:54:40.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: 6.17 deep-scrub ok 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[27010]: osdmap e598: 8 total, 8 up, 8 in 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: 1.0 scrub starts 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: 1.0 scrub ok 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: osdmap e597: 8 total, 8 up, 8 in 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: 6.17 deep-scrub starts 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: 6.17 deep-scrub ok 2024-04-25T05:54:40.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:40 smithi043 ceph-mon[37492]: osdmap e598: 8 total, 8 up, 8 in 2024-04-25T05:54:40.442 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: 1.0 scrub starts 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: 1.0 scrub ok 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4274158228' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: osdmap e597: 8 total, 8 up, 8 in 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: 6.17 deep-scrub starts 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: 6.17 deep-scrub ok 2024-04-25T05:54:40.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:40 smithi060 ceph-mon[30898]: osdmap e598: 8 total, 8 up, 8 in 2024-04-25T05:54:41.052 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:54:41.052 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:41.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-25T05:54:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[37492]: 1.1 scrub starts 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[37492]: 1.1 scrub ok 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[37492]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[37492]: 4.18 scrub starts 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[37492]: 4.18 scrub ok 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[27010]: 1.1 scrub starts 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[27010]: 1.1 scrub ok 2024-04-25T05:54:41.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[27010]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[27010]: 4.18 scrub starts 2024-04-25T05:54:41.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:41 smithi043 ceph-mon[27010]: 4.18 scrub ok 2024-04-25T05:54:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:41 smithi060 ceph-mon[30898]: 1.1 scrub starts 2024-04-25T05:54:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:41 smithi060 ceph-mon[30898]: 1.1 scrub ok 2024-04-25T05:54:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:41 smithi060 ceph-mon[30898]: pgmap v1127: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:54:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:41 smithi060 ceph-mon[30898]: 4.18 scrub starts 2024-04-25T05:54:41.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:41 smithi060 ceph-mon[30898]: 4.18 scrub ok 2024-04-25T05:54:41.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-25T05:54:42.055 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:42.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 1.a scrub starts 2024-04-25T05:54:42.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 1.a scrub ok 2024-04-25T05:54:42.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 4.6 scrub starts 2024-04-25T05:54:42.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 4.6 scrub ok 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 1.5 scrub starts 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 1.5 scrub ok 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 6.1 scrub starts 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: 6.1 scrub ok 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 1.a scrub starts 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 1.a scrub ok 2024-04-25T05:54:42.367 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 4.6 scrub starts 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 4.6 scrub ok 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 1.5 scrub starts 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 1.5 scrub ok 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 6.1 scrub starts 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: 6.1 scrub ok 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:42.368 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:42 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:42.437 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 1.a scrub starts 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 1.a scrub ok 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 4.6 scrub starts 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 4.6 scrub ok 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 1.5 scrub starts 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 1.5 scrub ok 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 6.1 scrub starts 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: 6.1 scrub ok 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:54:42.438 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:54:42.439 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:42 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:42.567 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:42.567 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:42.567 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:54:42.567 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:54:42.567 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T05:54:43.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: 3.1d scrub starts 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: 3.1d scrub ok 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: 5.1 scrub starts 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: 5.1 scrub ok 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:43.350 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[37492]: osdmap e599: 8 total, 8 up, 8 in 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: 3.1d scrub starts 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: 3.1d scrub ok 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: 5.1 scrub starts 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: 5.1 scrub ok 2024-04-25T05:54:43.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:43.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:43 smithi043 ceph-mon[27010]: osdmap e599: 8 total, 8 up, 8 in 2024-04-25T05:54:43.440 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: 3.1d scrub starts 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: 3.1d scrub ok 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: 5.1 scrub starts 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: 5.1 scrub ok 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3238842199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:43.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-mon[30898]: osdmap e599: 8 total, 8 up, 8 in 2024-04-25T05:54:43.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-25T05:54:43.966 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195454 10.1 10:8ac0cd3c:::benchmark_data_smithi043_152558_object67104:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.41249051400001, 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044723351, 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.967 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195454 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552780+0000", 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552782+0000", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7039999999999999e-06 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552789+0000", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8340000000000001e-06 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552799+0000", 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0064999999999999e-05 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.552808+0000", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3190000000000001e-06 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558556+0000", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057473109999999997 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558565+0000", 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3989999999999993e-06 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591223+0000", 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0326579060000001 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.597504+0000", 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0062808129999999997 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195456 10.1 10:80db890b:::benchmark_data_smithi043_152558_object67106:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.971 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.41058918599998, 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0479446400000001, 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195456 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554682+0000", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554683+0000", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554687+0000", 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3379999999999998e-06 2024-04-25T05:54:43.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554696+0000", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0720000000000006e-06 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554703+0000", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3239999999999997e-06 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558586+0000", 2024-04-25T05:54:43.974 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038829329999999999 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558699+0000", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011292 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602367+0000", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.04366767 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602626+0000", 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00025938399999999998 2024-04-25T05:54:43.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195470 10.4 10:2594b5ae:::benchmark_data_smithi043_152558_object67120:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.41055741700001, 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0253313630000001, 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195470 2024-04-25T05:54:43.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554714+0000", 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7500000000000001e-07 2024-04-25T05:54:43.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554716+0000", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3029999999999998e-06 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554721+0000", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.181e-06 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554726+0000", 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9300000000000002e-06 2024-04-25T05:54:43.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555243+0000", 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051627300000000004 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555251+0000", 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7050000000000005e-06 2024-04-25T05:54:43.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.578884+0000", 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0236325479999999 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.980 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.580045+0000", 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001161048 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195477 10.e 10:7c0c42f7:::benchmark_data_smithi043_152558_object67127:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.41053305200001, 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0658362960000001, 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:43.981 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195477 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.982 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554738+0000", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3299999999999998e-07 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554740+0000", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2330000000000001e-06 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554746+0000", 2024-04-25T05:54:43.983 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.13e-06 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554751+0000", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3750000000000002e-06 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554809+0000", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8230000000000003e-05 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554820+0000", 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0581000000000001e-05 2024-04-25T05:54:43.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596186+0000", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0413658189999999 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.598902+0000", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027164310000000001 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.599221+0000", 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000319031 2024-04-25T05:54:43.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.611133+0000", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011912219999999999 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.614119+0000", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029860350000000002 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620485+0000", 2024-04-25T05:54:43.986 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063658589999999998 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620537+0000", 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1572e-05 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.620574+0000", 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7447000000000002e-05 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195487 10.4 10:295dbb8c:::benchmark_data_smithi043_152558_object67137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.41051188300003, 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0346735600000001, 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.990 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195487 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.991 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554759+0000", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8599999999999999e-07 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554761+0000", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0310000000000001e-06 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.992 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554766+0000", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9450000000000001e-06 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.554771+0000", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0819999999999998e-06 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555283+0000", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00051210299999999997 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.555291+0000", 2024-04-25T05:54:43.993 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1690000000000003e-06 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.581742+0000", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0264516699999999 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582176+0000", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00043403700000000002 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.582318+0000", 2024-04-25T05:54:43.994 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000142003 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589262+0000", 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0069436080000000004 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589327+0000", 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4770000000000005e-05 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.995 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589367+0000", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9824999999999999e-05 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589405+0000", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8486999999999999e-05 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589433+0000", 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.7543999999999999e-05 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:43.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195490 10.4 10:28cf0ec2:::benchmark_data_smithi043_152558_object67140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.40698275699998, 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0313008349999999, 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195490 2024-04-25T05:54:43.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558288+0000", 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558289+0000", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1960000000000001e-06 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558293+0000", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8619999999999996e-06 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558301+0000", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.396e-06 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558308+0000", 2024-04-25T05:54:43.999 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3629999999999996e-06 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558920+0000", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00061224599999999999 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558926+0000", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.891e-06 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584147+0000", 2024-04-25T05:54:44.000 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0252211170000001 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584221+0000", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4258999999999995e-05 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.584294+0000", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2475e-05 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589475+0000", 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0051810500000000004 2024-04-25T05:54:44.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589486+0000", 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1388e-05 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589536+0000", 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0037999999999998e-05 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.002 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589566+0000", 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9144000000000001e-05 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589589+0000", 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3410000000000001e-05 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195491 10.4 10:22b59bea:::benchmark_data_smithi043_152558_object67141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.40694882100001, 2024-04-25T05:54:44.003 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0361324119999999, 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195491 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558322+0000", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4999999999999998e-07 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558325+0000", 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.2699999999999999e-06 2024-04-25T05:54:44.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558330+0000", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0189999999999997e-06 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558335+0000", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0799999999999996e-06 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558939+0000", 2024-04-25T05:54:44.006 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00060434400000000004 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.558943+0000", 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.036e-06 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.585996+0000", 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0270530360000001 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.007 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586084+0000", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7683999999999996e-05 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.586156+0000", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1772000000000003e-05 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589499+0000", 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033435169999999998 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.008 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594275+0000", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047758619999999996 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594326+0000", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0866999999999999e-05 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594406+0000", 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9572000000000003e-05 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.009 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594455+0000", 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9002999999999998e-05 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195496 10.4 10:23a4e9e2:::benchmark_data_smithi043_152558_object67146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.40269443199998, 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.032050406, 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.010 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195496 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562576+0000", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562578+0000", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0470000000000001e-06 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562581+0000", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.839e-06 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.012 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562590+0000", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2190000000000004e-06 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562595+0000", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.468e-06 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562692+0000", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7392999999999994e-05 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.562699+0000", 2024-04-25T05:54:44.013 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7109999999999998e-06 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587618+0000", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.024919208 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587708+0000", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0075999999999996e-05 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.587795+0000", 2024-04-25T05:54:44.014 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6837999999999998e-05 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594287+0000", 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0064916610000000001 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594498+0000", 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021106299999999999 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.015 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594545+0000", 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.655e-05 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594590+0000", 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5574000000000001e-05 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594627+0000", 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6758999999999998e-05 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.016 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195505 10.1 10:82918380:::benchmark_data_smithi043_152558_object67155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.38945143799998, 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.053126166, 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195505 2024-04-25T05:54:44.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575819+0000", 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.018 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575821+0000", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.08e-06 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575827+0000", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6819999999999997e-06 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575835+0000", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2160000000000006e-06 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575841+0000", 2024-04-25T05:54:44.019 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2499999999999997e-06 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575897+0000", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6360000000000002e-05 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.575904+0000", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1749999999999999e-06 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602674+0000", 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.02677006 2024-04-25T05:54:44.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603270+0000", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00059573700000000005 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.603459+0000", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018862499999999999 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615485+0000", 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.012026722 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621556+0000", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060705749999999999 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628800+0000", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072444809999999997 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628892+0000", 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1471000000000006e-05 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628946+0000", 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3732e-05 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195517 10.4 10:24331628:::benchmark_data_smithi043_152558_object67167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.37845141999998, 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007987988, 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.023 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195517 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586819+0000", 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586821+0000", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.252e-06 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586825+0000", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5419999999999999e-06 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586836+0000", 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0988999999999999e-05 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586845+0000", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5699999999999993e-06 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586911+0000", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5667000000000007e-05 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.586923+0000", 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2104e-05 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589644+0000", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0027209029999999 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589733+0000", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9178000000000006e-05 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.589854+0000", 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120975 2024-04-25T05:54:44.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594292+0000", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044382069999999996 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594675+0000", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00038344300000000002 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594716+0000", 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1041999999999998e-05 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594767+0000", 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0926e-05 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594807+0000", 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.019e-05 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.029 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195519 10.4 10:29710d26:::benchmark_data_smithi043_152558_object67169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.37427721699999, 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.005228536, 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195519 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.030 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590994+0000", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.590995+0000", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6339999999999999e-06 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.031 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591000+0000", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.147e-06 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591014+0000", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3421000000000001e-05 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591022+0000", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1899999999999995e-06 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.032 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591079+0000", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6852999999999999e-05 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591090+0000", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1545999999999999e-05 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591626+0000", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.000535591 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.033 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591729+0000", 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102774 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.591850+0000", 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121255 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.594508+0000", 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026583100000000001 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596021+0000", 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0015122289999999999 2024-04-25T05:54:44.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596061+0000", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0339999999999997e-05 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596174+0000", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112823 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596222+0000", 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8423e-05 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.036 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195520 10.1 10:8be1fae5:::benchmark_data_smithi043_152558_object67170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.37423110200001, 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0383997030000001, 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195520 2024-04-25T05:54:44.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591040+0000", 2024-04-25T05:54:44.038 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1899999999999998e-07 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591045+0000", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7779999999999998e-06 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591051+0000", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.784e-06 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591056+0000", 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3909999999999997e-06 2024-04-25T05:54:44.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591110+0000", 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3823000000000003e-05 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.591117+0000", 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0520000000000004e-06 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.040 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612643+0000", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0215256699999999 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612745+0000", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102353 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.612856+0000", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011074900000000001 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.041 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615525+0000", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002668784 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.628786+0000", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013261088000000001 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629032+0000", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024633800000000003 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629115+0000", 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2599e-05 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629440+0000", 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00032487500000000002 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195526 10.1 10:8fa6926a:::benchmark_data_smithi043_152558_object67176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.37021214399999, 2024-04-25T05:54:44.043 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.037524509, 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195526 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.044 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595059+0000", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595060+0000", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.488e-06 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595066+0000", 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6559999999999997e-06 2024-04-25T05:54:44.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595080+0000", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3925e-05 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595089+0000", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6810000000000004e-06 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595171+0000", 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2888999999999999e-05 2024-04-25T05:54:44.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.595186+0000", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4477e-05 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615572+0000", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0203861359999999 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.615673+0000", 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101087 2024-04-25T05:54:44.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.616535+0000", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00086202200000000005 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621514+0000", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0049784570000000004 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629450+0000", 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0079360639999999996 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632427+0000", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0029770019999999999 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632527+0000", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100078 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632583+0000", 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6546999999999999e-05 2024-04-25T05:54:44.049 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195534 10.4 10:2bc4d148:::benchmark_data_smithi043_152558_object67184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.35982770099997, 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.004935758, 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195534 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605443+0000", 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.051 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605445+0000", 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.59e-06 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605451+0000", 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3500000000000002e-06 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605466+0000", 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4555e-05 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.052 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605474+0000", 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8639999999999996e-06 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605539+0000", 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5467000000000002e-05 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.053 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.605553+0000", 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4328000000000001e-05 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596256+0000", 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99070311899999997 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596366+0000", 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109079 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.054 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.596569+0000", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020323699999999999 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.600295+0000", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00372573 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.602218+0000", 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001922972 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.055 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609480+0000", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0072625119999999996 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.609538+0000", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7531999999999997e-05 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.610379+0000", 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000841423 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.056 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:195540 10.1 10:8978d76f:::benchmark_data_smithi043_152558_object67190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 335.34989921099998, 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.017365112, 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 195540 2024-04-25T05:54:44.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615372+0000", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615373+0000", 2024-04-25T05:54:44.058 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.311e-06 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615378+0000", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3249999999999998e-06 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615391+0000", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2245e-05 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615397+0000", 2024-04-25T05:54:44.059 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7499999999999997e-06 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615462+0000", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4750000000000002e-05 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:08.615470+0000", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6550000000000004e-06 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.618963+0000", 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.003493653 2024-04-25T05:54:44.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619188+0000", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00022493599999999999 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.619317+0000", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012912399999999999 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.621613+0000", 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022951650000000001 2024-04-25T05:54:44.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.629456+0000", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0078430449999999999 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632635+0000", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003178883 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632691+0000", 2024-04-25T05:54:44.062 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5965999999999999e-05 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:09.632737+0000", 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6304000000000003e-05 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232724 10.17 10:eb2728fc:::benchmark_data_smithi043_152558_object104374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 301.64786430599997, 2024-04-25T05:54:44.063 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0079804210000001, 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232724 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.064 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317407+0000", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317409+0000", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9240000000000001e-06 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317418+0000", 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3579999999999992e-06 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317437+0000", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.9550000000000001e-05 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317449+0000", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1626e-05 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317528+0000", 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9333999999999996e-05 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.317539+0000", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0848e-05 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322050+0000", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0045109210000001 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322151+0000", 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000101123 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 3,6", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.322389+0000", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023737899999999999 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325247+0000", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002858401 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325260+0000", 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3125999999999999e-05 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.068 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325296+0000", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6239000000000003e-05 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325348+0000", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1251000000000001e-05 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325387+0000", 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9341000000000002e-05 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.069 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211897 11.e 11:7dd02b31:::benchmark_data_smithi043_179003_object81608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 89.468237689000006, 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.108093957, 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211897 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.070 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497033+0000", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497035+0000", 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.587e-06 2024-04-25T05:54:44.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497042+0000", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7710000000000001e-06 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497053+0000", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1044e-05 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497061+0000", 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4859999999999997e-06 2024-04-25T05:54:44.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497615+0000", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00055385700000000005 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497748+0000", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013329999999999999 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.497865+0000", 2024-04-25T05:54:44.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116537 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502484+0000", 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046191119999999999 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504633+0000", 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002148736 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605053+0000", 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100420537 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605093+0000", 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9514999999999997e-05 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605127+0000", 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4474999999999998e-05 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211899 11.e 11:79b67a63:::benchmark_data_smithi043_179003_object81610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 89.467115151000002, 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.107157866, 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211899 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498156+0000", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498157+0000", 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0729999999999999e-06 2024-04-25T05:54:44.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498161+0000", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0770000000000001e-06 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498172+0000", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0849999999999999e-05 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.498181+0000", 2024-04-25T05:54:44.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7420000000000003e-06 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500317+0000", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021363480000000002 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500437+0000", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120166 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.500515+0000", 2024-04-25T05:54:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8140999999999996e-05 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504641+0000", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041259390000000003 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504707+0000", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6043000000000004e-05 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605187+0000", 2024-04-25T05:54:44.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10047949 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605249+0000", 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2296000000000001e-05 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605314+0000", 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4701000000000003e-05 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211902 11.e 11:7cddbeec:::benchmark_data_smithi043_179003_object81613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 89.463949807000006, 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.104091134, 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211902 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501321+0000", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501322+0000", 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1540000000000001e-06 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501327+0000", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9200000000000003e-06 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501339+0000", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2099000000000001e-05 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501347+0000", 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4730000000000003e-06 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502526+0000", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0011796899999999999 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502660+0000", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013350800000000001 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502787+0000", 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012737499999999999 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504782+0000", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019942509999999998 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.505325+0000", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00054379099999999996 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605345+0000", 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100019497 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605374+0000", 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9099999999999999e-05 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.605412+0000", 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8275999999999997e-05 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:226478 11.e 11:78be6c36:::benchmark_data_smithi043_179003_object96189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:29.695479+0000", 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 74.269791832999999, 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.050648936999999998, 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 226478 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695479+0000", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695479+0000", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695480+0000", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1399999999999995e-07 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:44.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695485+0000", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2480000000000004e-06 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695496+0000", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0572e-05 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.695503+0000", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6359999999999994e-06 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:44.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.697820+0000", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023160659999999999 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.697902+0000", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2538000000000002e-05 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,7", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.725771+0000", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.027869373999999999 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.728812+0000", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030402279999999999 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.728825+0000", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3549e-05 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.745962+0000", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.017137243 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:44.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.746070+0000", 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107625 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:29.746128+0000", 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7944e-05 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:44.093 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:44.207 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:44.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:44 smithi043 ceph-mon[27010]: 2.1 scrub starts 2024-04-25T05:54:44.458 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:44 smithi043 ceph-mon[27010]: 2.1 scrub ok 2024-04-25T05:54:44.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:44 smithi043 ceph-mon[37492]: 2.1 scrub starts 2024-04-25T05:54:44.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:44 smithi043 ceph-mon[37492]: 2.1 scrub ok 2024-04-25T05:54:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:44 smithi060 ceph-mon[30898]: 2.1 scrub starts 2024-04-25T05:54:44.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:44 smithi060 ceph-mon[30898]: 2.1 scrub ok 2024-04-25T05:54:45.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-25T05:54:45.139 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-25T05:54:45.352 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:54:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: 6.1e scrub starts 2024-04-25T05:54:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: 6.1e scrub ok 2024-04-25T05:54:45.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/529412848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[27010]: osdmap e600: 8 total, 8 up, 8 in 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: 6.1e scrub starts 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: 6.1e scrub ok 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/529412848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:45.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:45 smithi043 ceph-mon[37492]: osdmap e600: 8 total, 8 up, 8 in 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: 6.1e scrub starts 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: 6.1e scrub ok 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/529412848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-25T05:54:45.475 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:45 smithi060 ceph-mon[30898]: osdmap e600: 8 total, 8 up, 8 in 2024-04-25T05:54:45.847 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:45.847 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:45.847 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:54:45.848 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:45.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-25T05:54:46.109 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:46.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 6.5 scrub starts 2024-04-25T05:54:46.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 5.7 deep-scrub starts 2024-04-25T05:54:46.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 5.7 deep-scrub ok 2024-04-25T05:54:46.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 6.5 scrub ok 2024-04-25T05:54:46.487 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 6.c scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 6.c scrub ok 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 4.9 deep-scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[27010]: 4.9 deep-scrub ok 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 6.5 scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 5.7 deep-scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 5.7 deep-scrub ok 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 6.5 scrub ok 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 6.c scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 6.c scrub ok 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 4.9 deep-scrub starts 2024-04-25T05:54:46.488 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:46 smithi043 ceph-mon[37492]: 4.9 deep-scrub ok 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 6.5 scrub starts 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 5.7 deep-scrub starts 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 5.7 deep-scrub ok 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 6.5 scrub ok 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 6.c scrub starts 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 6.c scrub ok 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 4.9 deep-scrub starts 2024-04-25T05:54:46.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:46 smithi060 ceph-mon[30898]: 4.9 deep-scrub ok 2024-04-25T05:54:46.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-25T05:54:46.793 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:47.223 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T05:54:47.304 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:47.305 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:47.305 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:54:47.305 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:54:47.305 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 5.9 scrub starts 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 5.9 scrub ok 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 4.13 scrub starts 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 4.13 scrub ok 2024-04-25T05:54:47.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 6.14 scrub starts 2024-04-25T05:54:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 6.14 scrub ok 2024-04-25T05:54:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 4.4 deep-scrub starts 2024-04-25T05:54:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: 4.4 deep-scrub ok 2024-04-25T05:54:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3957445494' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:47 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.624 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 5.9 scrub starts 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 5.9 scrub ok 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 4.13 scrub starts 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 4.13 scrub ok 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 6.14 scrub starts 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 6.14 scrub ok 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 4.4 deep-scrub starts 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: 4.4 deep-scrub ok 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3957445494' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.625 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 5.9 scrub starts 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 5.9 scrub ok 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: pgmap v1132: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 4.13 scrub starts 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 4.13 scrub ok 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 6.14 scrub starts 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 6.14 scrub ok 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 4.4 deep-scrub starts 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: 4.4 deep-scrub ok 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3957445494' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.626 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:47 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:47.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-25T05:54:48.134 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:48.178 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-25T05:54:48.436 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 5.16 deep-scrub starts 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 5.16 deep-scrub ok 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: osdmap e601: 8 total, 8 up, 8 in 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 4.3 scrub starts 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 4.3 scrub ok 2024-04-25T05:54:48.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 5.8 scrub starts 2024-04-25T05:54:48.500 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:48 smithi060 ceph-mon[30898]: 5.8 scrub ok 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 5.16 deep-scrub starts 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 5.16 deep-scrub ok 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: osdmap e601: 8 total, 8 up, 8 in 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 4.3 scrub starts 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 4.3 scrub ok 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 5.8 scrub starts 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[37492]: 5.8 scrub ok 2024-04-25T05:54:48.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 5.16 deep-scrub starts 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 5.16 deep-scrub ok 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: osdmap e601: 8 total, 8 up, 8 in 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 4.3 scrub starts 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 4.3 scrub ok 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 5.8 scrub starts 2024-04-25T05:54:48.560 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:48 smithi043 ceph-mon[27010]: 5.8 scrub ok 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:224215 10.9 10:9299bc49:::benchmark_data_smithi043_152558_object95865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 316.503914114, 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.21245112399999999, 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.645 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 224215 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.646 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139271+0000", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139272+0000", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.248e-06 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139278+0000", 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8490000000000002e-06 2024-04-25T05:54:48.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139294+0000", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6143000000000001e-05 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139302+0000", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2830000000000001e-06 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139363+0000", 2024-04-25T05:54:48.648 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0409999999999999e-05 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139498+0000", 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013544100000000001 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,7", 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.139614+0000", 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000115869 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.649 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144295+0000", 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046808809999999996 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.145168+0000", 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00087276499999999996 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.351707+0000", 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.206538738 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.650 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.351722+0000", 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5497000000000001e-05 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:224217 10.6 10:6c501e56:::benchmark_data_smithi043_152558_object95867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 316.50176061399998, 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.22087231800000001, 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.651 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "reached pg", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 224217 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141424+0000", 2024-04-25T05:54:48.652 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141426+0000", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.139e-06 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141431+0000", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5419999999999996e-06 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141442+0000", 2024-04-25T05:54:48.653 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0774000000000001e-05 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.141450+0000", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7030000000000006e-06 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144619+0000", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031695970000000001 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144806+0000", 2024-04-25T05:54:48.654 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018734599999999999 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.144937+0000", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013011199999999999 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.160892+0000", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015955882000000001 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.362282+0000", 2024-04-25T05:54:48.655 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.201389543 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:32.362297+0000", 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.468e-05 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171805 11.7 11:e5d8e593:::benchmark_data_smithi043_179003_object41516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.304747+0000", 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.33843845199999, 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.113796629, 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.656 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171805 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304747+0000", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304747+0000", 2024-04-25T05:54:48.657 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304748+0000", 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1090000000000001e-06 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304753+0000", 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1900000000000003e-06 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.658 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304763+0000", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.06e-05 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304769+0000", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8660000000000002e-06 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304846+0000", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6216000000000003e-05 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.659 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.304989+0000", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014341899999999999 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.305114+0000", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000124976 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.307917+0000", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002802854 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.660 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.308318+0000", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000401627 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418438+0000", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.110120015 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418495+0000", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6814999999999997e-05 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418543+0000", 2024-04-25T05:54:48.661 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7942000000000003e-05 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171816 11.7 11:e3dc0a58:::benchmark_data_smithi043_179003_object41527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.313311+0000", 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.32987383400001, 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10538792700000001, 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.662 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171816 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313311+0000", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313311+0000", 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313313+0000", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.497e-06 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313319+0000", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3080000000000004e-06 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313330+0000", 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0813e-05 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313338+0000", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1969999999999996e-06 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313399+0000", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0565e-05 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313524+0000", 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012534199999999999 2024-04-25T05:54:48.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.313672+0000", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014786199999999999 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.317285+0000", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036134299999999999 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.321681+0000", 2024-04-25T05:54:48.666 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043960279999999997 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418604+0000", 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096922675 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418657+0000", 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3223000000000002e-05 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.667 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418699+0000", 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1987000000000003e-05 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171820 11.7 11:ed082b63:::benchmark_data_smithi043_179003_object41531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.317990+0000", 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.32519549099999, 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.100894125, 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.668 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171820 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.317990+0000", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.317990+0000", 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.669 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.317991+0000", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.415e-06 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.317997+0000", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8540000000000002e-06 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.318012+0000", 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5409000000000001e-05 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.318020+0000", 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8569999999999995e-06 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.671 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.318074+0000", 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.431e-05 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.318204+0000", 2024-04-25T05:54:48.672 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012983100000000001 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.318324+0000", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119611 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.321460+0000", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031362759999999999 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.321798+0000", 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00033750499999999999 2024-04-25T05:54:48.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418760+0000", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.096962620999999999 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418838+0000", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8277999999999998e-05 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418884+0000", 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5157999999999997e-05 2024-04-25T05:54:48.674 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171828 11.7 11:ef48b6fa:::benchmark_data_smithi043_179003_object41539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.324808+0000", 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.31837689899999, 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.094225277999999996, 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.675 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171828 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324808+0000", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324808+0000", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324810+0000", 2024-04-25T05:54:48.676 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4649999999999999e-06 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324816+0000", 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0900000000000001e-06 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324831+0000", 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5709000000000002e-05 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.677 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324840+0000", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5960000000000004e-06 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.324899+0000", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9144999999999997e-05 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325038+0000", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139345 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.678 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325159+0000", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012102799999999999 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.330164+0000", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0050043709999999996 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.330219+0000", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4694999999999997e-05 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418931+0000", 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.088712297999999995 2024-04-25T05:54:48.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.418988+0000", 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7507999999999999e-05 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.419033+0000", 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5028000000000003e-05 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.680 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171829 11.7 11:eb24136b:::benchmark_data_smithi043_179003_object41540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.325204+0000", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.31798056700001, 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.104423167, 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171829 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325204+0000", 2024-04-25T05:54:48.681 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325204+0000", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325205+0000", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.61e-07 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325211+0000", 2024-04-25T05:54:48.682 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3990000000000003e-06 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325220+0000", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3400000000000004e-06 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.325228+0000", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0069999999999997e-06 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.327686+0000", 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024577380000000001 2024-04-25T05:54:48.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.327821+0000", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000135313 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.327919+0000", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7800000000000006e-05 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.330269+0000", 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023499810000000001 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.330278+0000", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4210000000000001e-06 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429528+0000", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.099250273999999999 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429582+0000", 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3375999999999999e-05 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.685 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429628+0000", 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5757000000000003e-05 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:171834 11.7 11:eb64424b:::benchmark_data_smithi043_179003_object41545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:31.331551+0000", 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.311634387, 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098212805, 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.686 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 171834 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331551+0000", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331551+0000", 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331552+0000", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1489999999999999e-06 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331556+0000", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0489999999999997e-06 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331565+0000", 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8279999999999992e-06 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331572+0000", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9859999999999997e-06 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331630+0000", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8545000000000003e-05 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331747+0000", 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116542 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.689 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.331863+0000", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000116655 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.334648+0000", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002784538 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.335037+0000", 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000388957 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.691 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.691 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429674+0000", 2024-04-25T05:54:48.691 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.094636814 2024-04-25T05:54:48.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.693 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.693 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429724+0000", 2024-04-25T05:54:48.693 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0260000000000003e-05 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:31.429763+0000", 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9481999999999999e-05 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:189153 11.7 11:eb4bf917:::benchmark_data_smithi043_179003_object58864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:51.299670+0000", 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 117.343514863, 2024-04-25T05:54:48.694 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.044701078999999998, 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 189153 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299670+0000", 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.695 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299670+0000", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299672+0000", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.621e-06 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299678+0000", 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3579999999999999e-06 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.696 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299696+0000", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.7445999999999999e-05 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299705+0000", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2369999999999998e-06 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299775+0000", 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0471999999999999e-05 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.697 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.299910+0000", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013448400000000001 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.300028+0000", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011852499999999999 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.303246+0000", 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0032174360000000002 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.303330+0000", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4499999999999994e-05 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.344272+0000", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.040942185999999998 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.344316+0000", 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3463e-05 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.699 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:51.344371+0000", 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5350999999999999e-05 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:209222 11.7 11:ecff28a9:::benchmark_data_smithi043_179003_object78933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:11.588805+0000", 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 97.054380260000002, 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.047356477000000001, 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.700 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 209222 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588805+0000", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588805+0000", 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588806+0000", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1999999999999998e-07 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588810+0000", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5499999999999996e-06 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588818+0000", 2024-04-25T05:54:48.702 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9400000000000002e-06 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588824+0000", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3870000000000003e-06 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588859+0000", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5601e-05 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.588941+0000", 2024-04-25T05:54:48.703 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2159999999999999e-05 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.589023+0000", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1872999999999994e-05 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.591546+0000", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002522406 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.591745+0000", 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019974599999999999 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.636076+0000", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.044330571999999999 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.636113+0000", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7008000000000002e-05 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.636161+0000", 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8313999999999999e-05 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.705 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:209451 11.7 11:ebbe6b59:::benchmark_data_smithi043_179003_object79162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:11.822555+0000", 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 96.820629745999994, 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.042679241999999999, 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 209451 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.706 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822555+0000", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822555+0000", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822557+0000", 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2750000000000001e-06 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822562+0000", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2220000000000001e-06 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822573+0000", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1433999999999999e-05 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822581+0000", 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0639999999999994e-06 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.708 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822653+0000", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1846000000000003e-05 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822742+0000", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8704999999999999e-05 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.822816+0000", 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4062999999999999e-05 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.709 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.825645+0000", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0028289560000000001 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.826041+0000", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039574199999999998 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.865162+0000", 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039121259999999998 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.865200+0000", 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8482000000000002e-05 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:11.865235+0000", 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4193000000000003e-05 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211905 11.7 11:efb4974e:::benchmark_data_smithi043_179003_object81616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:48.711 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.501581+0000", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 94.141603790000005, 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.10964755299999999, 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211905 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501581+0000", 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501581+0000", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501582+0000", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.074e-06 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501588+0000", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6699999999999999e-06 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.713 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501603+0000", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5276000000000001e-05 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501611+0000", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5449999999999998e-06 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501676+0000", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5500999999999995e-05 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.714 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501799+0000", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012258599999999999 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.501911+0000", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000112112 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504519+0000", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026076239999999998 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.715 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.506802+0000", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022838730000000001 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611128+0000", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.104325264 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611177+0000", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9638999999999997e-05 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.716 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611229+0000", 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1388999999999999e-05 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:211907 11.7 11:e61f7fb0:::benchmark_data_smithi043_179003_object81618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e555)", 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:14.502176+0000", 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 94.141008558999999, 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.109219952, 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.717 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 211907 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502176+0000", 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502176+0000", 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.718 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502177+0000", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1900000000000002e-07 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502181+0000", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5990000000000002e-06 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502192+0000", 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1513e-05 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.719 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.502198+0000", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.553e-06 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504549+0000", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002351049 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504692+0000", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000143153 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.504832+0000", 2024-04-25T05:54:48.720 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000140314 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.506861+0000", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020283269999999999 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.507000+0000", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013980000000000001 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611293+0000", 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.104292754 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611349+0000", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5727999999999999e-05 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:14.611396+0000", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7543000000000002e-05 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:228570 11.7 11:e882eace:::benchmark_data_smithi043_179003_object98281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:31.664743+0000", 2024-04-25T05:54:48.722 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 76.978442149000003, 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.044130965000000001, 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 228570 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664743+0000", 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664743+0000", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664744+0000", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4279999999999999e-06 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664750+0000", 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4820000000000002e-06 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664759+0000", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0389999999999993e-06 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.664766+0000", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8809999999999998e-06 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.675069+0000", 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010302877 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.725 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.675237+0000", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016813099999999999 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.675456+0000", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021897600000000001 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.687034+0000", 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011578316999999999 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.692422+0000", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0053877300000000003 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.708735+0000", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.016313580000000001 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.708813+0000", 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7420000000000001e-05 2024-04-25T05:54:48.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:31.708874+0000", 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1104000000000003e-05 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:240870 11.7 11:e59fcb1d:::benchmark_data_smithi043_179003_object110581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e569)", 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:43.537021+0000", 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 65.106163585000004, 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.047834005999999998, 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 240870 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537021+0000", 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.729 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537021+0000", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537023+0000", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5460000000000001e-06 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537034+0000", 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1419e-05 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.730 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537053+0000", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8587999999999999e-05 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537062+0000", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1370000000000008e-06 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537121+0000", 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9265999999999998e-05 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.731 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537265+0000", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014352399999999999 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.537378+0000", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011293099999999999 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.540495+0000", 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0031168760000000002 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.732 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.541802+0000", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0013070270000000001 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.584694+0000", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.042892129000000001 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.584806+0000", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011234999999999999 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.733 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:43.584855+0000", 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9212999999999998e-05 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:257963 11.7 11:e84b3b44:::benchmark_data_smithi043_179003_object127674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:53:59.234133+0000", 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 49.409052125000002, 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.042830752999999999, 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.734 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 257963 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234133+0000", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234133+0000", 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234134+0000", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9399999999999993e-07 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234146+0000", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1755e-05 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234156+0000", 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9799999999999993e-06 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.737 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.737 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234162+0000", 2024-04-25T05:54:48.737 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5320000000000003e-06 2024-04-25T05:54:48.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234204+0000", 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1458000000000002e-05 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234296+0000", 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1988000000000001e-05 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.738 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.234373+0000", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7155000000000002e-05 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.237331+0000", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002958362 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.237566+0000", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023465199999999999 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.739 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.276881+0000", 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039315645000000003 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.276917+0000", 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5751999999999999e-05 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:53:59.276964+0000", 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6480000000000002e-05 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:258909 11.7 11:e855f931:::benchmark_data_smithi043_179003_object128620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:00.196521+0000", 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 48.446664421000001, 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.045150366999999997, 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 258909 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.741 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196521+0000", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196521+0000", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196522+0000", 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3370000000000001e-06 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.742 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196530+0000", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7819999999999994e-06 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196548+0000", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8240000000000002e-05 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196558+0000", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0169999999999999e-05 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.743 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196629+0000", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.0611000000000005e-05 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196751+0000", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000122414 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.196879+0000", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127358 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.201580+0000", 2024-04-25T05:54:48.744 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0047012180000000001 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.201625+0000", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.5164000000000003e-05 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.241544+0000", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.039919494999999999 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.241616+0000", 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1167000000000005e-05 2024-04-25T05:54:48.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.241671+0000", 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5411000000000002e-05 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:258912 11.7 11:ef779de7:::benchmark_data_smithi043_179003_object128623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:00.198513+0000", 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 48.444671655, 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.045628275000000003, 2024-04-25T05:54:48.746 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 258912 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198513+0000", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.747 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198513+0000", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198514+0000", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.035e-06 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198520+0000", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2240000000000003e-06 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.748 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198531+0000", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1209000000000001e-05 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.198537+0000", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8629999999999996e-06 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.199547+0000", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010102780000000001 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.199647+0000", 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000100499 2024-04-25T05:54:48.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.199739+0000", 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1121999999999999e-05 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.201722+0000", 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001983589 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.750 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.201803+0000", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0496999999999999e-05 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.244040+0000", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.042236976000000002 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.244097+0000", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7429999999999997e-05 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.751 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:00.244142+0000", 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4552999999999998e-05 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260070 11.7 11:eb307dbc:::benchmark_data_smithi043_179003_object129781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.347492+0000", 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 47.295693092999997, 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.076540933000000005, 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.752 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260070 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347492+0000", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347492+0000", 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347493+0000", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3909999999999999e-06 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347512+0000", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.8340000000000001e-05 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347526+0000", 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3978000000000001e-05 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347534+0000", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.49e-06 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347603+0000", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9349000000000003e-05 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347723+0000", 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011910500000000001 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.347844+0000", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012122599999999999 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.350900+0000", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030565169999999999 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.351268+0000", 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036812299999999999 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.423898+0000", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.072629637999999996 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.423965+0000", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7385999999999995e-05 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424033+0000", 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7390000000000004e-05 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260097 11.7 11:e040ae52:::benchmark_data_smithi043_179003_object129808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.370183+0000", 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 47.273001544000003, 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.054010651, 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260097 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.758 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370183+0000", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370183+0000", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370185+0000", 2024-04-25T05:54:48.759 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.085e-06 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370192+0000", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6149999999999999e-06 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370206+0000", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3905e-05 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370215+0000", 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9150000000000002e-06 2024-04-25T05:54:48.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370288+0000", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2633999999999996e-05 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370431+0000", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014361500000000001 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,4", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370567+0000", 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000136143 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.761 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.373649+0000", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030819110000000001 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.373958+0000", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00030906500000000002 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424098+0000", 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.050139228000000001 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.762 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424145+0000", 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7398999999999998e-05 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424194+0000", 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9135999999999998e-05 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:48.763 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:49.232 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-25T05:54:49.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-25T05:54:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:49 smithi060 ceph-mon[30898]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:49 smithi060 ceph-mon[30898]: 3.1 scrub starts 2024-04-25T05:54:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:49 smithi060 ceph-mon[30898]: 3.1 scrub ok 2024-04-25T05:54:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:49 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:49.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:49 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:49.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[37492]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:49.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[37492]: 3.1 scrub starts 2024-04-25T05:54:49.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[37492]: 3.1 scrub ok 2024-04-25T05:54:49.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:49.617 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:49.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[27010]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:49.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[27010]: 3.1 scrub starts 2024-04-25T05:54:49.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[27010]: 3.1 scrub ok 2024-04-25T05:54:49.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:54:49.618 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-25T05:54:49.674 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:49.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:49 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:54:50.214 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:50.215 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:50.215 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:54:50.215 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:50.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-25T05:54:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:50 smithi060 ceph-mon[30898]: 3.15 scrub starts 2024-04-25T05:54:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:50 smithi060 ceph-mon[30898]: 3.15 scrub ok 2024-04-25T05:54:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:50 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:50.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:50 smithi060 ceph-mon[30898]: osdmap e602: 8 total, 8 up, 8 in 2024-04-25T05:54:50.601 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[27010]: 3.15 scrub starts 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[27010]: 3.15 scrub ok 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[27010]: osdmap e602: 8 total, 8 up, 8 in 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[37492]: 3.15 scrub starts 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[37492]: 3.15 scrub ok 2024-04-25T05:54:50.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2656202307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-25T05:54:50.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:50 smithi043 ceph-mon[37492]: osdmap e602: 8 total, 8 up, 8 in 2024-04-25T05:54:50.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-25T05:54:51.066 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:51.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 4.1f scrub starts 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 4.1f scrub ok 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 3.1a scrub starts 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 3.1a scrub ok 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 4.16 deep-scrub starts 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 4.16 deep-scrub ok 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 6.f scrub starts 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[27010]: 6.f scrub ok 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 4.1f scrub starts 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 4.1f scrub ok 2024-04-25T05:54:51.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 3.1a scrub starts 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 3.1a scrub ok 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 4.16 deep-scrub starts 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 4.16 deep-scrub ok 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 6.f scrub starts 2024-04-25T05:54:51.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:51 smithi043 ceph-mon[37492]: 6.f scrub ok 2024-04-25T05:54:51.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 4.1f scrub starts 2024-04-25T05:54:51.449 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 4.1f scrub ok 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: pgmap v1136: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 3.1a scrub starts 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 3.1a scrub ok 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 4.16 deep-scrub starts 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 4.16 deep-scrub ok 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 6.f scrub starts 2024-04-25T05:54:51.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:51 smithi060 ceph-mon[30898]: 6.f scrub ok 2024-04-25T05:54:51.577 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:51.577 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:51.577 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:54:51.577 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:54:51.577 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:52.248 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T05:54:52.332 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-25T05:54:52.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:52 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:52.581 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:52.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:52 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:52.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:52 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:53.045 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-25T05:54:53.070 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:53.070 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:54:53.070 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232758 10.16 10:6a536367:::benchmark_data_smithi043_152558_object104408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 310.71740729200002, 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99219000999999996, 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232758 2024-04-25T05:54:53.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350942+0000", 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.072 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350943+0000", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3699999999999999e-07 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350948+0000", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2919999999999997e-06 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350957+0000", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7089999999999997e-06 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.073 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.350965+0000", 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5610000000000001e-06 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351038+0000", 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3561999999999998e-05 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.351047+0000", 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4549999999999995e-06 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323496+0000", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97244899900000004 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323612+0000", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011592200000000001 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.323850+0000", 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00023792799999999999 2024-04-25T05:54:53.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326511+0000", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026609490000000001 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329075+0000", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002564607 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343036+0000", 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013961031 2024-04-25T05:54:53.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343089+0000", 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2522999999999998e-05 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343132+0000", 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3634999999999998e-05 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232859 10.16 10:6a2a3b0c:::benchmark_data_smithi043_152558_object104509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 310.629846058, 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90477883999999997, 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232859 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.078 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438504+0000", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438505+0000", 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5820000000000001e-06 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438514+0000", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3529999999999995e-06 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438522+0000", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7870000000000007e-06 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438528+0000", 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6030000000000004e-06 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438592+0000", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4029000000000006e-05 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.438602+0000", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8349999999999999e-06 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325262+0000", 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88666051999999995 2024-04-25T05:54:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325370+0000", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010740599999999999 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.325516+0000", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000146389 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329163+0000", 2024-04-25T05:54:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036465989999999999 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329175+0000", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2473999999999999e-05 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343188+0000", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.014012754000000001 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343242+0000", 2024-04-25T05:54:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3992000000000001e-05 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343283+0000", 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0516999999999998e-05 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232861 10.16 10:6d117552:::benchmark_data_smithi043_152558_object104511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 310.62611671899998, 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.90115485500000003, 2024-04-25T05:54:53.084 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232861 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.085 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442233+0000", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442234+0000", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2589999999999999e-06 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442239+0000", 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3900000000000003e-06 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442248+0000", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4299999999999995e-06 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442257+0000", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2199999999999998e-06 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442330+0000", 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2401e-05 2024-04-25T05:54:53.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.442341+0000", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1063999999999999e-05 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326560+0000", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88421923400000002 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326658+0000", 2024-04-25T05:54:53.088 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7768000000000003e-05 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.326835+0000", 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000177527 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329180+0000", 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023450599999999999 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.089 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330157+0000", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00097623399999999998 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343317+0000", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013160420000000001 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343352+0000", 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5067000000000002e-05 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.090 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.343388+0000", 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5781000000000002e-05 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232866 10.16 10:6de63d91:::benchmark_data_smithi043_152558_object104516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:54:53.091 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 310.61170683400002, 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88762149599999995, 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232866 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.092 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456643+0000", 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456644+0000", 2024-04-25T05:54:53.093 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5700000000000001e-07 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456648+0000", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.69e-06 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456658+0000", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0780000000000002e-06 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456663+0000", 2024-04-25T05:54:53.094 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5609999999999998e-06 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456723+0000", 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9577999999999998e-05 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:42.456731+0000", 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1929999999999993e-06 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327815+0000", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.871084317 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.327953+0000", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000137389 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,5", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.328167+0000", 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000214324 2024-04-25T05:54:53.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.329185+0000", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001018166 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.330231+0000", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0010455309999999999 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344181+0000", 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.013950071 2024-04-25T05:54:53.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.098 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344227+0000", 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6589999999999999e-05 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.344264+0000", 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7150999999999997e-05 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.099 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232888 10.12 10:4f53b376:::benchmark_data_smithi043_152558_object104538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 309.72062123500001, 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96954986499999996, 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232888 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.100 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347729+0000", 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347731+0000", 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.295e-06 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.101 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347736+0000", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3299999999999998e-06 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347752+0000", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6121e-05 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347760+0000", 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1659999999999994e-06 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347880+0000", 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011932199999999999 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.347889+0000", 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.482e-06 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.103 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314176+0000", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96628726499999995 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314286+0000", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010990899999999999 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.314365+0000", 2024-04-25T05:54:53.104 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9061999999999996e-05 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317147+0000", 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027818740000000002 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317183+0000", 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5639000000000002e-05 2024-04-25T05:54:53.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317219+0000", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6324e-05 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317250+0000", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.0596000000000003e-05 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317278+0000", 2024-04-25T05:54:53.106 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8479999999999998e-05 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:232966 10.12 10:49a888b9:::benchmark_data_smithi043_152558_object104616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 309.673163859, 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92239090000000001, 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.107 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 232966 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395186+0000", 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395187+0000", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2399999999999997e-07 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395190+0000", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4920000000000002e-06 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395200+0000", 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.7589999999999992e-06 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.109 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395208+0000", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8110000000000003e-06 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395287+0000", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9079000000000006e-05 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.395295+0000", 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8080000000000005e-06 2024-04-25T05:54:53.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315576+0000", 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.92028153000000001 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315678+0000", 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102095 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.111 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.315768+0000", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0304999999999997e-05 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317349+0000", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00158056 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317389+0000", 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9878e-05 2024-04-25T05:54:53.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317472+0000", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2762000000000005e-05 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317523+0000", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1730000000000001e-05 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317577+0000", 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3366999999999999e-05 2024-04-25T05:54:53.113 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.15975.0:233055 10.12 10:4ec1d645:::benchmark_data_smithi043_152558_object104705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 309.621242451, 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87067929700000002, 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.15975", 2024-04-25T05:54:53.114 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4274713654", 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 233055 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447107+0000", 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447109+0000", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4980000000000001e-06 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447114+0000", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1510000000000003e-06 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447128+0000", 2024-04-25T05:54:53.116 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4012000000000001e-05 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447136+0000", 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7579999999999992e-06 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447275+0000", 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013975899999999999 2024-04-25T05:54:53.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:43.447283+0000", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6590000000000007e-06 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316413+0000", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.86913016700000001 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316518+0000", 2024-04-25T05:54:53.118 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010422800000000001 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,5", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.316597+0000", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9097000000000005e-05 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317594+0000", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00099745499999999996 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317631+0000", 2024-04-25T05:54:53.119 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7342999999999998e-05 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317700+0000", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8861000000000004e-05 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317752+0000", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1872000000000001e-05 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:49:44.317787+0000", 2024-04-25T05:54:53.120 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4437000000000002e-05 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175433 11.1 11:8fba1ab6:::benchmark_data_smithi043_179003_object45144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.977574936, 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0345604850000001, 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.121 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175433 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090775+0000", 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.122 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090776+0000", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.429e-06 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090784+0000", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2649999999999999e-06 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090794+0000", 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0552e-05 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.123 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.090802+0000", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4819999999999997e-06 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.095151+0000", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043491149999999998 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.095160+0000", 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4530000000000008e-06 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113721+0000", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0185605760000001 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113832+0000", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011136 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.113914+0000", 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2316000000000003e-05 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.125 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120575+0000", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066603720000000003 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122121+0000", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001545886 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125210+0000", 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030893130000000002 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125277+0000", 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7334000000000003e-05 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125335+0000", 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8032000000000003e-05 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175439 11.1 11:8a7d81ef:::benchmark_data_smithi043_179003_object45150:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.96567328099999, 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0228662150000001, 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175439 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.128 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102676+0000", 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102678+0000", 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.451e-06 2024-04-25T05:54:53.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102683+0000", 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5180000000000004e-06 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102698+0000", 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4681999999999999e-05 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.130 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102706+0000", 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.157e-06 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102768+0000", 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1227000000000002e-05 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.102775+0000", 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3830000000000003e-06 2024-04-25T05:54:53.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116168+0000", 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0133934600000001 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116237+0000", 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8560999999999996e-05 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.132 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.116308+0000", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1373999999999998e-05 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122048+0000", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005739971 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125378+0000", 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0033293210000000001 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.133 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125435+0000", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7147e-05 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125495+0000", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0485000000000001e-05 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125543+0000", 2024-04-25T05:54:53.134 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7478000000000002e-05 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175466 11.1 11:8d3c0ad7:::benchmark_data_smithi043_179003_object45177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.92245785099999, 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98401796699999999, 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.135 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175466 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145892+0000", 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145894+0000", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.637e-06 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145899+0000", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6579999999999999e-06 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145909+0000", 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9199999999999999e-06 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.137 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.145917+0000", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1589999999999993e-06 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.146013+0000", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6075000000000004e-05 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.146022+0000", 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4169999999999992e-06 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117397+0000", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97137537299999999 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117465+0000", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7768000000000006e-05 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.117530+0000", 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5222000000000007e-05 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.139 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122052+0000", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045222259999999999 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125583+0000", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035309809999999999 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129846+0000", 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0042627150000000003 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129881+0000", 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.4703000000000003e-05 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129910+0000", 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9113000000000001e-05 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175468 11.1 11:81a32ea0:::benchmark_data_smithi043_179003_object45179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.141 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.92006651200001, 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.985623044, 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175468 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.142 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148283+0000", 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148284+0000", 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.569999999999999e-07 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.143 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148289+0000", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7129999999999996e-06 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148302+0000", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2636999999999999e-05 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148310+0000", 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3070000000000003e-06 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148364+0000", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4471000000000001e-05 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.148374+0000", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5480000000000007e-06 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118654+0000", 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.970280166 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118718+0000", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.4387000000000005e-05 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.118893+0000", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017421799999999999 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123418+0000", 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045254049999999997 2024-04-25T05:54:53.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129746+0000", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0063283230000000003 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133841+0000", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0040942069999999999 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133874+0000", 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3290000000000001e-05 2024-04-25T05:54:53.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133906+0000", 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2415000000000001e-05 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175488 11.1 11:883cd18e:::benchmark_data_smithi043_179003_object45199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.90069630799999, 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96641953199999997, 2024-04-25T05:54:53.148 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175488 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.149 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167653+0000", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167655+0000", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5680000000000001e-06 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167660+0000", 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7600000000000002e-06 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.150 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167671+0000", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0910000000000001e-05 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167676+0000", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4809999999999997e-06 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167732+0000", 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5695999999999997e-05 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.167741+0000", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4979999999999995e-06 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120665+0000", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95292317699999995 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120734+0000", 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9831000000000003e-05 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.120808+0000", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3175000000000003e-05 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125163+0000", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004355261 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129797+0000", 2024-04-25T05:54:53.153 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0046343199999999999 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.133968+0000", 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0041712779999999996 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134028+0000", 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9709e-05 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.154 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134073+0000", 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4867999999999998e-05 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175491 11.1 11:86b5808b:::benchmark_data_smithi043_179003_object45202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.89707386200001, 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.974324462, 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175491 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171276+0000", 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171277+0000", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6899999999999996e-07 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171281+0000", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2169999999999996e-06 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171290+0000", 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8130000000000002e-06 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171295+0000", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5199999999999997e-06 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171346+0000", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0958000000000002e-05 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.171356+0000", 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3519999999999996e-06 2024-04-25T05:54:53.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122164+0000", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95080842499999996 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122231+0000", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7368999999999998e-05 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.122299+0000", 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.7893999999999997e-05 2024-04-25T05:54:53.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125386+0000", 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030862279999999999 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129945+0000", 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0045593869999999998 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.160 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145473+0000", 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0155277 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145548+0000", 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5539999999999998e-05 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.145600+0000", 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.219e-05 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175505 11.1 11:8036578e:::benchmark_data_smithi043_179003_object45216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.88546991499999, 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.96335497000000003, 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175505 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.162 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182880+0000", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182881+0000", 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.054e-06 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.163 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182886+0000", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8539999999999996e-06 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182895+0000", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6309999999999998e-06 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182904+0000", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1529999999999997e-06 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.164 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182990+0000", 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6946000000000004e-05 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.182999+0000", 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7150000000000004e-06 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123438+0000", 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.94043926 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123505+0000", 2024-04-25T05:54:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6879999999999997e-05 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.123591+0000", 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.5607999999999998e-05 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125590+0000", 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0019993179999999999 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130005+0000", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044150659999999996 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146160+0000", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.016154294999999999 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146193+0000", 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3114999999999997e-05 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.146235+0000", 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2074999999999999e-05 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175516 11.1 11:8230b7a3:::benchmark_data_smithi043_179003_object45227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.87475249400001, 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.95606739399999996, 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175516 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193597+0000", 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193599+0000", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4360000000000001e-06 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193605+0000", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2310000000000001e-06 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193619+0000", 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4037e-05 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193627+0000", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2109999999999998e-06 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193690+0000", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3099e-05 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.193696+0000", 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2020000000000001e-06 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125612+0000", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.93191594499999997 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125737+0000", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012487699999999999 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.125873+0000", 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000135437 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129755+0000", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0038821340000000002 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.140090+0000", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010335247000000001 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149600+0000", 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0095101639999999998 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149636+0000", 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5814e-05 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149665+0000", 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8560000000000001e-05 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175586 11.1 11:81fb6593:::benchmark_data_smithi043_179003_object45297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.80628526699999, 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.88770023200000003, 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175586 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.176 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262064+0000", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262066+0000", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.6810000000000001e-06 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.177 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262072+0000", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1630000000000001e-06 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262088+0000", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.5690999999999999e-05 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262096+0000", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.3829999999999992e-06 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.178 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262160+0000", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3906999999999996e-05 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.262169+0000", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6840000000000002e-06 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.127891+0000", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.86572203599999997 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.179 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.128019+0000", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000127787 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.128139+0000", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119997 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.129956+0000", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0018173009999999999 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.180 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.140176+0000", 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.010219788 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149703+0000", 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0095273489999999992 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149735+0000", 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.2048000000000002e-05 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149765+0000", 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9417e-05 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175599 11.1 11:8d1f405c:::benchmark_data_smithi043_179003_object45310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.790597529, 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.87487917199999998, 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.182 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175599 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.183 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277752+0000", 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277753+0000", 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7000000000000003e-07 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277761+0000", 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5660000000000001e-06 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.184 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277770+0000", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5920000000000007e-06 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277776+0000", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4199999999999998e-06 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277820+0000", 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4669000000000002e-05 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.185 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.277826+0000", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2689999999999999e-06 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130039+0000", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85221334000000004 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130112+0000", 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3145000000000005e-05 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.130178+0000", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5712999999999999e-05 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.134087+0000", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003909432 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149796+0000", 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.015708895 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.187 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152515+0000", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.002718416 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152587+0000", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2083999999999996e-05 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152631+0000", 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4761000000000001e-05 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.188 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:175618 11.1 11:812e2f85:::benchmark_data_smithi043_179003_object45329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 137.769464419, 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.85387319399999995, 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.189 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 175618 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298885+0000", 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.190 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298886+0000", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0330000000000001e-06 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298891+0000", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298899+0000", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0129999999999993e-06 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.191 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298904+0000", 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9989999999999999e-06 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298961+0000", 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7679000000000001e-05 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:35.298971+0000", 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2359999999999993e-06 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.192 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132062+0000", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.83309193000000004 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132184+0000", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121699 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.132318+0000", 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000134268 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.138055+0000", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0057369680000000003 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.149845+0000", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.011789714999999999 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152668+0000", 2024-04-25T05:54:53.194 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0028230780000000001 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152723+0000", 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5148999999999997e-05 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:36.152759+0000", 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5185999999999998e-05 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:194360 11.1 11:803a97fb:::benchmark_data_smithi043_179003_object64071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 116.34564140499999, 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.085195066999999999, 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 194360 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722708+0000", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722709+0000", 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0690000000000001e-06 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.197 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722714+0000", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6859999999999997e-06 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722723+0000", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3850000000000008e-06 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722729+0000", 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.587e-06 2024-04-25T05:54:53.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722779+0000", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0232000000000003e-05 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722871+0000", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2007000000000002e-05 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.722998+0000", 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012624899999999999 2024-04-25T05:54:53.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.725628+0000", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026307570000000001 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.725649+0000", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.0752000000000001e-05 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807786+0000", 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.082136946000000002 2024-04-25T05:54:53.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807852+0000", 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6313e-05 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:52:56.807903+0000", 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1084000000000001e-05 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.201 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260044 11.1 11:8368061f:::benchmark_data_smithi043_179003_object129755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.252171+0000", 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 51.816178542999999, 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.098839641000000006, 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260044 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:53.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252171+0000", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252171+0000", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252172+0000", 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3200000000000003e-07 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252176+0000", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8680000000000001e-06 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252185+0000", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7080000000000003e-06 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252190+0000", 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5979999999999996e-06 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252242+0000", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1323999999999999e-05 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252361+0000", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00011904 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 0,3", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.252500+0000", 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000139114 2024-04-25T05:54:53.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.257702+0000", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0052024039999999999 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.258407+0000", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00070472600000000003 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.350871+0000", 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.092464376000000001 2024-04-25T05:54:53.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.350949+0000", 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7719999999999994e-05 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.351011+0000", 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1830999999999998e-05 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:53.207 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:53.303 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[27010]: 3.c scrub starts 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[27010]: 3.c scrub ok 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[27010]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[27010]: osdmap e603: 8 total, 8 up, 8 in 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[37492]: 3.c scrub starts 2024-04-25T05:54:53.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[37492]: 3.c scrub ok 2024-04-25T05:54:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[37492]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:53.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:53 smithi043 ceph-mon[37492]: osdmap e603: 8 total, 8 up, 8 in 2024-04-25T05:54:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-mon[30898]: 3.c scrub starts 2024-04-25T05:54:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-mon[30898]: 3.c scrub ok 2024-04-25T05:54:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-mon[30898]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-04-25T05:54:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/655477878' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:53.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-mon[30898]: osdmap e603: 8 total, 8 up, 8 in 2024-04-25T05:54:53.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-25T05:54:53.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:54:53 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:54:54.010 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:54.249 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-25T05:54:54.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:54 smithi060 ceph-mon[30898]: 1.7 deep-scrub starts 2024-04-25T05:54:54.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:54 smithi060 ceph-mon[30898]: 1.7 deep-scrub ok 2024-04-25T05:54:54.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:54 smithi060 ceph-mon[30898]: 3.12 scrub starts 2024-04-25T05:54:54.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:54 smithi060 ceph-mon[30898]: 3.12 scrub ok 2024-04-25T05:54:54.450 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:54 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[37492]: 1.7 deep-scrub starts 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[37492]: 1.7 deep-scrub ok 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[37492]: 3.12 scrub starts 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[37492]: 3.12 scrub ok 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:54.576 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[27010]: 1.7 deep-scrub starts 2024-04-25T05:54:54.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[27010]: 1.7 deep-scrub ok 2024-04-25T05:54:54.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[27010]: 3.12 scrub starts 2024-04-25T05:54:54.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[27010]: 3.12 scrub ok 2024-04-25T05:54:54.577 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:54 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-25T05:54:54.580 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:54.580 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:54.580 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-04-25T05:54:54.581 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:54.941 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-25T05:54:55.192 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:55.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: 3.d scrub starts 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: 3.d scrub ok 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: osdmap e604: 8 total, 8 up, 8 in 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: 5.1e scrub starts 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[27010]: 5.1e scrub ok 2024-04-25T05:54:55.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: 3.d scrub starts 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: 3.d scrub ok 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: osdmap e604: 8 total, 8 up, 8 in 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: 5.1e scrub starts 2024-04-25T05:54:55.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:55 smithi043 ceph-mon[37492]: 5.1e scrub ok 2024-04-25T05:54:55.604 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: 3.d scrub starts 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: 3.d scrub ok 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1929879911' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: osdmap e604: 8 total, 8 up, 8 in 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: 5.1e scrub starts 2024-04-25T05:54:55.727 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:55 smithi060 ceph-mon[30898]: 5.1e scrub ok 2024-04-25T05:54:56.096 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:56.097 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-04-25T05:54:56.097 INFO:teuthology.orchestra.run.smithi060.stdout: "complaint_time": 30, 2024-04-25T05:54:56.097 INFO:teuthology.orchestra.run.smithi060.stdout: "num_blocked_ops": 0 2024-04-25T05:54:56.097 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T05:54:56.404 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:56 smithi060 ceph-mon[30898]: 3.e scrub starts 2024-04-25T05:54:56.404 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:56 smithi060 ceph-mon[30898]: 3.e scrub ok 2024-04-25T05:54:56.404 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:56 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[27010]: 3.e scrub starts 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[27010]: 3.e scrub ok 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[37492]: 3.e scrub starts 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[37492]: 3.e scrub ok 2024-04-25T05:54:56.627 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:56 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:56.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-25T05:54:56.982 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-25T05:54:56.991 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T05:54:57.232 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "size": 20, 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 600, 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [ 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234262 9.13 9:c8a64bff:::benchmark_data_smithi043_129820_object112948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 524.06707073999996, 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.044775727, 2024-04-25T05:54:57.488 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234262 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.489 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419523+0000", 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419524+0000", 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.114e-06 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419529+0000", 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1320000000000002e-06 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.490 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419541+0000", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2517000000000001e-05 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419551+0000", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.55e-06 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419626+0000", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5401999999999994e-05 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.491 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.419639+0000", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3045e-05 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458481+0000", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0388411980000001 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458596+0000", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000114869 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.458847+0000", 2024-04-25T05:54:57.492 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000251838 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463718+0000", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0048708349999999996 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.463779+0000", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0671000000000002e-05 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464065+0000", 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00028592400000000001 2024-04-25T05:54:57.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464124+0000", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8913000000000003e-05 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464180+0000", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6740000000000002e-05 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464298+0000", 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000117979 2024-04-25T05:54:57.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:234266 9.13 9:cde636a1:::benchmark_data_smithi043_129820_object112952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 524.06330954800001, 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.043099716, 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.495 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 234266 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423284+0000", 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.496 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423285+0000", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.47e-07 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423290+0000", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9289999999999997e-06 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423301+0000", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1249e-05 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.497 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423309+0000", 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9359999999999999e-06 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423370+0000", 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.1496999999999998e-05 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:13.423380+0000", 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3589999999999997e-06 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.498 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.459982+0000", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.036602386 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460104+0000", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121814 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,3,6", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.460347+0000", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00024315799999999999 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.499 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464108+0000", 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.003761347 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.464306+0000", 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00019793999999999999 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466211+0000", 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001904836 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.500 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.501 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466254+0000", 2024-04-25T05:54:57.501 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2517e-05 2024-04-25T05:54:57.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.501 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.501 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466302+0000", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.8791000000000001e-05 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:14.466384+0000", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1110000000000001e-05 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237263 9.10 9:09fb57ed:::benchmark_data_smithi043_129820_object115949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:54:57.502 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.39040216599994, 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0207323559999999, 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237263 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096191+0000", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096192+0000", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8400000000000004e-07 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096195+0000", 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1750000000000001e-06 2024-04-25T05:54:57.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096202+0000", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8600000000000004e-06 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096206+0000", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2409999999999997e-06 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096257+0000", 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0695000000000002e-05 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.505 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.096262+0000", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5300000000000004e-06 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.113813+0000", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0175506910000001 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114018+0000", 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000205124 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.114193+0000", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000174981 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116636+0000", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0024427059999999998 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116671+0000", 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5380999999999999e-05 2024-04-25T05:54:57.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116825+0000", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000153993 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116873+0000", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7308000000000002e-05 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116924+0000", 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1087000000000001e-05 2024-04-25T05:54:57.508 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237265 9.13 9:ced551e7:::benchmark_data_smithi043_129820_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.38917871900003, 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.029123689, 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237265 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097415+0000", 2024-04-25T05:54:57.510 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3300000000000001e-07 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097420+0000", 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.0969999999999999e-06 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097429+0000", 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0380000000000005e-06 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.511 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097436+0000", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7449999999999995e-06 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097483+0000", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6343e-05 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.097489+0000", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.6830000000000002e-06 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.512 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115159+0000", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0176693729999999 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115269+0000", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109979 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115548+0000", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00027962100000000003 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118288+0000", 2024-04-25T05:54:57.513 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0027397879999999999 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120356+0000", 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0020682280000000001 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126451+0000", 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0060945569999999996 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.514 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126495+0000", 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3942e-05 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126538+0000", 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3562000000000001e-05 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237270 9.13 9:ccb5a786:::benchmark_data_smithi043_129820_object115956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.38581313899999, 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.025924235, 2024-04-25T05:54:57.515 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237270 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.516 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100780+0000", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100781+0000", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1200000000000002e-07 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100789+0000", 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7179999999999996e-06 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.517 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100798+0000", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8599999999999999e-06 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100804+0000", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0009999999999998e-06 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100867+0000", 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.3851999999999998e-05 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.518 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.100877+0000", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.1940000000000004e-06 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117090+0000", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016213622 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117202+0000", 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000111869 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117405+0000", 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00020259799999999999 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126115+0000", 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0087104629999999999 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.520 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126211+0000", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5789000000000001e-05 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126619+0000", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00040798499999999998 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126663+0000", 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3543e-05 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.521 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126705+0000", 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2029e-05 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237273 9.10 9:0de19dcd:::benchmark_data_smithi043_129820_object115959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.38404185599995, 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0209831819999999, 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.522 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237273 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102552+0000", 2024-04-25T05:54:57.523 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102553+0000", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.034e-06 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102559+0000", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.9089999999999997e-06 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102570+0000", 2024-04-25T05:54:57.524 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.112e-05 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102578+0000", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.2709999999999992e-06 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102617+0000", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.8695999999999997e-05 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.102625+0000", 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9929999999999995e-06 2024-04-25T05:54:57.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115511+0000", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0128864559999999 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115643+0000", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000131746 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.115827+0000", 2024-04-25T05:54:57.526 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00018383200000000001 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118880+0000", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0030529400000000001 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119838+0000", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00095799800000000005 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123433+0000", 2024-04-25T05:54:57.527 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0035950230000000001 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123491+0000", 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8242000000000002e-05 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123535+0000", 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3921999999999997e-05 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237293 9.13 9:cf509854:::benchmark_data_smithi043_129820_object115979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.37592303300005, 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.016106186, 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237293 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110670+0000", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110671+0000", 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9900000000000009e-07 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.530 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110676+0000", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6029999999999998e-06 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110685+0000", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6149999999999997e-06 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110691+0000", 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.8569999999999997e-06 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.531 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110748+0000", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6978999999999998e-05 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.110757+0000", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.7949999999999996e-06 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118323+0000", 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007566237 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118419+0000", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.5356e-05 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.118567+0000", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014814099999999999 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126167+0000", 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.007599522 2024-04-25T05:54:57.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126302+0000", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013507600000000001 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126712+0000", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00041020800000000001 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126745+0000", 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.3250999999999997e-05 2024-04-25T05:54:57.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126777+0000", 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1547000000000001e-05 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237310 9.13 9:cc1e0a4c:::benchmark_data_smithi043_129820_object115996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.366694983, 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.007006198, 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.535 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237310 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119898+0000", 2024-04-25T05:54:57.536 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119899+0000", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.9100000000000003e-07 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119903+0000", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5090000000000002e-06 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119912+0000", 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0440000000000001e-06 2024-04-25T05:54:57.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119917+0000", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.2959999999999998e-06 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119983+0000", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.5779999999999997e-05 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.119993+0000", 2024-04-25T05:54:57.538 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.8179999999999999e-06 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119292+0000", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99929933400000004 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119417+0000", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000125449 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119591+0000", 2024-04-25T05:54:57.539 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00017395799999999999 2024-04-25T05:54:57.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.540 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:57 smithi060 ceph-mon[30898]: 5.2 scrub starts 2024-04-25T05:54:57.540 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:57 smithi060 ceph-mon[30898]: 5.2 scrub ok 2024-04-25T05:54:57.540 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:57 smithi060 ceph-mon[30898]: pgmap v1141: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-04-25T05:54:57.540 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:57 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:57.540 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:57 smithi060 ceph-mon[30898]: osdmap e605: 8 total, 8 up, 8 in 2024-04-25T05:54:57.541 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126261+0000", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0066693810000000003 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126390+0000", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000129164 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126784+0000", 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00039376400000000001 2024-04-25T05:54:57.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126863+0000", 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.8963999999999998e-05 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126905+0000", 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2045999999999997e-05 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.543 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237325 9.13 9:cbb3ea78:::benchmark_data_smithi043_129820_object116011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.35709817099996, 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99763348100000004, 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237325 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.544 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129495+0000", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129496+0000", 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4600000000000004e-07 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.545 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129500+0000", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1790000000000001e-06 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129511+0000", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0611000000000001e-05 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129518+0000", 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6020000000000002e-06 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129565+0000", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6158000000000001e-05 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.129573+0000", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.6240000000000008e-06 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120395+0000", 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.99082194099999998 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120502+0000", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00010669 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,6", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.120678+0000", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000175976 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126345+0000", 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.005667025 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.548 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.126911+0000", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00056641199999999997 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127019+0000", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107938 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127082+0000", 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.2848999999999994e-05 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.127129+0000", 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6730000000000002e-05 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.25487.0:237346 9.10 9:0cf8b6b3:::benchmark_data_smithi043_129820_object116032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 521.34530922199997, 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.98237561100000004, 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.550 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.25487", 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/4230058272", 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 237346 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.551 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141284+0000", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141285+0000", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3699999999999999e-07 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141290+0000", 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6820000000000002e-06 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.552 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141300+0000", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.9890000000000004e-06 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141309+0000", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.9840000000000007e-06 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141364+0000", 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.4886999999999998e-05 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.553 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for peered", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:16.141373+0000", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.2369999999999998e-06 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.116981+0000", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.97560839600000004 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117101+0000", 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000119911 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 4,6", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.117270+0000", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016895300000000001 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.119880+0000", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0026103020000000001 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.121458+0000", 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001577422 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123574+0000", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021161970000000002 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123625+0000", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.0562000000000001e-05 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:46:17.123660+0000", 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.5151999999999998e-05 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.556 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260085 11.4 11:282765cf:::benchmark_data_smithi043_179003_object129796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.361138+0000", 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.125455174999999, 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.28115671399999997, 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260085 2024-04-25T05:54:57.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361138+0000", 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361138+0000", 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.558 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361140+0000", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3090000000000001e-06 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361144+0000", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4479999999999996e-06 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361153+0000", 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.3430000000000002e-06 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.361161+0000", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4359999999999996e-06 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.369888+0000", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0087269089999999997 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370034+0000", 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014591199999999999 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370178+0000", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014485499999999999 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.374995+0000", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.004816051 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.376745+0000", 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001750875 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642205+0000", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26545995700000002 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642242+0000", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.6199999999999999e-05 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642295+0000", 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3418999999999998e-05 2024-04-25T05:54:57.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260089 11.4 11:28ca0e53:::benchmark_data_smithi043_179003_object129800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.363421+0000", 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.123172883000002, 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.28010668300000002, 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260089 2024-04-25T05:54:57.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363421+0000", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363421+0000", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363422+0000", 2024-04-25T05:54:57.564 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.207e-06 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363427+0000", 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7139999999999998e-06 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363440+0000", 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.2707e-05 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.565 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.363448+0000", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.0979999999999994e-06 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.372431+0000", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0089830020000000003 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.372552+0000", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120821 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:54:57.566 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.372673+0000", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121178 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375002+0000", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0023286600000000002 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.376778+0000", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00177634 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.567 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.643433+0000", 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.266654791 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.643484+0000", 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.1377e-05 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.643527+0000", 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.2787999999999998e-05 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.568 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260095 11.f 11:f30005ba:::benchmark_data_smithi043_179003_object129806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.370096+0000", 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.116496980000001, 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26479023000000002, 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260095 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.569 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370096+0000", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370096+0000", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370097+0000", 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.4200000000000005e-07 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.570 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370101+0000", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.1359999999999999e-06 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370109+0000", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.9419999999999995e-06 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370115+0000", 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5090000000000001e-06 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370153+0000", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7975000000000001e-05 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370226+0000", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.3034999999999994e-05 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.370334+0000", 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000107879 2024-04-25T05:54:57.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.377675+0000", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.007340971 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424328+0000", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.046653739999999999 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.634803+0000", 2024-04-25T05:54:57.573 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.21047485499999999 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.634847+0000", 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.3544999999999997e-05 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.634887+0000", 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.9801000000000001e-05 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260098 11.f 11:f76527f5:::benchmark_data_smithi043_179003_object129809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.371223+0000", 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.115370386000002, 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27146442300000001, 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260098 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.575 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371223+0000", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371223+0000", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371224+0000", 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.0800000000000003e-07 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371232+0000", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.824e-06 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371245+0000", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3077e-05 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371252+0000", 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.4919999999999996e-06 2024-04-25T05:54:57.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.372965+0000", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001713015 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.373085+0000", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000120059 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.373206+0000", 2024-04-25T05:54:57.578 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000121051 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.377682+0000", 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0044753689999999999 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424389+0000", 2024-04-25T05:54:57.579 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.046707566999999998 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642628+0000", 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.21823847099999999 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642658+0000", 2024-04-25T05:54:57.580 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9989999999999999e-05 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.642687+0000", 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9600000000000001e-05 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260100 11.f 11:f7d22ae4:::benchmark_data_smithi043_179003_object129811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.371604+0000", 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.114989213000001, 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.277836737, 2024-04-25T05:54:57.581 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260100 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371604+0000", 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371604+0000", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371605+0000", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.1200000000000002e-07 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371609+0000", 2024-04-25T05:54:57.583 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.425e-06 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371623+0000", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.3574000000000001e-05 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.371631+0000", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.6699999999999994e-06 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375320+0000", 2024-04-25T05:54:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0036890299999999998 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375456+0000", 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00013648299999999999 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375597+0000", 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00014135299999999999 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.585 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.379913+0000", 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0043157229999999996 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424441+0000", 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.044527942000000001 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649352+0000", 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.22491092900000001 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.586 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649412+0000", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 6.0421999999999998e-05 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649441+0000", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.8473999999999999e-05 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260104 11.f 11:f77b3f46:::benchmark_data_smithi043_179003_object129815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.375578+0000", 2024-04-25T05:54:57.587 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.111014959000002, 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.27396528599999997, 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260104 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375578+0000", 2024-04-25T05:54:57.588 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375578+0000", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375580+0000", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.198e-06 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375584+0000", 2024-04-25T05:54:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.7870000000000001e-06 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375596+0000", 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.1119e-05 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.375603+0000", 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7649999999999993e-06 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.590 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.377708+0000", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0021049390000000001 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.377836+0000", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00012726999999999999 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 1,2", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.378010+0000", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000174649 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.591 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.379969+0000", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.001958787 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.424495+0000", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.044526099999999999 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649474+0000", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.224979122 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.592 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649506+0000", 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.1883e-05 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.649544+0000", 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 3.7666999999999997e-05 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.16413.0:260109 11.4 11:209e5b67:::benchmark_data_smithi043_179003_object129820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:01.381026+0000", 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 56.105567776999997, 2024-04-25T05:54:57.593 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26535209100000001, 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.16413", 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/3846254651", 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 260109 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381026+0000", 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381026+0000", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381027+0000", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6200000000000006e-07 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381031+0000", 2024-04-25T05:54:57.595 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4519999999999999e-06 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381041+0000", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.4460000000000007e-06 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381046+0000", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.3630000000000002e-06 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381102+0000", 2024-04-25T05:54:57.596 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.5951e-05 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381198+0000", 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 9.6405999999999998e-05 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "waiting for subops from 2,6", 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.381275+0000", 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7213999999999996e-05 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "op_commit", 2024-04-25T05:54:57.597 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.383481+0000", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.0022057219999999998 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.383698+0000", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00021676900000000001 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "sub_op_commit_rec", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.646277+0000", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.26257860399999999 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.598 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.646333+0000", 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.6397999999999998e-05 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:01.646378+0000", 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4804e-05 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_repop(osd.0.0:87755 4.16 e602/497 4:6849fd06:gc::gc.17:head v 602'34, mlcod=581'32)", 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:50.265842+0000", 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 7.2207512850000004, 2024-04-25T05:54:57.599 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00048873799999999995, 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265842+0000", 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265842+0000", 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.600 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265843+0000", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.8500000000000001e-07 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265851+0000", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.636e-06 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265860+0000", 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.8720000000000008e-06 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.601 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265865+0000", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.9030000000000003e-06 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265912+0000", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.6486999999999997e-05 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.265926+0000", 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.4241e-05 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.602 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "commit_sent", 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.266302+0000", 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000375788 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.266331+0000", 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.9226000000000001e-05 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.24397.0:1237 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e601)", 2024-04-25T05:54:57.603 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:50.362878+0000", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 7.1237149630000003, 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00036454899999999998, 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.24397", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.60:0/1934853514", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 1237 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362878+0000", 2024-04-25T05:54:57.604 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362878+0000", 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362881+0000", 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.683e-06 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.605 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362892+0000", 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 1.0976e-05 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362921+0000", 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.904e-05 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.362945+0000", 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 2.3513e-05 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.606 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.363108+0000", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.00016330399999999999 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.363185+0000", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.7488000000000001e-05 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:50.363243+0000", 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 5.7544999999999999e-05 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.607 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "description": "osd_op(client.14484.0:4784 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94445971027456 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e602)", 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "initiated_at": "2024-04-25T05:54:53.987916+0000", 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "age": 3.4986769789999999, 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000355867, 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "continuous": false, 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "type_data": { 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "flag_point": "started", 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "client_info": { 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "client": "client.14484", 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "client_addr": "172.21.15.43:0/312817651", 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: "tid": 4784 2024-04-25T05:54:57.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "events": [ 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "initiated", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987916+0000", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "header_read", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987916+0000", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "throttled", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987917+0000", 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.5700000000000002e-07 2024-04-25T05:54:57.609 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "all_read", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987922+0000", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 4.4839999999999998e-06 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "dispatched", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987929+0000", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 7.2080000000000003e-06 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "queued_for_pg", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.987937+0000", 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 8.1880000000000002e-06 2024-04-25T05:54:57.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "reached_pg", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.988047+0000", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000109789 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "started", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.988149+0000", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000102154 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "event": "done", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "time": "2024-04-25T05:54:53.988272+0000", 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: "duration": 0.000123287 2024-04-25T05:54:57.611 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.612 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.612 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.612 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-04-25T05:54:57.612 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-04-25T05:54:57.612 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-04-25T05:54:57.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[37492]: 5.2 scrub starts 2024-04-25T05:54:57.612 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[37492]: 5.2 scrub ok 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[37492]: pgmap v1141: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[37492]: osdmap e605: 8 total, 8 up, 8 in 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[27010]: 5.2 scrub starts 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[27010]: 5.2 scrub ok 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[27010]: pgmap v1141: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1583068727' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:57.613 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:57 smithi043 ceph-mon[27010]: osdmap e605: 8 total, 8 up, 8 in 2024-04-25T05:54:58.267 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-25T05:54:58.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:58 smithi060 ceph-mon[30898]: 3.11 scrub starts 2024-04-25T05:54:58.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:58 smithi060 ceph-mon[30898]: 3.11 scrub ok 2024-04-25T05:54:58.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:58 smithi060 ceph-mon[30898]: 6.1f scrub starts 2024-04-25T05:54:58.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:58 smithi060 ceph-mon[30898]: 6.1f scrub ok 2024-04-25T05:54:58.448 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:58 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:58.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[27010]: 3.11 scrub starts 2024-04-25T05:54:58.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[27010]: 3.11 scrub ok 2024-04-25T05:54:58.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[27010]: 6.1f scrub starts 2024-04-25T05:54:58.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[27010]: 6.1f scrub ok 2024-04-25T05:54:58.572 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:58.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[37492]: 3.11 scrub starts 2024-04-25T05:54:58.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[37492]: 3.11 scrub ok 2024-04-25T05:54:58.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[37492]: 6.1f scrub starts 2024-04-25T05:54:58.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[37492]: 6.1f scrub ok 2024-04-25T05:54:58.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:58 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-25T05:54:59.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 5.10 scrub starts 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 5.10 scrub ok 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: osdmap e606: 8 total, 8 up, 8 in 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 3.17 scrub starts 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 3.9 scrub starts 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 3.17 scrub ok 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[27010]: 3.9 scrub ok 2024-04-25T05:54:59.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 5.10 scrub starts 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 5.10 scrub ok 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: osdmap e606: 8 total, 8 up, 8 in 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 3.17 scrub starts 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 3.9 scrub starts 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 3.17 scrub ok 2024-04-25T05:54:59.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-mon[37492]: 3.9 scrub ok 2024-04-25T05:54:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 5.10 scrub starts 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 5.10 scrub ok 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: pgmap v1143: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3111832352' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: osdmap e606: 8 total, 8 up, 8 in 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 3.17 scrub starts 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 3.9 scrub starts 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 3.17 scrub ok 2024-04-25T05:54:59.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:54:59 smithi060 ceph-mon[30898]: 3.9 scrub ok 2024-04-25T05:54:59.831 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:54:59 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:54:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:55:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[27010]: 3.13 scrub starts 2024-04-25T05:55:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[27010]: 3.13 scrub ok 2024-04-25T05:55:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[27010]: 6.11 scrub starts 2024-04-25T05:55:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[27010]: 6.11 scrub ok 2024-04-25T05:55:00.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[37492]: 3.13 scrub starts 2024-04-25T05:55:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[37492]: 3.13 scrub ok 2024-04-25T05:55:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[37492]: 6.11 scrub starts 2024-04-25T05:55:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[37492]: 6.11 scrub ok 2024-04-25T05:55:00.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:00 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:00 smithi060 ceph-mon[30898]: 3.13 scrub starts 2024-04-25T05:55:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:00 smithi060 ceph-mon[30898]: 3.13 scrub ok 2024-04-25T05:55:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:00 smithi060 ceph-mon[30898]: 6.11 scrub starts 2024-04-25T05:55:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:00 smithi060 ceph-mon[30898]: 6.11 scrub ok 2024-04-25T05:55:00.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:00 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:01.142 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 3.a deep-scrub starts 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 3.a deep-scrub ok 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 6.8 scrub starts 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 6.8 scrub ok 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 3.1c scrub starts 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 3.1c scrub ok 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.19 scrub starts 2024-04-25T05:55:01.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.19 scrub ok 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.d deep-scrub starts 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.d deep-scrub ok 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.b scrub starts 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 4.b scrub ok 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 5.d scrub starts 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 5.d scrub ok 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: 6.3 scrub starts 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[27010]: osdmap e607: 8 total, 8 up, 8 in 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 3.a deep-scrub starts 2024-04-25T05:55:01.462 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 3.a deep-scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 6.8 scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 6.8 scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 3.1c scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 3.1c scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.19 scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.19 scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.d deep-scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.d deep-scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.b scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 4.b scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 5.d scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 5.d scrub ok 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: 6.3 scrub starts 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:01.463 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:01 smithi043 ceph-mon[37492]: osdmap e607: 8 total, 8 up, 8 in 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 3.a deep-scrub starts 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 3.a deep-scrub ok 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 6.8 scrub starts 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 6.8 scrub ok 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 3.1c scrub starts 2024-04-25T05:55:01.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 3.1c scrub ok 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.19 scrub starts 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.19 scrub ok 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.d deep-scrub starts 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.d deep-scrub ok 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.b scrub starts 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 4.b scrub ok 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 5.d scrub starts 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 5.d scrub ok 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: 6.3 scrub starts 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3819256879' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:01.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:01 smithi060 ceph-mon[30898]: osdmap e607: 8 total, 8 up, 8 in 2024-04-25T05:55:01.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-25T05:55:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:02.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 6.15 scrub starts 2024-04-25T05:55:02.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 6.d scrub starts 2024-04-25T05:55:02.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 6.15 scrub ok 2024-04-25T05:55:02.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 6.d scrub ok 2024-04-25T05:55:02.270 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 6.3 scrub ok 2024-04-25T05:55:02.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 6.15 scrub starts 2024-04-25T05:55:02.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 6.d scrub starts 2024-04-25T05:55:02.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 6.15 scrub ok 2024-04-25T05:55:02.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 6.d scrub ok 2024-04-25T05:55:02.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 6.3 scrub ok 2024-04-25T05:55:02.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 4.f deep-scrub starts 2024-04-25T05:55:02.532 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[37492]: 4.f deep-scrub ok 2024-04-25T05:55:02.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 4.f deep-scrub starts 2024-04-25T05:55:02.532 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:02 smithi043 ceph-mon[27010]: 4.f deep-scrub ok 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 6.15 scrub starts 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 6.d scrub starts 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 6.15 scrub ok 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 6.d scrub ok 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 6.3 scrub ok 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 4.f deep-scrub starts 2024-04-25T05:55:02.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:02 smithi060 ceph-mon[30898]: 4.f deep-scrub ok 2024-04-25T05:55:03.278 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-25T05:55:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[27010]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[27010]: osdmap e608: 8 total, 8 up, 8 in 2024-04-25T05:55:03.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[37492]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[37492]: osdmap e608: 8 total, 8 up, 8 in 2024-04-25T05:55:03.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:03 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-mon[30898]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-mon[30898]: osdmap e608: 8 total, 8 up, 8 in 2024-04-25T05:55:03.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-25T05:55:04.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-25T05:55:04.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:55:03 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:55:04.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 4.15 scrub starts 2024-04-25T05:55:04.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 4.15 scrub ok 2024-04-25T05:55:04.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 5.13 deep-scrub starts 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 5.13 deep-scrub ok 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: osdmap e609: 8 total, 8 up, 8 in 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 3.1f scrub starts 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: 3.1f scrub ok 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 4.15 scrub starts 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 4.15 scrub ok 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 5.13 deep-scrub starts 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 5.13 deep-scrub ok 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:04.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: osdmap e609: 8 total, 8 up, 8 in 2024-04-25T05:55:04.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 3.1f scrub starts 2024-04-25T05:55:04.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: 3.1f scrub ok 2024-04-25T05:55:04.393 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:04 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:04.399 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 4.15 scrub starts 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 4.15 scrub ok 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 5.13 deep-scrub starts 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 5.13 deep-scrub ok 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2172493979' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: osdmap e609: 8 total, 8 up, 8 in 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 3.1f scrub starts 2024-04-25T05:55:04.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: 3.1f scrub ok 2024-04-25T05:55:04.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:04 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: 3.f scrub starts 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: 3.f scrub ok 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: osdmap e610: 8 total, 8 up, 8 in 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/4205280949' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: 3.f scrub starts 2024-04-25T05:55:05.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: 3.f scrub ok 2024-04-25T05:55:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:55:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: osdmap e610: 8 total, 8 up, 8 in 2024-04-25T05:55:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/4205280949' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:05.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:05 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: 3.f scrub starts 2024-04-25T05:55:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: 3.f scrub ok 2024-04-25T05:55:05.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:05.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-25T05:55:05.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: osdmap e610: 8 total, 8 up, 8 in 2024-04-25T05:55:05.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/4205280949' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:05.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:05 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:06.149 INFO:teuthology.orchestra.run.smithi043.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.4 scrub starts 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.4 scrub ok 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 3.6 scrub starts 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 3.6 scrub ok 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.12 scrub starts 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.12 scrub ok 2024-04-25T05:55:06.528 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.a scrub starts 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: 5.a scrub ok 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[37492]: osdmap e611: 8 total, 8 up, 8 in 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.4 scrub starts 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.4 scrub ok 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 3.6 scrub starts 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 3.6 scrub ok 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.12 scrub starts 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.12 scrub ok 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.a scrub starts 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: 5.a scrub ok 2024-04-25T05:55:06.529 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:06.530 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:06 smithi043 ceph-mon[27010]: osdmap e611: 8 total, 8 up, 8 in 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.4 scrub starts 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.4 scrub ok 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 3.6 scrub starts 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 3.6 scrub ok 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.12 scrub starts 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.12 scrub ok 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.a scrub starts 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: 5.a scrub ok 2024-04-25T05:55:06.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:06.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:06 smithi060 ceph-mon[30898]: osdmap e611: 8 total, 8 up, 8 in 2024-04-25T05:55:06.989 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-25T05:55:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:07.546 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: 6.12 deep-scrub starts 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: 6.12 deep-scrub ok 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: 6.b scrub starts 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: 6.b scrub ok 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[27010]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: 6.12 deep-scrub starts 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: 6.12 deep-scrub ok 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: 6.b scrub starts 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: 6.b scrub ok 2024-04-25T05:55:07.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:07.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:07.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:07 smithi043 ceph-mon[37492]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: 6.12 deep-scrub starts 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: 6.12 deep-scrub ok 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: 6.b scrub starts 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: 6.b scrub ok 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:07.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:07.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:07 smithi060 ceph-mon[30898]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:08.297 INFO:teuthology.orchestra.run.smithi043.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-25T05:55:08.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[27010]: 5.e scrub starts 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[27010]: 5.e scrub ok 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1397380894' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[37492]: 5.e scrub starts 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[37492]: 5.e scrub ok 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1397380894' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:08.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:08 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:08.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:08 smithi060 ceph-mon[30898]: 5.e scrub starts 2024-04-25T05:55:08.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:08 smithi060 ceph-mon[30898]: 5.e scrub ok 2024-04-25T05:55:08.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:08 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1397380894' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:08.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:08 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-25T05:55:09.137 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-25T05:55:09.138 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-25T05:55:09.138 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-25T05:55:09.138 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-25T05:55:09.138 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T05:55:09.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T05:55:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 5.f scrub starts 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 5.f scrub ok 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 5.14 scrub starts 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 5.14 scrub ok 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: osdmap e612: 8 total, 8 up, 8 in 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 1.6 deep-scrub starts 2024-04-25T05:55:09.499 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:09 smithi060 ceph-mon[30898]: 1.6 deep-scrub ok 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 5.f scrub starts 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 5.f scrub ok 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 5.14 scrub starts 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 5.14 scrub ok 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: osdmap e612: 8 total, 8 up, 8 in 2024-04-25T05:55:09.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 1.6 deep-scrub starts 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[37492]: 1.6 deep-scrub ok 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 5.f scrub starts 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 5.f scrub ok 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 5.14 scrub starts 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 5.14 scrub ok 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: pgmap v1154: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: osdmap e612: 8 total, 8 up, 8 in 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 1.6 deep-scrub starts 2024-04-25T05:55:09.574 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-mon[27010]: 1.6 deep-scrub ok 2024-04-25T05:55:09.876 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:09 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:55:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:55:10.216 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:55:10.216 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":612,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","created":"2024-04-25T05:28:34.192515+0000","modified":"2024-04-25T05:55:08.286169+0000","last_up_change":"2024-04-25T05:33:46.700068+0000","last_in_change":"2024-04-25T05:51:39.176833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T05:31:44.863613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":551,"last_epoch_started":550,"last_epoch_clean":550,"source_version":"79'128","target_version":"79'128"},"last_change":"587","last_force_op_resend":"0","last_force_op_resend_prenautilus":"551","last_force_op_resend_preluminous":"549","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-25T05:33:56.045664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-25T05:33:57.225173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-25T05:33:58.748103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-25T05:34:00.700767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-25T05:34:02.767681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef370ce4-85d8-41ed-9655-2e2eaab3b48e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":541,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":3557191248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3557191248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3557191248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":3557191248}]},"public_addr":"172.21.15.43:6801/3557191248","cluster_addr":"172.21.15.43:6802/3557191248","heartbeat_back_addr":"172.21.15.43:6804/3557191248","heartbeat_front_addr":"172.21.15.43:6803/3557191248","state":["exists","up"]},{"osd":1,"uuid":"ed504d08-dc59-4584-bd5f-8059e02a4161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":551,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":235229295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":235229295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":235229295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":235229295}]},"public_addr":"172.21.15.43:6805/235229295","cluster_addr":"172.21.15.43:6806/235229295","heartbeat_back_addr":"172.21.15.43:6808/235229295","heartbeat_front_addr":"172.21.15.43:6807/235229295","state":["exists","up"]},{"osd":2,"uuid":"11ff9a46-8686-4b36-a530-5cc1dca1f088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":540,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":3416612512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3416612512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3416612512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3416612512}]},"public_addr":"172.21.15.43:6809/3416612512","cluster_addr":"172.21.15.43:6810/3416612512","heartbeat_back_addr":"172.21.15.43:6812/3416612512","heartbeat_front_addr":"172.21.15.43:6811/3416612512","state":["exists","up"]},{"osd":3,"uuid":"f3314c45-11c4-46ee-be0b-6f639faa7da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":557,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":3391766005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3391766005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3391766005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6815","nonce":3391766005}]},"public_addr":"172.21.15.43:6813/3391766005","cluster_addr":"172.21.15.43:6814/3391766005","heartbeat_back_addr":"172.21.15.43:6816/3391766005","heartbeat_front_addr":"172.21.15.43:6815/3391766005","state":["exists","up"]},{"osd":4,"uuid":"8c5337b2-94d1-4784-9321-db771de75f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1310286745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6801","nonce":1310286745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1310286745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1310286745}]},"public_addr":"172.21.15.60:6800/1310286745","cluster_addr":"172.21.15.60:6801/1310286745","heartbeat_back_addr":"172.21.15.60:6803/1310286745","heartbeat_front_addr":"172.21.15.60:6802/1310286745","state":["exists","up"]},{"osd":5,"uuid":"fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":527,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3586186657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3586186657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":3586186657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3586186657}]},"public_addr":"172.21.15.60:6804/3586186657","cluster_addr":"172.21.15.60:6805/3586186657","heartbeat_back_addr":"172.21.15.60:6807/3586186657","heartbeat_front_addr":"172.21.15.60:6806/3586186657","state":["exists","up"]},{"osd":6,"uuid":"5b946188-0dd1-4e68-bcfc-a3daa342edaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":539,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1675137598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":1675137598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1675137598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1675137598}]},"public_addr":"172.21.15.60:6808/1675137598","cluster_addr":"172.21.15.60:6809/1675137598","heartbeat_back_addr":"172.21.15.60:6811/1675137598","heartbeat_front_addr":"172.21.15.60:6810/1675137598","state":["exists","up"]},{"osd":7,"uuid":"dec84bb8-9d2b-469f-9641-62ff40b9b9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":531,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3982012700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":3982012700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":3982012700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3982012700}]},"public_addr":"172.21.15.60:6812/3982012700","cluster_addr":"172.21.15.60:6813/3982012700","heartbeat_back_addr":"172.21.15.60:6815/3982012700","heartbeat_front_addr":"172.21.15.60:6814/3982012700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:53:56.979708+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:53:42.115456+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:36.581834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:53:50.801428+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:06.379919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:06.092788+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:03.968238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:02.143279+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.1c","osds":[6,0,5]},{"pgid":"6.7","osds":[1,2,4]}],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":7}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.0","mappings":[{"from":1,"to":6}]},{"pgid":"5.1a","mappings":[{"from":4,"to":3}]},{"pgid":"6.3","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/3423182021":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/2082036420":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/3958950586":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/3953560108":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/4077364595":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/830746728":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/786479021":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/3735093998":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/1835545567":"2024-04-26T05:29:02.328794+0000","172.21.15.43:6800/149142905":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/1408327650":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/2239747746":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/4084484879":"2024-04-26T05:29:30.781325+0000","172.21.15.43:0/1247294293":"2024-04-26T05:29:30.781325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T05:55:10.571 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[27010]: 4.11 scrub starts 2024-04-25T05:55:10.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[27010]: 4.11 scrub ok 2024-04-25T05:55:10.573 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3530481446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:10.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[37492]: 4.11 scrub starts 2024-04-25T05:55:10.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[37492]: 4.11 scrub ok 2024-04-25T05:55:10.573 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:10 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3530481446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:10 smithi060 ceph-mon[30898]: 4.11 scrub starts 2024-04-25T05:55:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:10 smithi060 ceph-mon[30898]: 4.11 scrub ok 2024-04-25T05:55:10.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:10 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3530481446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:10.920 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T05:55:10.921 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T05:55:11.174 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:11.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[37492]: 3.8 scrub starts 2024-04-25T05:55:11.584 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[37492]: 3.8 scrub ok 2024-04-25T05:55:11.585 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[37492]: pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:11.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[27010]: 3.8 scrub starts 2024-04-25T05:55:11.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[27010]: 3.8 scrub ok 2024-04-25T05:55:11.585 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:11 smithi043 ceph-mon[27010]: pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:11 smithi060 ceph-mon[30898]: 3.8 scrub starts 2024-04-25T05:55:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:11 smithi060 ceph-mon[30898]: 3.8 scrub ok 2024-04-25T05:55:11.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:11 smithi060 ceph-mon[30898]: pgmap v1156: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:11.909 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:55:11.910 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":613,"fsid":"84884b5e-02c4-11ef-bc93-c7b262605968","created":"2024-04-25T05:28:34.192515+0000","modified":"2024-04-25T05:55:11.293602+0000","last_up_change":"2024-04-25T05:33:46.700068+0000","last_in_change":"2024-04-25T05:51:39.176833+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T05:31:44.863613+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":551,"last_epoch_started":550,"last_epoch_clean":550,"source_version":"79'128","target_version":"79'128"},"last_change":"587","last_force_op_resend":"0","last_force_op_resend_prenautilus":"551","last_force_op_resend_preluminous":"549","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-25T05:33:56.045664+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-25T05:33:57.225173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-25T05:33:58.748103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-25T05:34:00.700767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-25T05:34:02.767681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ef370ce4-85d8-41ed-9655-2e2eaab3b48e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":541,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":3557191248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":3557191248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":3557191248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":3557191248}]},"public_addr":"172.21.15.43:6801/3557191248","cluster_addr":"172.21.15.43:6802/3557191248","heartbeat_back_addr":"172.21.15.43:6804/3557191248","heartbeat_front_addr":"172.21.15.43:6803/3557191248","state":["exists","up"]},{"osd":1,"uuid":"ed504d08-dc59-4584-bd5f-8059e02a4161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":551,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":235229295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":235229295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":235229295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":235229295}]},"public_addr":"172.21.15.43:6805/235229295","cluster_addr":"172.21.15.43:6806/235229295","heartbeat_back_addr":"172.21.15.43:6808/235229295","heartbeat_front_addr":"172.21.15.43:6807/235229295","state":["exists","up"]},{"osd":2,"uuid":"11ff9a46-8686-4b36-a530-5cc1dca1f088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":540,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6809","nonce":3416612512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3416612512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3416612512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":3416612512}]},"public_addr":"172.21.15.43:6809/3416612512","cluster_addr":"172.21.15.43:6810/3416612512","heartbeat_back_addr":"172.21.15.43:6812/3416612512","heartbeat_front_addr":"172.21.15.43:6811/3416612512","state":["exists","up"]},{"osd":3,"uuid":"f3314c45-11c4-46ee-be0b-6f639faa7da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":557,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6813","nonce":3391766005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3391766005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3391766005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6815","nonce":3391766005}]},"public_addr":"172.21.15.43:6813/3391766005","cluster_addr":"172.21.15.43:6814/3391766005","heartbeat_back_addr":"172.21.15.43:6816/3391766005","heartbeat_front_addr":"172.21.15.43:6815/3391766005","state":["exists","up"]},{"osd":4,"uuid":"8c5337b2-94d1-4784-9321-db771de75f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":543,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1310286745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6801","nonce":1310286745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":1310286745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":1310286745}]},"public_addr":"172.21.15.60:6800/1310286745","cluster_addr":"172.21.15.60:6801/1310286745","heartbeat_back_addr":"172.21.15.60:6803/1310286745","heartbeat_front_addr":"172.21.15.60:6802/1310286745","state":["exists","up"]},{"osd":5,"uuid":"fd119ddc-e7bf-4b70-af8c-2e87f07e1ba2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":527,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":3586186657}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":3586186657}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":3586186657}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":3586186657}]},"public_addr":"172.21.15.60:6804/3586186657","cluster_addr":"172.21.15.60:6805/3586186657","heartbeat_back_addr":"172.21.15.60:6807/3586186657","heartbeat_front_addr":"172.21.15.60:6806/3586186657","state":["exists","up"]},{"osd":6,"uuid":"5b946188-0dd1-4e68-bcfc-a3daa342edaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":539,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":1675137598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":1675137598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":1675137598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1675137598}]},"public_addr":"172.21.15.60:6808/1675137598","cluster_addr":"172.21.15.60:6809/1675137598","heartbeat_back_addr":"172.21.15.60:6811/1675137598","heartbeat_front_addr":"172.21.15.60:6810/1675137598","state":["exists","up"]},{"osd":7,"uuid":"dec84bb8-9d2b-469f-9641-62ff40b9b9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":531,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":3982012700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":3982012700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":3982012700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":3982012700}]},"public_addr":"172.21.15.60:6812/3982012700","cluster_addr":"172.21.15.60:6813/3982012700","heartbeat_back_addr":"172.21.15.60:6815/3982012700","heartbeat_front_addr":"172.21.15.60:6814/3982012700","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:10.708878+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:11.177246+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:36.581834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:53:50.801428+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:06.379919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:06.092788+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:54:03.968238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T05:55:02.143279+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.1c","osds":[6,0,5]},{"pgid":"6.7","osds":[1,2,4]}],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":7}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]},{"pgid":"5.0","mappings":[{"from":1,"to":6}]},{"pgid":"5.1a","mappings":[{"from":4,"to":3}]},{"pgid":"6.3","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/3423182021":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/2082036420":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/3958950586":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/3953560108":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/4077364595":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/830746728":"2024-04-26T05:29:30.781325+0000","172.21.15.43:6800/786479021":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/3735093998":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/1835545567":"2024-04-26T05:29:02.328794+0000","172.21.15.43:6800/149142905":"2024-04-26T05:34:31.636703+0000","172.21.15.43:0/1408327650":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/2239747746":"2024-04-26T05:29:02.328794+0000","172.21.15.43:0/4084484879":"2024-04-26T05:29:30.781325+0000","172.21.15.43:0/1247294293":"2024-04-26T05:29:30.781325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T05:55:12.249 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:55:11 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=cleanup t=2024-04-25T05:55:11.775851135Z level=info msg="Completed cleanup jobs" duration=7.2242ms 2024-04-25T05:55:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: 6.19 scrub starts 2024-04-25T05:55:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:12.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: 6.19 scrub ok 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: osdmap e613: 8 total, 8 up, 8 in 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/937869351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[27010]: 6.e scrub starts 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: 6.19 scrub starts 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:12.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: 6.19 scrub ok 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: osdmap e613: 8 total, 8 up, 8 in 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/937869351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:12.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:12 smithi043 ceph-mon[37492]: 6.e scrub starts 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: 6.19 scrub starts 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: 6.19 scrub ok 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: osdmap e613: 8 total, 8 up, 8 in 2024-04-25T05:55:12.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/937869351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T05:55:12.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:12 smithi060 ceph-mon[30898]: 6.e scrub starts 2024-04-25T05:55:12.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T05:55:12.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T05:55:12.802 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T05:55:12.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T05:55:12.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T05:55:12.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T05:55:12.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T05:55:12.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T05:55:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.357 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.364 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.365 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.389 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:13.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[27010]: 5.1b scrub starts 2024-04-25T05:55:13.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[27010]: 5.1b scrub ok 2024-04-25T05:55:13.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[27010]: pgmap v1158: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:13.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[27010]: 6.e scrub ok 2024-04-25T05:55:13.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[37492]: 5.1b scrub starts 2024-04-25T05:55:13.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[37492]: 5.1b scrub ok 2024-04-25T05:55:13.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[37492]: pgmap v1158: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:13.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:13 smithi043 ceph-mon[37492]: 6.e scrub ok 2024-04-25T05:55:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:13 smithi060 ceph-mon[30898]: 5.1b scrub starts 2024-04-25T05:55:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:13 smithi060 ceph-mon[30898]: 5.1b scrub ok 2024-04-25T05:55:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:13 smithi060 ceph-mon[30898]: pgmap v1158: 142 pgs: 142 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:13.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:13 smithi060 ceph-mon[30898]: 6.e scrub ok 2024-04-25T05:55:14.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:55:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:55:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[27010]: pgmap v1159: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[27010]: 3.3 scrub starts 2024-04-25T05:55:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[27010]: 3.3 scrub ok 2024-04-25T05:55:15.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[27010]: osdmap e614: 8 total, 8 up, 8 in 2024-04-25T05:55:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[37492]: pgmap v1159: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[37492]: 3.3 scrub starts 2024-04-25T05:55:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[37492]: 3.3 scrub ok 2024-04-25T05:55:15.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:15 smithi043 ceph-mon[37492]: osdmap e614: 8 total, 8 up, 8 in 2024-04-25T05:55:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:15 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:15 smithi060 ceph-mon[30898]: pgmap v1159: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:15 smithi060 ceph-mon[30898]: 3.3 scrub starts 2024-04-25T05:55:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:15 smithi060 ceph-mon[30898]: 3.3 scrub ok 2024-04-25T05:55:15.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:15 smithi060 ceph-mon[30898]: osdmap e614: 8 total, 8 up, 8 in 2024-04-25T05:55:15.955 INFO:teuthology.orchestra.run.smithi043.stdout:133143986450 2024-04-25T05:55:15.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T05:55:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:15.963 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:16.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 3.4 scrub starts 2024-04-25T05:55:16.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 3.4 scrub ok 2024-04-25T05:55:16.475 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 4.17 deep-scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 4.17 deep-scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 6.0 scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 6.0 scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 1.9 scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 1.9 scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 5.b scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[27010]: 5.b scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 3.4 scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 3.4 scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 4.17 deep-scrub starts 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 4.17 deep-scrub ok 2024-04-25T05:55:16.476 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 6.0 scrub starts 2024-04-25T05:55:16.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 6.0 scrub ok 2024-04-25T05:55:16.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 1.9 scrub starts 2024-04-25T05:55:16.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 1.9 scrub ok 2024-04-25T05:55:16.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 5.b scrub starts 2024-04-25T05:55:16.477 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:16 smithi043 ceph-mon[37492]: 5.b scrub ok 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 3.4 scrub starts 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 3.4 scrub ok 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 4.17 deep-scrub starts 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 4.17 deep-scrub ok 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 6.0 scrub starts 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 6.0 scrub ok 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 1.9 scrub starts 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 1.9 scrub ok 2024-04-25T05:55:16.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 5.b scrub starts 2024-04-25T05:55:16.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:16 smithi060 ceph-mon[30898]: 5.b scrub ok 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[27010]: 4.1d deep-scrub starts 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[27010]: 4.1d deep-scrub ok 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[27010]: pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[27010]: 3.14 scrub starts 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[27010]: 3.14 scrub ok 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[37492]: 4.1d deep-scrub starts 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[37492]: 4.1d deep-scrub ok 2024-04-25T05:55:17.567 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[37492]: pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:55:17.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[37492]: 3.14 scrub starts 2024-04-25T05:55:17.568 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:17 smithi043 ceph-mon[37492]: 3.14 scrub ok 2024-04-25T05:55:17.593 INFO:teuthology.orchestra.run.smithi043.stdout:253403070725 2024-04-25T05:55:17.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T05:55:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:17 smithi060 ceph-mon[30898]: 4.1d deep-scrub starts 2024-04-25T05:55:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:17 smithi060 ceph-mon[30898]: 4.1d deep-scrub ok 2024-04-25T05:55:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:17 smithi060 ceph-mon[30898]: pgmap v1161: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-25T05:55:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:17 smithi060 ceph-mon[30898]: 3.14 scrub starts 2024-04-25T05:55:17.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:17 smithi060 ceph-mon[30898]: 3.14 scrub ok 2024-04-25T05:55:18.054 INFO:teuthology.orchestra.run.smithi043.stdout:77309411613 2024-04-25T05:55:18.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T05:55:18.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: 4.0 scrub starts 2024-04-25T05:55:18.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: 4.0 scrub ok 2024-04-25T05:55:18.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:18.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: osdmap e615: 8 total, 8 up, 8 in 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: 5.6 scrub starts 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[27010]: 5.6 scrub ok 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: 4.0 scrub starts 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: 4.0 scrub ok 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: osdmap e615: 8 total, 8 up, 8 in 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: 5.6 scrub starts 2024-04-25T05:55:18.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:18 smithi043 ceph-mon[37492]: 5.6 scrub ok 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: 4.0 scrub starts 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: 4.0 scrub ok 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: osdmap e615: 8 total, 8 up, 8 in 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: 5.6 scrub starts 2024-04-25T05:55:18.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:18 smithi060 ceph-mon[30898]: 5.6 scrub ok 2024-04-25T05:55:19.300 INFO:teuthology.orchestra.run.smithi043.stdout:34359738664 2024-04-25T05:55:19.300 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T05:55:19.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:19 smithi043 ceph-mon[27010]: pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-04-25T05:55:19.531 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:19 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:19.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:19 smithi043 ceph-mon[37492]: pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-04-25T05:55:19.531 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:19 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:19.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:19 smithi060 ceph-mon[30898]: pgmap v1163: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-04-25T05:55:19.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:19 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:19.787 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:19 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:55:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:55:20.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 5.17 scrub starts 2024-04-25T05:55:20.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 5.17 scrub ok 2024-04-25T05:55:20.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 6.10 scrub starts 2024-04-25T05:55:20.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 6.10 scrub ok 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 4.1a deep-scrub starts 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: 4.1a deep-scrub ok 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[27010]: osdmap e616: 8 total, 8 up, 8 in 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 5.17 scrub starts 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 5.17 scrub ok 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 6.10 scrub starts 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 6.10 scrub ok 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 4.1a deep-scrub starts 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: 4.1a deep-scrub ok 2024-04-25T05:55:20.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:20 smithi043 ceph-mon[37492]: osdmap e616: 8 total, 8 up, 8 in 2024-04-25T05:55:20.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 5.17 scrub starts 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 5.17 scrub ok 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 6.10 scrub starts 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 6.10 scrub ok 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 4.1a deep-scrub starts 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: 4.1a deep-scrub ok 2024-04-25T05:55:20.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:20 smithi060 ceph-mon[30898]: osdmap e616: 8 total, 8 up, 8 in 2024-04-25T05:55:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: 4.14 scrub starts 2024-04-25T05:55:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: 4.14 scrub ok 2024-04-25T05:55:21.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: 5.1c scrub starts 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: 5.1c scrub ok 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: 4.14 scrub starts 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: 4.14 scrub ok 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: 5.1c scrub starts 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: 5.1c scrub ok 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:21.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:21 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: 4.14 scrub starts 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: 4.14 scrub ok 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: pgmap v1165: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: 5.1c scrub starts 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: 5.1c scrub ok 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:21.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:21 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:22.413 INFO:teuthology.orchestra.run.smithi043.stdout:214748365066 2024-04-25T05:55:22.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T05:55:22.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[27010]: 4.7 scrub starts 2024-04-25T05:55:22.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[27010]: 4.7 scrub ok 2024-04-25T05:55:22.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[27010]: osdmap e617: 8 total, 8 up, 8 in 2024-04-25T05:55:22.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[27010]: 4.1b scrub starts 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[27010]: 4.1b scrub ok 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[37492]: 4.7 scrub starts 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[37492]: 4.7 scrub ok 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[37492]: osdmap e617: 8 total, 8 up, 8 in 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[37492]: 4.1b scrub starts 2024-04-25T05:55:22.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:22 smithi043 ceph-mon[37492]: 4.1b scrub ok 2024-04-25T05:55:22.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:22 smithi060 ceph-mon[30898]: 4.7 scrub starts 2024-04-25T05:55:22.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:22 smithi060 ceph-mon[30898]: 4.7 scrub ok 2024-04-25T05:55:22.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:22 smithi060 ceph-mon[30898]: osdmap e617: 8 total, 8 up, 8 in 2024-04-25T05:55:22.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:22 smithi060 ceph-mon[30898]: 4.1b scrub starts 2024-04-25T05:55:22.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:22 smithi060 ceph-mon[30898]: 4.1b scrub ok 2024-04-25T05:55:22.873 INFO:teuthology.orchestra.run.smithi043.stdout:55834575139 2024-04-25T05:55:22.873 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T05:55:23.370 INFO:teuthology.orchestra.run.smithi043.stdout:171798692111 2024-04-25T05:55:23.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T05:55:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[27010]: 3.7 scrub starts 2024-04-25T05:55:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[27010]: 3.7 scrub ok 2024-04-25T05:55:23.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[27010]: pgmap v1167: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[27010]: 5.3 scrub starts 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[27010]: 5.3 scrub ok 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[37492]: 3.7 scrub starts 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[37492]: 3.7 scrub ok 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[37492]: pgmap v1167: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[37492]: 5.3 scrub starts 2024-04-25T05:55:23.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:23 smithi043 ceph-mon[37492]: 5.3 scrub ok 2024-04-25T05:55:23.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-mon[30898]: 3.7 scrub starts 2024-04-25T05:55:23.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-mon[30898]: 3.7 scrub ok 2024-04-25T05:55:23.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-mon[30898]: pgmap v1167: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-25T05:55:23.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-mon[30898]: 5.3 scrub starts 2024-04-25T05:55:23.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-mon[30898]: 5.3 scrub ok 2024-04-25T05:55:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:23.902 INFO:teuthology.orchestra.run.smithi043.stdout:111669149976 2024-04-25T05:55:23.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T05:55:23.911 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:24.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:55:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:55:24.502 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:24.543 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:24.718 INFO:teuthology.orchestra.run.smithi043.stdout:253403070727 2024-04-25T05:55:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: 6.1d scrub starts 2024-04-25T05:55:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: 6.1d scrub ok 2024-04-25T05:55:25.391 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/524412167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[27010]: osdmap e618: 8 total, 8 up, 8 in 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: 6.1d scrub starts 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: 6.1d scrub ok 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/524412167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:55:25.392 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:25 smithi043 ceph-mon[37492]: osdmap e618: 8 total, 8 up, 8 in 2024-04-25T05:55:25.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070725 got 253403070727 for osd.7 2024-04-25T05:55:25.499 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: 6.1d scrub starts 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: 6.1d scrub ok 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: pgmap v1168: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/524412167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T05:55:25.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:25 smithi060 ceph-mon[30898]: osdmap e618: 8 total, 8 up, 8 in 2024-04-25T05:55:26.212 INFO:teuthology.orchestra.run.smithi043.stdout:77309411615 2024-04-25T05:55:26.432 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[27010]: 6.1a scrub starts 2024-04-25T05:55:26.432 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[27010]: 6.1a scrub ok 2024-04-25T05:55:26.432 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1761277425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:55:26.433 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[37492]: 6.1a scrub starts 2024-04-25T05:55:26.433 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[37492]: 6.1a scrub ok 2024-04-25T05:55:26.433 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:26 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1761277425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:55:26.562 INFO:teuthology.orchestra.run.smithi043.stdout:34359738666 2024-04-25T05:55:26.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:26 smithi060 ceph-mon[30898]: 6.1a scrub starts 2024-04-25T05:55:26.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:26 smithi060 ceph-mon[30898]: 6.1a scrub ok 2024-04-25T05:55:26.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:26 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1761277425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T05:55:27.349 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738664 got 34359738666 for osd.0 2024-04-25T05:55:27.350 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:27 smithi043 ceph-mon[27010]: pgmap v1170: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:27.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:27 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/2618801778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:55:27.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:27 smithi043 ceph-mon[37492]: pgmap v1170: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:27.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:27 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/2618801778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:55:27.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:27 smithi060 ceph-mon[30898]: pgmap v1170: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-25T05:55:27.650 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:27 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/2618801778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T05:55:27.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411613 got 77309411615 for osd.2 2024-04-25T05:55:27.905 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:28.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:28 smithi043 ceph-mon[27010]: 3.19 scrub starts 2024-04-25T05:55:28.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:28 smithi043 ceph-mon[27010]: 3.19 scrub ok 2024-04-25T05:55:28.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:28 smithi043 ceph-mon[37492]: 3.19 scrub starts 2024-04-25T05:55:28.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:28 smithi043 ceph-mon[37492]: 3.19 scrub ok 2024-04-25T05:55:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:28.431 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:28.462 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:28.478 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:28.653 INFO:teuthology.orchestra.run.smithi043.stdout:111669149977 2024-04-25T05:55:28.748 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:28 smithi060 ceph-mon[30898]: 3.19 scrub starts 2024-04-25T05:55:28.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:28 smithi060 ceph-mon[30898]: 3.19 scrub ok 2024-04-25T05:55:29.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[27010]: 5.0 scrub starts 2024-04-25T05:55:29.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[27010]: 5.0 scrub ok 2024-04-25T05:55:29.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[27010]: pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:29.620 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3104222608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:55:29.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[37492]: 5.0 scrub starts 2024-04-25T05:55:29.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[37492]: 5.0 scrub ok 2024-04-25T05:55:29.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[37492]: pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:29.621 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3104222608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:55:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:29 smithi060 ceph-mon[30898]: 5.0 scrub starts 2024-04-25T05:55:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:29 smithi060 ceph-mon[30898]: 5.0 scrub ok 2024-04-25T05:55:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:29 smithi060 ceph-mon[30898]: pgmap v1171: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:29.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:29 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3104222608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T05:55:29.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149976 got 111669149977 for osd.3 2024-04-25T05:55:29.784 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:29.891 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:29 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:55:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:55:30.370 INFO:teuthology.orchestra.run.smithi043.stdout:133143986453 2024-04-25T05:55:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[27010]: 6.7 deep-scrub starts 2024-04-25T05:55:30.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[27010]: 6.7 deep-scrub ok 2024-04-25T05:55:30.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3100716604' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:55:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[37492]: 6.7 deep-scrub starts 2024-04-25T05:55:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[37492]: 6.7 deep-scrub ok 2024-04-25T05:55:30.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:30 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3100716604' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:55:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:30 smithi060 ceph-mon[30898]: 6.7 deep-scrub starts 2024-04-25T05:55:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:30 smithi060 ceph-mon[30898]: 6.7 deep-scrub ok 2024-04-25T05:55:30.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:30 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3100716604' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T05:55:31.021 INFO:teuthology.orchestra.run.smithi043.stdout:55834575140 2024-04-25T05:55:31.105 INFO:teuthology.orchestra.run.smithi043.stdout:214748365068 2024-04-25T05:55:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[37492]: 1.8 scrub starts 2024-04-25T05:55:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[37492]: 1.8 scrub ok 2024-04-25T05:55:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[37492]: pgmap v1172: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:31.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/1035944105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/217619118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[27010]: 1.8 scrub starts 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[27010]: 1.8 scrub ok 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[27010]: pgmap v1172: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/1035944105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:55:31.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:31 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/217619118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:55:31.722 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575139 got 55834575140 for osd.1 2024-04-25T05:55:31.722 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:31 smithi060 ceph-mon[30898]: 1.8 scrub starts 2024-04-25T05:55:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:31 smithi060 ceph-mon[30898]: 1.8 scrub ok 2024-04-25T05:55:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:31 smithi060 ceph-mon[30898]: pgmap v1172: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/1035944105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T05:55:31.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:31 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/217619118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T05:55:32.210 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986450 got 133143986453 for osd.4 2024-04-25T05:55:32.210 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[27010]: 3.18 scrub starts 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[27010]: 3.18 scrub ok 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[37492]: 3.18 scrub starts 2024-04-25T05:55:32.590 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[37492]: 3.18 scrub ok 2024-04-25T05:55:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:32.591 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:32 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:32 smithi060 ceph-mon[30898]: 3.18 scrub starts 2024-04-25T05:55:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:32 smithi060 ceph-mon[30898]: 3.18 scrub ok 2024-04-25T05:55:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:32 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:32.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:32 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:33.212 INFO:teuthology.orchestra.run.smithi043.stdout:171798692113 2024-04-25T05:55:33.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 1.4 scrub starts 2024-04-25T05:55:33.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 1.4 scrub ok 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 5.18 scrub starts 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 5.18 scrub ok 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: osdmap e619: 8 total, 8 up, 8 in 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 4.a scrub starts 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: 4.a scrub ok 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3798205197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 1.4 scrub starts 2024-04-25T05:55:33.595 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 1.4 scrub ok 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 5.18 scrub starts 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 5.18 scrub ok 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: osdmap e619: 8 total, 8 up, 8 in 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 4.a scrub starts 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: 4.a scrub ok 2024-04-25T05:55:33.596 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:33 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3798205197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 1.4 scrub starts 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 1.4 scrub ok 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: pgmap v1173: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 5.18 scrub starts 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 5.18 scrub ok 2024-04-25T05:55:33.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: osdmap e619: 8 total, 8 up, 8 in 2024-04-25T05:55:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 4.a scrub starts 2024-04-25T05:55:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: 4.a scrub ok 2024-04-25T05:55:33.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3798205197' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T05:55:34.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:55:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:55:34.310 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692111 got 171798692113 for osd.5 2024-04-25T05:55:34.310 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:34.426 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365066 got 214748365068 for osd.6 2024-04-25T05:55:34.426 DEBUG:teuthology.parallel:result is None 2024-04-25T05:55:34.426 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T05:55:34.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 6.a scrub starts 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 3.2 scrub starts 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 6.a scrub ok 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 3.2 scrub ok 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 4.e scrub starts 2024-04-25T05:55:34.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: 4.e scrub ok 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 6.a scrub starts 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 3.2 scrub starts 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 6.a scrub ok 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 3.2 scrub ok 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 4.e scrub starts 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: 4.e scrub ok 2024-04-25T05:55:34.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:34 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:34.682 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 6.a scrub starts 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 3.2 scrub starts 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 6.a scrub ok 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 3.2 scrub ok 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 4.e scrub starts 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: 4.e scrub ok 2024-04-25T05:55:34.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:34 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-25T05:55:35.463 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:55:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:55:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[27010]: 4.10 scrub starts 2024-04-25T05:55:35.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[27010]: 4.10 scrub ok 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[27010]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[27010]: pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[27010]: osdmap e620: 8 total, 8 up, 8 in 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[37492]: 4.10 scrub starts 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[37492]: 4.10 scrub ok 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[37492]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[37492]: pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:35.642 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:35 smithi043 ceph-mon[37492]: osdmap e620: 8 total, 8 up, 8 in 2024-04-25T05:55:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:35 smithi060 ceph-mon[30898]: 4.10 scrub starts 2024-04-25T05:55:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:35 smithi060 ceph-mon[30898]: 4.10 scrub ok 2024-04-25T05:55:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:35 smithi060 ceph-mon[30898]: from='client.24394 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-25T05:55:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:35 smithi060 ceph-mon[30898]: pgmap v1175: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-25T05:55:35.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:35 smithi060 ceph-mon[30898]: osdmap e620: 8 total, 8 up, 8 in 2024-04-25T05:55:36.181 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1176,"stamp":"2024-04-25T05:55:35.144964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3117,"num_read_kb":2837,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":544337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1804,"ondisk_log_size":1804,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":427,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7110324,"kb_used_data":18092,"kb_used_omap":12,"kb_used_meta":7092147,"kb_avail":742850892,"statfs":{"total":767960285184,"available":760679313408,"internally_reserved":0,"allocated":18526208,"data_stored":11105981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7262359088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.997298"},"pg_stats":[{"pgid":"4.c","version":"581'30","reported_seq":686,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244055+0000","last_change":"2024-04-25T05:54:26.189530+0000","last_active":"2024-04-25T05:55:33.244055+0000","last_peered":"2024-04-25T05:55:33.244055+0000","last_clean":"2024-04-25T05:55:33.244055+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:55:33.244055+0000","last_undegraded":"2024-04-25T05:55:33.244055+0000","last_fullsized":"2024-04-25T05:55:33.244055+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'30","last_scrub_stamp":"2024-04-25T05:54:26.186309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.186309+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:45.156742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1187,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372196+0000","last_change":"2024-04-25T05:53:57.738227+0000","last_active":"2024-04-25T05:55:32.372196+0000","last_peered":"2024-04-25T05:55:32.372196+0000","last_clean":"2024-04-25T05:55:32.372196+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:55:32.372196+0000","last_undegraded":"2024-04-25T05:55:32.372196+0000","last_fullsized":"2024-04-25T05:55:32.372196+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:57.736955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.736955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":659,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586052+0000","last_change":"2024-04-25T05:55:00.529284+0000","last_active":"2024-04-25T05:55:25.586052+0000","last_peered":"2024-04-25T05:55:25.586052+0000","last_clean":"2024-04-25T05:55:25.586052+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:55:25.586052+0000","last_undegraded":"2024-04-25T05:55:25.586052+0000","last_fullsized":"2024-04-25T05:55:25.586052+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:55:00.529198+0000","last_deep_scrub":"388'14","last_deep_scrub_stamp":"2024-04-25T05:53:57.528618+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.529198+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:03.248967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":756,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244584+0000","last_change":"2024-04-25T05:55:12.203013+0000","last_active":"2024-04-25T05:55:33.244584+0000","last_peered":"2024-04-25T05:55:33.244584+0000","last_clean":"2024-04-25T05:55:33.244584+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:55:33.244584+0000","last_undegraded":"2024-04-25T05:55:33.244584+0000","last_fullsized":"2024-04-25T05:55:33.244584+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:12.202870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:55:12.202870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.010806+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1536,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588999+0000","last_change":"2024-04-25T05:55:15.450232+0000","last_active":"2024-04-25T05:55:32.588999+0000","last_peered":"2024-04-25T05:55:32.588999+0000","last_clean":"2024-04-25T05:55:32.588999+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:55:32.588999+0000","last_undegraded":"2024-04-25T05:55:32.588999+0000","last_fullsized":"2024-04-25T05:55:32.588999+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:15.450125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.450125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.386702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"581'53","reported_seq":701,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244535+0000","last_change":"2024-04-25T05:55:00.259864+0000","last_active":"2024-04-25T05:55:33.244535+0000","last_peered":"2024-04-25T05:55:33.244535+0000","last_clean":"2024-04-25T05:55:33.244535+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:55:33.244535+0000","last_undegraded":"2024-04-25T05:55:33.244535+0000","last_fullsized":"2024-04-25T05:55:33.244535+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'53","last_scrub_stamp":"2024-04-25T05:55:00.259736+0000","last_deep_scrub":"581'53","last_deep_scrub_stamp":"2024-04-25T05:55:00.259736+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.259736+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:05.573152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":909,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371257+0000","last_change":"2024-04-25T05:54:59.495532+0000","last_active":"2024-04-25T05:55:32.371257+0000","last_peered":"2024-04-25T05:55:32.371257+0000","last_clean":"2024-04-25T05:55:32.371257+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:55:32.371257+0000","last_undegraded":"2024-04-25T05:55:32.371257+0000","last_fullsized":"2024-04-25T05:55:32.371257+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.495451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:59.495451+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.495451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.576932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1200,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372760+0000","last_change":"2024-04-25T05:54:28.469911+0000","last_active":"2024-04-25T05:55:32.372760+0000","last_peered":"2024-04-25T05:55:32.372760+0000","last_clean":"2024-04-25T05:55:32.372760+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:55:32.372760+0000","last_undegraded":"2024-04-25T05:55:32.372760+0000","last_fullsized":"2024-04-25T05:55:32.372760+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:28.469809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:28.469809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:52.028889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":535,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586330+0000","last_change":"2024-04-25T05:54:50.675033+0000","last_active":"2024-04-25T05:55:25.586330+0000","last_peered":"2024-04-25T05:55:25.586330+0000","last_clean":"2024-04-25T05:55:25.586330+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:55:25.586330+0000","last_undegraded":"2024-04-25T05:55:25.586330+0000","last_fullsized":"2024-04-25T05:55:25.586330+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:50.674927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.674927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:51.561974+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":1135,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755345+0000","last_change":"2024-04-25T05:55:29.769300+0000","last_active":"2024-04-25T05:55:32.755345+0000","last_peered":"2024-04-25T05:55:32.755345+0000","last_clean":"2024-04-25T05:55:32.755345+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:55:32.755345+0000","last_undegraded":"2024-04-25T05:55:32.755345+0000","last_fullsized":"2024-04-25T05:55:32.755345+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:55:29.769188+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:55:29.769188+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:30.899875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"581'33","reported_seq":1000,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.245992+0000","last_change":"2024-04-25T05:55:33.246042+0000","last_active":"2024-04-25T05:55:33.245992+0000","last_peered":"2024-04-25T05:55:33.245992+0000","last_clean":"2024-04-25T05:55:33.245992+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:55:33.245992+0000","last_undegraded":"2024-04-25T05:55:33.245992+0000","last_fullsized":"2024-04-25T05:55:33.245992+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'33","last_scrub_stamp":"2024-04-25T05:55:33.245892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:33.245892+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:57:02.767474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243451+0000","last_change":"2024-04-25T05:54:58.287691+0000","last_active":"2024-04-25T05:55:33.243451+0000","last_peered":"2024-04-25T05:55:33.243451+0000","last_clean":"2024-04-25T05:55:33.243451+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:55:33.243451+0000","last_undegraded":"2024-04-25T05:55:33.243451+0000","last_fullsized":"2024-04-25T05:55:33.243451+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.287555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.287555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.327665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756547+0000","last_change":"2024-04-25T05:55:07.677768+0000","last_active":"2024-04-25T05:55:32.756547+0000","last_peered":"2024-04-25T05:55:32.756547+0000","last_clean":"2024-04-25T05:55:32.756547+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:55:32.756547+0000","last_undegraded":"2024-04-25T05:55:32.756547+0000","last_fullsized":"2024-04-25T05:55:32.756547+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:07.677664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.677664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:17.643226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1175,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372465+0000","last_change":"2024-04-25T05:54:44.855689+0000","last_active":"2024-04-25T05:55:32.372465+0000","last_peered":"2024-04-25T05:55:32.372465+0000","last_clean":"2024-04-25T05:55:32.372465+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:55:32.372465+0000","last_undegraded":"2024-04-25T05:55:32.372465+0000","last_fullsized":"2024-04-25T05:55:32.372465+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.855611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.855611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.985647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"582'46","reported_seq":1242,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371435+0000","last_change":"2024-04-25T05:55:01.543726+0000","last_active":"2024-04-25T05:55:32.371435+0000","last_peered":"2024-04-25T05:55:32.371435+0000","last_clean":"2024-04-25T05:55:32.371435+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:55:32.371435+0000","last_undegraded":"2024-04-25T05:55:32.371435+0000","last_fullsized":"2024-04-25T05:55:32.371435+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'46","last_scrub_stamp":"2024-04-25T05:55:01.543612+0000","last_deep_scrub":"582'46","last_deep_scrub_stamp":"2024-04-25T05:55:01.543612+0000","last_clean_scrub_stamp":"2024-04-25T05:55:01.543612+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:06.221133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1171,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371395+0000","last_change":"2024-04-25T05:55:09.783532+0000","last_active":"2024-04-25T05:55:32.371395+0000","last_peered":"2024-04-25T05:55:32.371395+0000","last_clean":"2024-04-25T05:55:32.371395+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:55:32.371395+0000","last_undegraded":"2024-04-25T05:55:32.371395+0000","last_fullsized":"2024-04-25T05:55:32.371395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:09.783445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:09.783445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.266015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":909,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244532+0000","last_change":"2024-04-25T05:55:07.198140+0000","last_active":"2024-04-25T05:55:33.244532+0000","last_peered":"2024-04-25T05:55:33.244532+0000","last_clean":"2024-04-25T05:55:33.244532+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:55:33.244532+0000","last_undegraded":"2024-04-25T05:55:33.244532+0000","last_fullsized":"2024-04-25T05:55:33.244532+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:55:07.198024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.198024+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.769365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756412+0000","last_change":"2024-04-25T05:55:00.668485+0000","last_active":"2024-04-25T05:55:32.756412+0000","last_peered":"2024-04-25T05:55:32.756412+0000","last_clean":"2024-04-25T05:55:32.756412+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:55:32.756412+0000","last_undegraded":"2024-04-25T05:55:32.756412+0000","last_fullsized":"2024-04-25T05:55:32.756412+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.668373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.668373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:21.496320+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":1091,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755853+0000","last_change":"2024-04-25T05:54:40.775438+0000","last_active":"2024-04-25T05:55:32.755853+0000","last_peered":"2024-04-25T05:55:32.755853+0000","last_clean":"2024-04-25T05:55:32.755853+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:55:32.755853+0000","last_undegraded":"2024-04-25T05:55:32.755853+0000","last_fullsized":"2024-04-25T05:55:32.755853+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:40.775348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:54:40.775348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.684559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243439+0000","last_change":"2024-04-25T05:54:42.286100+0000","last_active":"2024-04-25T05:55:33.243439+0000","last_peered":"2024-04-25T05:55:33.243439+0000","last_clean":"2024-04-25T05:55:33.243439+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:55:33.243439+0000","last_undegraded":"2024-04-25T05:55:33.243439+0000","last_fullsized":"2024-04-25T05:55:33.243439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:42.286019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:42.286019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.929826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"581'54","reported_seq":1372,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371484+0000","last_change":"2024-04-25T05:55:16.816884+0000","last_active":"2024-04-25T05:55:32.371484+0000","last_peered":"2024-04-25T05:55:32.371484+0000","last_clean":"2024-04-25T05:55:32.371484+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:55:32.371484+0000","last_undegraded":"2024-04-25T05:55:32.371484+0000","last_fullsized":"2024-04-25T05:55:32.371484+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'54","last_scrub_stamp":"2024-04-25T05:55:16.816759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:16.816759+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.887708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373276+0000","last_change":"2024-04-25T05:55:21.685763+0000","last_active":"2024-04-25T05:55:32.373276+0000","last_peered":"2024-04-25T05:55:32.373276+0000","last_clean":"2024-04-25T05:55:32.373276+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:55:32.373276+0000","last_undegraded":"2024-04-25T05:55:32.373276+0000","last_fullsized":"2024-04-25T05:55:32.373276+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:21.685639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:21.685639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.568034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":743,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243626+0000","last_change":"2024-04-25T05:54:28.228531+0000","last_active":"2024-04-25T05:55:33.243626+0000","last_peered":"2024-04-25T05:55:33.243626+0000","last_clean":"2024-04-25T05:55:33.243626+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:55:33.243626+0000","last_undegraded":"2024-04-25T05:55:33.243626+0000","last_fullsized":"2024-04-25T05:55:33.243626+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:28.228445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:28.228445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.356012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1466,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243785+0000","last_change":"2024-04-25T05:54:41.265026+0000","last_active":"2024-04-25T05:55:33.243785+0000","last_peered":"2024-04-25T05:55:33.243785+0000","last_clean":"2024-04-25T05:55:33.243785+0000","last_became_active":"2024-04-25T05:52:47.115419+0000","last_became_peered":"2024-04-25T05:52:47.115419+0000","last_unstale":"2024-04-25T05:55:33.243785+0000","last_undegraded":"2024-04-25T05:55:33.243785+0000","last_fullsized":"2024-04-25T05:55:33.243785+0000","mapping_epoch":379,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:41.264918+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.264918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.305989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1442,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371962+0000","last_change":"2024-04-25T05:55:27.750052+0000","last_active":"2024-04-25T05:55:32.371962+0000","last_peered":"2024-04-25T05:55:32.371962+0000","last_clean":"2024-04-25T05:55:32.371962+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:55:32.371962+0000","last_undegraded":"2024-04-25T05:55:32.371962+0000","last_fullsized":"2024-04-25T05:55:32.371962+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:55:27.749966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:27.749966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.544075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"582'48","reported_seq":894,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243711+0000","last_change":"2024-04-25T05:54:32.250833+0000","last_active":"2024-04-25T05:55:33.243711+0000","last_peered":"2024-04-25T05:55:33.243711+0000","last_clean":"2024-04-25T05:55:33.243711+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:55:33.243711+0000","last_undegraded":"2024-04-25T05:55:33.243711+0000","last_fullsized":"2024-04-25T05:55:33.243711+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'48","last_scrub_stamp":"2024-04-25T05:54:32.250707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.250707+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.911453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":273,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778682+0000","last_change":"2024-04-25T05:55:05.201807+0000","last_active":"2024-04-25T05:55:32.778682+0000","last_peered":"2024-04-25T05:55:32.778682+0000","last_clean":"2024-04-25T05:55:32.778682+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:55:32.778682+0000","last_undegraded":"2024-04-25T05:55:32.778682+0000","last_fullsized":"2024-04-25T05:55:32.778682+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:55:05.201722+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:54:01.147559+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.201722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:23.179419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":561,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372208+0000","last_change":"2024-04-25T05:55:00.772872+0000","last_active":"2024-04-25T05:55:32.372208+0000","last_peered":"2024-04-25T05:55:32.372208+0000","last_clean":"2024-04-25T05:55:32.372208+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:55:32.372208+0000","last_undegraded":"2024-04-25T05:55:32.372208+0000","last_fullsized":"2024-04-25T05:55:32.372208+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.772758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.772758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.545490+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1432,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372297+0000","last_change":"2024-04-25T05:55:31.750049+0000","last_active":"2024-04-25T05:55:32.372297+0000","last_peered":"2024-04-25T05:55:32.372297+0000","last_clean":"2024-04-25T05:55:32.372297+0000","last_became_active":"2024-04-25T05:52:51.531591+0000","last_became_peered":"2024-04-25T05:52:51.531591+0000","last_unstale":"2024-04-25T05:55:32.372297+0000","last_undegraded":"2024-04-25T05:55:32.372297+0000","last_fullsized":"2024-04-25T05:55:32.372297+0000","mapping_epoch":387,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:31.749899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:55:31.749899+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.988148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":319,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778834+0000","last_change":"2024-04-25T05:55:22.205572+0000","last_active":"2024-04-25T05:55:32.778834+0000","last_peered":"2024-04-25T05:55:32.778834+0000","last_clean":"2024-04-25T05:55:32.778834+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:55:32.778834+0000","last_undegraded":"2024-04-25T05:55:32.778834+0000","last_fullsized":"2024-04-25T05:55:32.778834+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:55:22.205480+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:55:22.205480+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:39.004832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"581'42","reported_seq":936,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371436+0000","last_change":"2024-04-25T05:54:36.487946+0000","last_active":"2024-04-25T05:55:32.371436+0000","last_peered":"2024-04-25T05:55:32.371436+0000","last_clean":"2024-04-25T05:55:32.371436+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:55:32.371436+0000","last_undegraded":"2024-04-25T05:55:32.371436+0000","last_fullsized":"2024-04-25T05:55:32.371436+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'42","last_scrub_stamp":"2024-04-25T05:54:36.487842+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:54:36.487842+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:42.043257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":512,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756421+0000","last_change":"2024-04-25T05:54:34.810815+0000","last_active":"2024-04-25T05:55:32.756421+0000","last_peered":"2024-04-25T05:55:32.756421+0000","last_clean":"2024-04-25T05:55:32.756421+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:55:32.756421+0000","last_undegraded":"2024-04-25T05:55:32.756421+0000","last_fullsized":"2024-04-25T05:55:32.756421+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.810673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.810673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:39.599925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778279+0000","last_change":"2024-04-25T05:55:15.149927+0000","last_active":"2024-04-25T05:55:32.778279+0000","last_peered":"2024-04-25T05:55:32.778279+0000","last_clean":"2024-04-25T05:55:32.778279+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:55:32.778279+0000","last_undegraded":"2024-04-25T05:55:32.778279+0000","last_fullsized":"2024-04-25T05:55:32.778279+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.149819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.149819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:36.742334+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1421,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371865+0000","last_change":"2024-04-25T05:54:52.483978+0000","last_active":"2024-04-25T05:55:32.371865+0000","last_peered":"2024-04-25T05:55:32.371865+0000","last_clean":"2024-04-25T05:55:32.371865+0000","last_became_active":"2024-04-25T05:52:39.109573+0000","last_became_peered":"2024-04-25T05:52:39.109573+0000","last_unstale":"2024-04-25T05:55:32.371865+0000","last_undegraded":"2024-04-25T05:55:32.371865+0000","last_fullsized":"2024-04-25T05:55:32.371865+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:52.483888+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:54:52.483888+0000","last_clean_scrub_stamp":"2024-04-25T05:54:52.483888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:56.751854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":945,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371378+0000","last_change":"2024-04-25T05:54:55.473987+0000","last_active":"2024-04-25T05:55:32.371378+0000","last_peered":"2024-04-25T05:55:32.371378+0000","last_clean":"2024-04-25T05:55:32.371378+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:55:32.371378+0000","last_undegraded":"2024-04-25T05:55:32.371378+0000","last_fullsized":"2024-04-25T05:55:32.371378+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:55.473897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:54:55.473897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:07.847727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"581'45","reported_seq":304,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778769+0000","last_change":"2024-04-25T05:54:47.240397+0000","last_active":"2024-04-25T05:55:32.778769+0000","last_peered":"2024-04-25T05:55:32.778769+0000","last_clean":"2024-04-25T05:55:32.778769+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:55:32.778769+0000","last_undegraded":"2024-04-25T05:55:32.778769+0000","last_fullsized":"2024-04-25T05:55:32.778769+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'45","last_scrub_stamp":"2024-04-25T05:54:47.240244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:47.240244+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:12.021917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1217,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372600+0000","last_change":"2024-04-25T05:55:14.679679+0000","last_active":"2024-04-25T05:55:32.372600+0000","last_peered":"2024-04-25T05:55:32.372600+0000","last_clean":"2024-04-25T05:55:32.372600+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:55:32.372600+0000","last_undegraded":"2024-04-25T05:55:32.372600+0000","last_fullsized":"2024-04-25T05:55:32.372600+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:14.679597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:14.679597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.036407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1191,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372158+0000","last_change":"2024-04-25T05:54:41.480708+0000","last_active":"2024-04-25T05:55:32.372158+0000","last_peered":"2024-04-25T05:55:32.372158+0000","last_clean":"2024-04-25T05:55:32.372158+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:55:32.372158+0000","last_undegraded":"2024-04-25T05:55:32.372158+0000","last_fullsized":"2024-04-25T05:55:32.372158+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:41.480591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.480591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.057402+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1448,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371737+0000","last_change":"2024-04-25T05:55:08.557057+0000","last_active":"2024-04-25T05:55:32.371737+0000","last_peered":"2024-04-25T05:55:32.371737+0000","last_clean":"2024-04-25T05:55:32.371737+0000","last_became_active":"2024-04-25T05:52:43.436401+0000","last_became_peered":"2024-04-25T05:52:43.436401+0000","last_unstale":"2024-04-25T05:55:32.371737+0000","last_undegraded":"2024-04-25T05:55:32.371737+0000","last_fullsized":"2024-04-25T05:55:32.371737+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:08.556938+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:55:08.556938+0000","last_clean_scrub_stamp":"2024-04-25T05:55:08.556938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.347835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":518,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756594+0000","last_change":"2024-04-25T05:54:44.773940+0000","last_active":"2024-04-25T05:55:32.756594+0000","last_peered":"2024-04-25T05:55:32.756594+0000","last_clean":"2024-04-25T05:55:32.756594+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:55:32.756594+0000","last_undegraded":"2024-04-25T05:55:32.756594+0000","last_fullsized":"2024-04-25T05:55:32.756594+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.773835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:44.773835+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.773835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.593004+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"581'36","reported_seq":215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778311+0000","last_change":"2024-04-25T05:54:41.168240+0000","last_active":"2024-04-25T05:55:32.778311+0000","last_peered":"2024-04-25T05:55:32.778311+0000","last_clean":"2024-04-25T05:55:32.778311+0000","last_became_active":"2024-04-25T05:52:43.437539+0000","last_became_peered":"2024-04-25T05:52:43.437539+0000","last_unstale":"2024-04-25T05:55:32.778311+0000","last_undegraded":"2024-04-25T05:55:32.778311+0000","last_fullsized":"2024-04-25T05:55:32.778311+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'36","last_scrub_stamp":"2024-04-25T05:54:41.168118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.168118+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.078238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2515,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588390+0000","last_change":"2024-04-25T05:53:53.683465+0000","last_active":"2024-04-25T05:55:32.588390+0000","last_peered":"2024-04-25T05:55:32.588390+0000","last_clean":"2024-04-25T05:55:32.588390+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:55:32.588390+0000","last_undegraded":"2024-04-25T05:55:32.588390+0000","last_fullsized":"2024-04-25T05:55:32.588390+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:53:53.683339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.683339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1275,"num_read_kb":1270,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778546+0000","last_change":"2024-04-25T05:54:48.225902+0000","last_active":"2024-04-25T05:55:32.778546+0000","last_peered":"2024-04-25T05:55:32.778546+0000","last_clean":"2024-04-25T05:55:32.778546+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:55:32.778546+0000","last_undegraded":"2024-04-25T05:55:32.778546+0000","last_fullsized":"2024-04-25T05:55:32.778546+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:48.225753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:54:48.225753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.650917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1179,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372376+0000","last_change":"2024-04-25T05:54:35.488312+0000","last_active":"2024-04-25T05:55:32.372376+0000","last_peered":"2024-04-25T05:55:32.372376+0000","last_clean":"2024-04-25T05:55:32.372376+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:55:32.372376+0000","last_undegraded":"2024-04-25T05:55:32.372376+0000","last_fullsized":"2024-04-25T05:55:32.372376+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:35.488226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:35.488226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.587910+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1428,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372102+0000","last_change":"2024-04-25T05:54:38.471994+0000","last_active":"2024-04-25T05:55:32.372102+0000","last_peered":"2024-04-25T05:55:32.372102+0000","last_clean":"2024-04-25T05:55:32.372102+0000","last_became_active":"2024-04-25T05:52:55.109763+0000","last_became_peered":"2024-04-25T05:52:55.109763+0000","last_unstale":"2024-04-25T05:55:32.372102+0000","last_undegraded":"2024-04-25T05:55:32.372102+0000","last_fullsized":"2024-04-25T05:55:32.372102+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:38.471909+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:54:38.471909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:51.162526+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1208,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589202+0000","last_change":"2024-04-25T05:55:04.510991+0000","last_active":"2024-04-25T05:55:32.589202+0000","last_peered":"2024-04-25T05:55:32.589202+0000","last_clean":"2024-04-25T05:55:32.589202+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:55:32.589202+0000","last_undegraded":"2024-04-25T05:55:32.589202+0000","last_fullsized":"2024-04-25T05:55:32.589202+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:04.510872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:55:04.510872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:22.322702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"581'51","reported_seq":1155,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371839+0000","last_change":"2024-04-25T05:54:26.423017+0000","last_active":"2024-04-25T05:55:32.371839+0000","last_peered":"2024-04-25T05:55:32.371839+0000","last_clean":"2024-04-25T05:55:32.371839+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:55:32.371839+0000","last_undegraded":"2024-04-25T05:55:32.371839+0000","last_fullsized":"2024-04-25T05:55:32.371839+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'51","last_scrub_stamp":"2024-04-25T05:54:26.422816+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.422816+0000","objects_scrubbed":8,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:44.329991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779141+0000","last_change":"2024-04-25T05:55:32.779181+0000","last_active":"2024-04-25T05:55:32.779141+0000","last_peered":"2024-04-25T05:55:32.779141+0000","last_clean":"2024-04-25T05:55:32.779141+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:55:32.779141+0000","last_undegraded":"2024-04-25T05:55:32.779141+0000","last_fullsized":"2024-04-25T05:55:32.779141+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.779056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.779056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.628666+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1553,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588440+0000","last_change":"2024-04-25T05:54:38.761794+0000","last_active":"2024-04-25T05:55:32.588440+0000","last_peered":"2024-04-25T05:55:32.588440+0000","last_clean":"2024-04-25T05:55:32.588440+0000","last_became_active":"2024-04-25T05:52:35.091072+0000","last_became_peered":"2024-04-25T05:52:35.091072+0000","last_unstale":"2024-04-25T05:55:32.588440+0000","last_undegraded":"2024-04-25T05:55:32.588440+0000","last_fullsized":"2024-04-25T05:55:32.588440+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":19,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:38.761693+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:54:38.761693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.195038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":485,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373738+0000","last_change":"2024-04-25T05:55:29.127381+0000","last_active":"2024-04-25T05:55:32.373738+0000","last_peered":"2024-04-25T05:55:32.373738+0000","last_clean":"2024-04-25T05:55:32.373738+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:55:32.373738+0000","last_undegraded":"2024-04-25T05:55:32.373738+0000","last_fullsized":"2024-04-25T05:55:32.373738+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:29.127262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:55:29.127262+0000","last_clean_scrub_stamp":"2024-04-25T05:55:29.127262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:42.796405+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":541,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586234+0000","last_change":"2024-04-25T05:55:17.794404+0000","last_active":"2024-04-25T05:55:25.586234+0000","last_peered":"2024-04-25T05:55:25.586234+0000","last_clean":"2024-04-25T05:55:25.586234+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:55:25.586234+0000","last_undegraded":"2024-04-25T05:55:25.586234+0000","last_fullsized":"2024-04-25T05:55:25.586234+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:17.794316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:17.794316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.671449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"581'33","reported_seq":1351,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371318+0000","last_change":"2024-04-25T05:55:20.756691+0000","last_active":"2024-04-25T05:55:32.371318+0000","last_peered":"2024-04-25T05:55:32.371318+0000","last_clean":"2024-04-25T05:55:32.371318+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:55:32.371318+0000","last_undegraded":"2024-04-25T05:55:32.371318+0000","last_fullsized":"2024-04-25T05:55:32.371318+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'33","last_scrub_stamp":"2024-04-25T05:55:20.756598+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.756598+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.540095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":541,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585008+0000","last_change":"2024-04-25T05:54:43.418290+0000","last_active":"2024-04-25T05:55:25.585008+0000","last_peered":"2024-04-25T05:55:25.585008+0000","last_clean":"2024-04-25T05:55:25.585008+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:55:25.585008+0000","last_undegraded":"2024-04-25T05:55:25.585008+0000","last_fullsized":"2024-04-25T05:55:25.585008+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:43.418207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:54:43.418207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.125299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371196+0000","last_change":"2024-04-25T05:54:27.494562+0000","last_active":"2024-04-25T05:55:32.371196+0000","last_peered":"2024-04-25T05:55:32.371196+0000","last_clean":"2024-04-25T05:55:32.371196+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:55:32.371196+0000","last_undegraded":"2024-04-25T05:55:32.371196+0000","last_fullsized":"2024-04-25T05:55:32.371196+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:27.494478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:27.494478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:39.511823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1186,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589652+0000","last_change":"2024-04-25T05:54:44.788203+0000","last_active":"2024-04-25T05:55:32.589652+0000","last_peered":"2024-04-25T05:55:32.589652+0000","last_clean":"2024-04-25T05:55:32.589652+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:55:32.589652+0000","last_undegraded":"2024-04-25T05:55:32.589652+0000","last_fullsized":"2024-04-25T05:55:32.589652+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.788114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.788114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.659567+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1561,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588680+0000","last_change":"2024-04-25T05:54:37.771913+0000","last_active":"2024-04-25T05:55:32.588680+0000","last_peered":"2024-04-25T05:55:32.588680+0000","last_clean":"2024-04-25T05:55:32.588680+0000","last_became_active":"2024-04-25T05:52:27.138282+0000","last_became_peered":"2024-04-25T05:52:27.138282+0000","last_unstale":"2024-04-25T05:55:32.588680+0000","last_undegraded":"2024-04-25T05:55:32.588680+0000","last_fullsized":"2024-04-25T05:55:32.588680+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:37.771809+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:54:37.771809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:41.922250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778744+0000","last_change":"2024-04-25T05:54:26.260460+0000","last_active":"2024-04-25T05:55:32.778744+0000","last_peered":"2024-04-25T05:55:32.778744+0000","last_clean":"2024-04-25T05:55:32.778744+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:55:32.778744+0000","last_undegraded":"2024-04-25T05:55:32.778744+0000","last_fullsized":"2024-04-25T05:55:32.778744+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:26.260375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.260375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:49.182432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"581'71","reported_seq":1603,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.995345+0000","last_change":"2024-04-25T05:54:46.506499+0000","last_active":"2024-04-25T05:55:33.995345+0000","last_peered":"2024-04-25T05:55:33.995345+0000","last_clean":"2024-04-25T05:55:33.995345+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:55:33.995345+0000","last_undegraded":"2024-04-25T05:55:33.995345+0000","last_fullsized":"2024-04-25T05:55:33.995345+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'71","last_scrub_stamp":"2024-04-25T05:54:46.506365+0000","last_deep_scrub":"581'71","last_deep_scrub_stamp":"2024-04-25T05:54:46.506365+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.506365+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:53.945333+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":762,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243653+0000","last_change":"2024-04-25T05:55:15.108209+0000","last_active":"2024-04-25T05:55:33.243653+0000","last_peered":"2024-04-25T05:55:33.243653+0000","last_clean":"2024-04-25T05:55:33.243653+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:55:33.243653+0000","last_undegraded":"2024-04-25T05:55:33.243653+0000","last_fullsized":"2024-04-25T05:55:33.243653+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.108110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.108110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:32.957117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"506'3","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755594+0000","last_change":"2024-04-25T05:54:27.940078+0000","last_active":"2024-04-25T05:55:32.755594+0000","last_peered":"2024-04-25T05:55:32.755594+0000","last_clean":"2024-04-25T05:55:32.755594+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:55:32.755594+0000","last_undegraded":"2024-04-25T05:55:32.755594+0000","last_fullsized":"2024-04-25T05:55:32.755594+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'3","last_scrub_stamp":"2024-04-25T05:54:27.939936+0000","last_deep_scrub":"506'3","last_deep_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_clean_scrub_stamp":"2024-04-25T05:54:27.939936+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1151,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372903+0000","last_change":"2024-04-25T05:54:32.917114+0000","last_active":"2024-04-25T05:55:32.372903+0000","last_peered":"2024-04-25T05:55:32.372903+0000","last_clean":"2024-04-25T05:55:32.372903+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:55:32.372903+0000","last_undegraded":"2024-04-25T05:55:32.372903+0000","last_fullsized":"2024-04-25T05:55:32.372903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:32.917028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:32.917028+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.917028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:40.343262+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":1262,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755496+0000","last_change":"2024-04-25T05:54:39.795686+0000","last_active":"2024-04-25T05:55:32.755496+0000","last_peered":"2024-04-25T05:55:32.755496+0000","last_clean":"2024-04-25T05:55:32.755496+0000","last_became_active":"2024-04-25T05:52:31.177637+0000","last_became_peered":"2024-04-25T05:52:31.177637+0000","last_unstale":"2024-04-25T05:55:32.755496+0000","last_undegraded":"2024-04-25T05:55:32.755496+0000","last_fullsized":"2024-04-25T05:55:32.755496+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:39.795598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:54:39.795598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.695810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"581'32","reported_seq":290,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778592+0000","last_change":"2024-04-25T05:55:00.330495+0000","last_active":"2024-04-25T05:55:32.778592+0000","last_peered":"2024-04-25T05:55:32.778592+0000","last_clean":"2024-04-25T05:55:32.778592+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:55:32.778592+0000","last_undegraded":"2024-04-25T05:55:32.778592+0000","last_fullsized":"2024-04-25T05:55:32.778592+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:55:00.330385+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.330385+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.895940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756173+0000","last_change":"2024-04-25T05:54:51.735918+0000","last_active":"2024-04-25T05:55:32.756173+0000","last_peered":"2024-04-25T05:55:32.756173+0000","last_clean":"2024-04-25T05:55:32.756173+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:55:32.756173+0000","last_undegraded":"2024-04-25T05:55:32.756173+0000","last_fullsized":"2024-04-25T05:55:32.756173+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:51.735807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:54:51.735807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:03.296810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":535,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585142+0000","last_change":"2024-04-25T05:55:05.716998+0000","last_active":"2024-04-25T05:55:25.585142+0000","last_peered":"2024-04-25T05:55:25.585142+0000","last_clean":"2024-04-25T05:55:25.585142+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:55:25.585142+0000","last_undegraded":"2024-04-25T05:55:25.585142+0000","last_fullsized":"2024-04-25T05:55:25.585142+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:05.716901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.716901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.752558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778179+0000","last_change":"2024-04-25T05:54:25.300412+0000","last_active":"2024-04-25T05:55:32.778179+0000","last_peered":"2024-04-25T05:55:32.778179+0000","last_clean":"2024-04-25T05:55:32.778179+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:55:32.778179+0000","last_undegraded":"2024-04-25T05:55:32.778179+0000","last_fullsized":"2024-04-25T05:55:32.778179+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:25.300308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:25.300308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.594574+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"582'67","reported_seq":997,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.592469+0000","last_change":"2024-04-25T05:55:32.592521+0000","last_active":"2024-04-25T05:55:32.592469+0000","last_peered":"2024-04-25T05:55:32.592469+0000","last_clean":"2024-04-25T05:55:32.592469+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:55:32.592469+0000","last_undegraded":"2024-04-25T05:55:32.592469+0000","last_fullsized":"2024-04-25T05:55:32.592469+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'67","last_scrub_stamp":"2024-04-25T05:55:32.592394+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.592394+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:57:01.099599+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":1232,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589400+0000","last_change":"2024-04-25T05:54:53.582781+0000","last_active":"2024-04-25T05:55:32.589400+0000","last_peered":"2024-04-25T05:55:32.589400+0000","last_clean":"2024-04-25T05:55:32.589400+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:55:32.589400+0000","last_undegraded":"2024-04-25T05:55:32.589400+0000","last_fullsized":"2024-04-25T05:55:32.589400+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:54:53.582673+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:54:53.582673+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.091409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":547,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585612+0000","last_change":"2024-04-25T05:55:15.729050+0000","last_active":"2024-04-25T05:55:25.585612+0000","last_peered":"2024-04-25T05:55:25.585612+0000","last_clean":"2024-04-25T05:55:25.585612+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:55:25.585612+0000","last_undegraded":"2024-04-25T05:55:25.585612+0000","last_fullsized":"2024-04-25T05:55:25.585612+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.728949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.728949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:34.671139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588299+0000","last_change":"2024-04-25T05:54:59.518039+0000","last_active":"2024-04-25T05:55:32.588299+0000","last_peered":"2024-04-25T05:55:32.588299+0000","last_clean":"2024-04-25T05:55:32.588299+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:55:32.588299+0000","last_undegraded":"2024-04-25T05:55:32.588299+0000","last_fullsized":"2024-04-25T05:55:32.588299+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.517922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.517922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.907823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":547,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.584789+0000","last_change":"2024-04-25T05:54:47.398323+0000","last_active":"2024-04-25T05:55:25.584789+0000","last_peered":"2024-04-25T05:55:25.584789+0000","last_clean":"2024-04-25T05:55:25.584789+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:55:25.584789+0000","last_undegraded":"2024-04-25T05:55:25.584789+0000","last_fullsized":"2024-04-25T05:55:25.584789+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:47.398225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:47.398225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:04.378970+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"600'43","reported_seq":618,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244327+0000","last_change":"2024-04-25T05:54:45.296651+0000","last_active":"2024-04-25T05:55:33.244327+0000","last_peered":"2024-04-25T05:55:33.244327+0000","last_clean":"2024-04-25T05:55:33.244327+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:55:33.244327+0000","last_undegraded":"2024-04-25T05:55:33.244327+0000","last_fullsized":"2024-04-25T05:55:33.244327+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'43","last_scrub_stamp":"2024-04-25T05:54:45.296530+0000","last_deep_scrub":"600'43","last_deep_scrub_stamp":"2024-04-25T05:54:45.296530+0000","last_clean_scrub_stamp":"2024-04-25T05:54:45.296530+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.109602+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589467+0000","last_change":"2024-04-25T05:54:54.552769+0000","last_active":"2024-04-25T05:55:32.589467+0000","last_peered":"2024-04-25T05:55:32.589467+0000","last_clean":"2024-04-25T05:55:32.589467+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:55:32.589467+0000","last_undegraded":"2024-04-25T05:55:32.589467+0000","last_fullsized":"2024-04-25T05:55:32.589467+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:54.552681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:54:54.552681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.031257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1163,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372386+0000","last_change":"2024-04-25T05:55:06.051836+0000","last_active":"2024-04-25T05:55:32.372386+0000","last_peered":"2024-04-25T05:55:32.372386+0000","last_clean":"2024-04-25T05:55:32.372386+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:55:32.372386+0000","last_undegraded":"2024-04-25T05:55:32.372386+0000","last_fullsized":"2024-04-25T05:55:32.372386+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:06.051728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:06.051728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:22.918905+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1471,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588417+0000","last_change":"2024-04-25T05:54:45.736450+0000","last_active":"2024-04-25T05:55:32.588417+0000","last_peered":"2024-04-25T05:55:32.588417+0000","last_clean":"2024-04-25T05:55:32.588417+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:55:32.588417+0000","last_undegraded":"2024-04-25T05:55:32.588417+0000","last_fullsized":"2024-04-25T05:55:32.588417+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:54:45.736320+0000","last_deep_scrub":"208'11","last_deep_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_clean_scrub_stamp":"2024-04-25T05:54:45.736320+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.710562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"581'51","reported_seq":550,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755339+0000","last_change":"2024-04-25T05:54:30.886889+0000","last_active":"2024-04-25T05:55:32.755339+0000","last_peered":"2024-04-25T05:55:32.755339+0000","last_clean":"2024-04-25T05:55:32.755339+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:55:32.755339+0000","last_undegraded":"2024-04-25T05:55:32.755339+0000","last_fullsized":"2024-04-25T05:55:32.755339+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'51","last_scrub_stamp":"2024-04-25T05:54:30.886719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.886719+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:33.869436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1245,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589577+0000","last_change":"2024-04-25T05:55:03.505601+0000","last_active":"2024-04-25T05:55:32.589577+0000","last_peered":"2024-04-25T05:55:32.589577+0000","last_clean":"2024-04-25T05:55:32.589577+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:55:32.589577+0000","last_undegraded":"2024-04-25T05:55:32.589577+0000","last_fullsized":"2024-04-25T05:55:32.589577+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:55:03.505497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:03.505497+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:21.467173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":518,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.777051+0000","last_change":"2024-04-25T05:55:32.777078+0000","last_active":"2024-04-25T05:55:32.777051+0000","last_peered":"2024-04-25T05:55:32.777051+0000","last_clean":"2024-04-25T05:55:32.777051+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:55:32.777051+0000","last_undegraded":"2024-04-25T05:55:32.777051+0000","last_fullsized":"2024-04-25T05:55:32.777051+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.776959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.776959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.153078+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"613'28","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373560+0000","last_change":"2024-04-25T05:55:14.809995+0000","last_active":"2024-04-25T05:55:32.373560+0000","last_peered":"2024-04-25T05:55:32.373560+0000","last_clean":"2024-04-25T05:55:32.373560+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:55:32.373560+0000","last_undegraded":"2024-04-25T05:55:32.373560+0000","last_fullsized":"2024-04-25T05:55:32.373560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'28","last_scrub_stamp":"2024-04-25T05:55:14.809911+0000","last_deep_scrub":"613'28","last_deep_scrub_stamp":"2024-04-25T05:55:14.809911+0000","last_clean_scrub_stamp":"2024-04-25T05:55:14.809911+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:19.085723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":944,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.370891+0000","last_change":"2024-04-25T05:54:24.388418+0000","last_active":"2024-04-25T05:55:32.370891+0000","last_peered":"2024-04-25T05:55:32.370891+0000","last_clean":"2024-04-25T05:55:32.370891+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:55:32.370891+0000","last_undegraded":"2024-04-25T05:55:32.370891+0000","last_fullsized":"2024-04-25T05:55:32.370891+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:24.388316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:54:24.388316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:53.660202+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756389+0000","last_change":"2024-04-25T05:54:46.752748+0000","last_active":"2024-04-25T05:55:32.756389+0000","last_peered":"2024-04-25T05:55:32.756389+0000","last_clean":"2024-04-25T05:55:32.756389+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:55:32.756389+0000","last_undegraded":"2024-04-25T05:55:32.756389+0000","last_fullsized":"2024-04-25T05:55:32.756389+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:46.752658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:46.752658+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.752658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:01.381614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589044+0000","last_change":"2024-04-25T05:55:00.655656+0000","last_active":"2024-04-25T05:55:32.589044+0000","last_peered":"2024-04-25T05:55:32.589044+0000","last_clean":"2024-04-25T05:55:32.589044+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:55:32.589044+0000","last_undegraded":"2024-04-25T05:55:32.589044+0000","last_fullsized":"2024-04-25T05:55:32.589044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.655554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.655554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.183884+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"602'39","reported_seq":298,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778686+0000","last_change":"2024-04-25T05:54:50.269023+0000","last_active":"2024-04-25T05:55:32.778686+0000","last_peered":"2024-04-25T05:55:32.778686+0000","last_clean":"2024-04-25T05:55:32.778686+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:55:32.778686+0000","last_undegraded":"2024-04-25T05:55:32.778686+0000","last_fullsized":"2024-04-25T05:55:32.778686+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'39","last_scrub_stamp":"2024-04-25T05:54:50.268891+0000","last_deep_scrub":"602'39","last_deep_scrub_stamp":"2024-04-25T05:54:50.268891+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.268891+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.400862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1203,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588567+0000","last_change":"2024-04-25T05:54:56.527212+0000","last_active":"2024-04-25T05:55:32.588567+0000","last_peered":"2024-04-25T05:55:32.588567+0000","last_clean":"2024-04-25T05:55:32.588567+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:55:32.588567+0000","last_undegraded":"2024-04-25T05:55:32.588567+0000","last_fullsized":"2024-04-25T05:55:32.588567+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:56.527124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:54:56.527124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.197169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1154,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373482+0000","last_change":"2024-04-25T05:55:18.784318+0000","last_active":"2024-04-25T05:55:32.373482+0000","last_peered":"2024-04-25T05:55:32.373482+0000","last_clean":"2024-04-25T05:55:32.373482+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:55:32.373482+0000","last_undegraded":"2024-04-25T05:55:32.373482+0000","last_fullsized":"2024-04-25T05:55:32.373482+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:18.784225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:18.784225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.971233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":526,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585911+0000","last_change":"2024-04-25T05:54:46.431180+0000","last_active":"2024-04-25T05:55:25.585911+0000","last_peered":"2024-04-25T05:55:25.585911+0000","last_clean":"2024-04-25T05:55:25.585911+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:55:25.585911+0000","last_undegraded":"2024-04-25T05:55:25.585911+0000","last_fullsized":"2024-04-25T05:55:25.585911+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:46.431061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.431061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.757044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"581'32","reported_seq":532,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756066+0000","last_change":"2024-04-25T05:55:02.624343+0000","last_active":"2024-04-25T05:55:32.756066+0000","last_peered":"2024-04-25T05:55:32.756066+0000","last_clean":"2024-04-25T05:55:32.756066+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:55:32.756066+0000","last_undegraded":"2024-04-25T05:55:32.756066+0000","last_fullsized":"2024-04-25T05:55:32.756066+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:55:02.624225+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:55:02.624225+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:04.850542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778401+0000","last_change":"2024-04-25T05:54:53.256101+0000","last_active":"2024-04-25T05:55:32.778401+0000","last_peered":"2024-04-25T05:55:32.778401+0000","last_clean":"2024-04-25T05:55:32.778401+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:55:32.778401+0000","last_undegraded":"2024-04-25T05:55:32.778401+0000","last_fullsized":"2024-04-25T05:55:32.778401+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:53.256012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:54:53.256012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.289454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1443,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373139+0000","last_change":"2024-04-25T05:55:07.811741+0000","last_active":"2024-04-25T05:55:32.373139+0000","last_peered":"2024-04-25T05:55:32.373139+0000","last_clean":"2024-04-25T05:55:32.373139+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:55:32.373139+0000","last_undegraded":"2024-04-25T05:55:32.373139+0000","last_fullsized":"2024-04-25T05:55:32.373139+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:55:07.811600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.811600+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:13.028382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":725,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244731+0000","last_change":"2024-04-25T05:54:39.440135+0000","last_active":"2024-04-25T05:55:33.244731+0000","last_peered":"2024-04-25T05:55:33.244731+0000","last_clean":"2024-04-25T05:55:33.244731+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:55:33.244731+0000","last_undegraded":"2024-04-25T05:55:33.244731+0000","last_fullsized":"2024-04-25T05:55:33.244731+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:39.440013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:39.440013+0000","last_clean_scrub_stamp":"2024-04-25T05:54:39.440013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.436716+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"581'42","reported_seq":1236,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372624+0000","last_change":"2024-04-25T05:55:19.744864+0000","last_active":"2024-04-25T05:55:32.372624+0000","last_peered":"2024-04-25T05:55:32.372624+0000","last_clean":"2024-04-25T05:55:32.372624+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:55:32.372624+0000","last_undegraded":"2024-04-25T05:55:32.372624+0000","last_fullsized":"2024-04-25T05:55:32.372624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'42","last_scrub_stamp":"2024-04-25T05:55:19.744753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:19.744753+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.237215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1197,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588751+0000","last_change":"2024-04-25T05:54:58.515557+0000","last_active":"2024-04-25T05:55:32.588751+0000","last_peered":"2024-04-25T05:55:32.588751+0000","last_clean":"2024-04-25T05:55:32.588751+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:55:32.588751+0000","last_undegraded":"2024-04-25T05:55:32.588751+0000","last_fullsized":"2024-04-25T05:55:32.588751+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.515470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.515470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:27.679494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":624,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756273+0000","last_change":"2024-04-25T05:54:32.801763+0000","last_active":"2024-04-25T05:55:32.756273+0000","last_peered":"2024-04-25T05:55:32.756273+0000","last_clean":"2024-04-25T05:55:32.756273+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:55:32.756273+0000","last_undegraded":"2024-04-25T05:55:32.756273+0000","last_fullsized":"2024-04-25T05:55:32.756273+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'14","last_scrub_stamp":"2024-04-25T05:54:32.801648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.801648+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.540836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778909+0000","last_change":"2024-04-25T05:54:29.432405+0000","last_active":"2024-04-25T05:55:32.778909+0000","last_peered":"2024-04-25T05:55:32.778909+0000","last_clean":"2024-04-25T05:55:32.778909+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:55:32.778909+0000","last_undegraded":"2024-04-25T05:55:32.778909+0000","last_fullsized":"2024-04-25T05:55:32.778909+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:29.432303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.432303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.154796+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"581'32","reported_seq":1337,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243085+0000","last_change":"2024-04-25T05:54:46.258177+0000","last_active":"2024-04-25T05:55:33.243085+0000","last_peered":"2024-04-25T05:55:33.243085+0000","last_clean":"2024-04-25T05:55:33.243085+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:55:33.243085+0000","last_undegraded":"2024-04-25T05:55:33.243085+0000","last_fullsized":"2024-04-25T05:55:33.243085+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:54:46.258082+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.258082+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.893531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":779,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243775+0000","last_change":"2024-04-25T05:55:17.118213+0000","last_active":"2024-04-25T05:55:33.243775+0000","last_peered":"2024-04-25T05:55:33.243775+0000","last_clean":"2024-04-25T05:55:33.243775+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:55:33.243775+0000","last_undegraded":"2024-04-25T05:55:33.243775+0000","last_fullsized":"2024-04-25T05:55:33.243775+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:55:17.118107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:17.118107+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:44.384424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371266+0000","last_change":"2024-04-25T05:55:05.572752+0000","last_active":"2024-04-25T05:55:32.371266+0000","last_peered":"2024-04-25T05:55:32.371266+0000","last_clean":"2024-04-25T05:55:32.371266+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:55:32.371266+0000","last_undegraded":"2024-04-25T05:55:32.371266+0000","last_fullsized":"2024-04-25T05:55:32.371266+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:05.572644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.572644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:33.809245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1201,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371613+0000","last_change":"2024-04-25T05:54:58.863337+0000","last_active":"2024-04-25T05:55:32.371613+0000","last_peered":"2024-04-25T05:55:32.371613+0000","last_clean":"2024-04-25T05:55:32.371613+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:55:32.371613+0000","last_undegraded":"2024-04-25T05:55:32.371613+0000","last_fullsized":"2024-04-25T05:55:32.371613+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.863207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.863207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:15.678236+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"581'25","reported_seq":741,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371864+0000","last_change":"2024-04-25T05:54:30.437415+0000","last_active":"2024-04-25T05:55:32.371864+0000","last_peered":"2024-04-25T05:55:32.371864+0000","last_clean":"2024-04-25T05:55:32.371864+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:55:32.371864+0000","last_undegraded":"2024-04-25T05:55:32.371864+0000","last_fullsized":"2024-04-25T05:55:32.371864+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'25","last_scrub_stamp":"2024-04-25T05:54:30.437264+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.437264+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.390257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1184,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589573+0000","last_change":"2024-04-25T05:54:48.642072+0000","last_active":"2024-04-25T05:55:32.589573+0000","last_peered":"2024-04-25T05:55:32.589573+0000","last_clean":"2024-04-25T05:55:32.589573+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:55:32.589573+0000","last_undegraded":"2024-04-25T05:55:32.589573+0000","last_fullsized":"2024-04-25T05:55:32.589573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:48.641984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:54:48.641984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:12.941861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372578+0000","last_change":"2024-04-25T05:55:02.893757+0000","last_active":"2024-04-25T05:55:32.372578+0000","last_peered":"2024-04-25T05:55:32.372578+0000","last_clean":"2024-04-25T05:55:32.372578+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:55:32.372578+0000","last_undegraded":"2024-04-25T05:55:32.372578+0000","last_fullsized":"2024-04-25T05:55:32.372578+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:02.893671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:55:02.893671+0000","last_clean_scrub_stamp":"2024-04-25T05:55:02.893671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.595330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778143+0000","last_change":"2024-04-25T05:55:19.230360+0000","last_active":"2024-04-25T05:55:32.778143+0000","last_peered":"2024-04-25T05:55:32.778143+0000","last_clean":"2024-04-25T05:55:32.778143+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:55:32.778143+0000","last_undegraded":"2024-04-25T05:55:32.778143+0000","last_fullsized":"2024-04-25T05:55:32.778143+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:19.230225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:19.230225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.604414+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"582'39","reported_seq":1268,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371940+0000","last_change":"2024-04-25T05:55:08.765847+0000","last_active":"2024-04-25T05:55:32.371940+0000","last_peered":"2024-04-25T05:55:32.371940+0000","last_clean":"2024-04-25T05:55:32.371940+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:55:32.371940+0000","last_undegraded":"2024-04-25T05:55:32.371940+0000","last_fullsized":"2024-04-25T05:55:32.371940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'39","last_scrub_stamp":"2024-04-25T05:55:08.765698+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:55:08.765698+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.183048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756183+0000","last_change":"2024-04-25T05:54:35.811475+0000","last_active":"2024-04-25T05:55:32.756183+0000","last_peered":"2024-04-25T05:55:32.756183+0000","last_clean":"2024-04-25T05:55:32.756183+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:55:32.756183+0000","last_undegraded":"2024-04-25T05:55:32.756183+0000","last_fullsized":"2024-04-25T05:55:32.756183+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:35.811379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:35.811379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.744260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1195,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588652+0000","last_change":"2024-04-25T05:54:57.536648+0000","last_active":"2024-04-25T05:55:32.588652+0000","last_peered":"2024-04-25T05:55:32.588652+0000","last_clean":"2024-04-25T05:55:32.588652+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:55:32.588652+0000","last_undegraded":"2024-04-25T05:55:32.588652+0000","last_fullsized":"2024-04-25T05:55:32.588652+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:57.536536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:57.536536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:08.582347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372485+0000","last_change":"2024-04-25T05:54:06.026590+0000","last_active":"2024-04-25T05:55:32.372485+0000","last_peered":"2024-04-25T05:55:32.372485+0000","last_clean":"2024-04-25T05:55:32.372485+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:55:32.372485+0000","last_undegraded":"2024-04-25T05:55:32.372485+0000","last_fullsized":"2024-04-25T05:55:32.372485+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:06.026456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:06.026456+0000","last_clean_scrub_stamp":"2024-04-25T05:54:06.026456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"581'12","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.779003+0000","last_change":"2024-04-25T05:55:33.779033+0000","last_active":"2024-04-25T05:55:33.779003+0000","last_peered":"2024-04-25T05:55:33.779003+0000","last_clean":"2024-04-25T05:55:33.779003+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:55:33.779003+0000","last_undegraded":"2024-04-25T05:55:33.779003+0000","last_fullsized":"2024-04-25T05:55:33.779003+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'12","last_scrub_stamp":"2024-04-25T05:55:33.778925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:33.778925+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.063264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779031+0000","last_change":"2024-04-25T05:54:58.287258+0000","last_active":"2024-04-25T05:55:32.779031+0000","last_peered":"2024-04-25T05:55:32.779031+0000","last_clean":"2024-04-25T05:55:32.779031+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:55:32.779031+0000","last_undegraded":"2024-04-25T05:55:32.779031+0000","last_fullsized":"2024-04-25T05:55:32.779031+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.287131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.287131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:01.930630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":900,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.370804+0000","last_change":"2024-04-25T05:54:31.535891+0000","last_active":"2024-04-25T05:55:32.370804+0000","last_peered":"2024-04-25T05:55:32.370804+0000","last_clean":"2024-04-25T05:55:32.370804+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:55:32.370804+0000","last_undegraded":"2024-04-25T05:55:32.370804+0000","last_fullsized":"2024-04-25T05:55:32.370804+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:31.535778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:54:31.535778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.619524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"610'2","reported_seq":1195,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588782+0000","last_change":"2024-04-25T05:55:05.525112+0000","last_active":"2024-04-25T05:55:32.588782+0000","last_peered":"2024-04-25T05:55:32.588782+0000","last_clean":"2024-04-25T05:55:32.588782+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:55:32.588782+0000","last_undegraded":"2024-04-25T05:55:32.588782+0000","last_fullsized":"2024-04-25T05:55:32.588782+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'2","last_scrub_stamp":"2024-04-25T05:55:05.524993+0000","last_deep_scrub":"610'2","last_deep_scrub_stamp":"2024-04-25T05:55:05.524993+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.524993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:19.201505+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"582'39","reported_seq":1033,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372059+0000","last_change":"2024-04-25T05:54:49.441851+0000","last_active":"2024-04-25T05:55:32.372059+0000","last_peered":"2024-04-25T05:55:32.372059+0000","last_clean":"2024-04-25T05:55:32.372059+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:55:32.372059+0000","last_undegraded":"2024-04-25T05:55:32.372059+0000","last_fullsized":"2024-04-25T05:55:32.372059+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'39","last_scrub_stamp":"2024-04-25T05:54:49.441705+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:54:49.441705+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.417754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1200,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371196+0000","last_change":"2024-04-25T05:55:30.766563+0000","last_active":"2024-04-25T05:55:32.371196+0000","last_peered":"2024-04-25T05:55:32.371196+0000","last_clean":"2024-04-25T05:55:32.371196+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:55:32.371196+0000","last_undegraded":"2024-04-25T05:55:32.371196+0000","last_fullsized":"2024-04-25T05:55:32.371196+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:30.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:55:30.766478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:46.739046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778565+0000","last_change":"2024-04-25T05:54:54.305334+0000","last_active":"2024-04-25T05:55:32.778565+0000","last_peered":"2024-04-25T05:55:32.778565+0000","last_clean":"2024-04-25T05:55:32.778565+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:55:32.778565+0000","last_undegraded":"2024-04-25T05:55:32.778565+0000","last_fullsized":"2024-04-25T05:55:32.778565+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:54.305250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:54.305250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.261701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1183,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372879+0000","last_change":"2024-04-25T05:55:24.017478+0000","last_active":"2024-04-25T05:55:32.372879+0000","last_peered":"2024-04-25T05:55:32.372879+0000","last_clean":"2024-04-25T05:55:32.372879+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:55:32.372879+0000","last_undegraded":"2024-04-25T05:55:32.372879+0000","last_fullsized":"2024-04-25T05:55:32.372879+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:24.017335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:55:24.017335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.356248+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"582'42","reported_seq":290,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778798+0000","last_change":"2024-04-25T05:54:06.134244+0000","last_active":"2024-04-25T05:55:32.778798+0000","last_peered":"2024-04-25T05:55:32.778798+0000","last_clean":"2024-04-25T05:55:32.778798+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:55:32.778798+0000","last_undegraded":"2024-04-25T05:55:32.778798+0000","last_fullsized":"2024-04-25T05:55:32.778798+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:54:06.134150+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:54:06.134150+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1186,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371144+0000","last_change":"2024-04-25T05:55:26.803181+0000","last_active":"2024-04-25T05:55:32.371144+0000","last_peered":"2024-04-25T05:55:32.371144+0000","last_clean":"2024-04-25T05:55:32.371144+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:55:32.371144+0000","last_undegraded":"2024-04-25T05:55:32.371144+0000","last_fullsized":"2024-04-25T05:55:32.371144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:55:26.803087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:26.803087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.832618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":1105,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371824+0000","last_change":"2024-04-25T05:54:29.519271+0000","last_active":"2024-04-25T05:55:32.371824+0000","last_peered":"2024-04-25T05:55:32.371824+0000","last_clean":"2024-04-25T05:55:32.371824+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:55:32.371824+0000","last_undegraded":"2024-04-25T05:55:32.371824+0000","last_fullsized":"2024-04-25T05:55:32.371824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:54:29.519168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.519168+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:55.489245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"507'2","reported_seq":1214,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589165+0000","last_change":"2024-04-25T05:54:33.756679+0000","last_active":"2024-04-25T05:55:32.589165+0000","last_peered":"2024-04-25T05:55:32.589165+0000","last_clean":"2024-04-25T05:55:32.589165+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:55:32.589165+0000","last_undegraded":"2024-04-25T05:55:32.589165+0000","last_fullsized":"2024-04-25T05:55:32.589165+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-04-25T05:54:33.756467+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_clean_scrub_stamp":"2024-04-25T05:54:33.756467+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:47.408397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"614'36","reported_seq":1217,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372393+0000","last_change":"2024-04-25T05:55:15.821674+0000","last_active":"2024-04-25T05:55:32.372393+0000","last_peered":"2024-04-25T05:55:32.372393+0000","last_clean":"2024-04-25T05:55:32.372393+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:55:32.372393+0000","last_undegraded":"2024-04-25T05:55:32.372393+0000","last_fullsized":"2024-04-25T05:55:32.372393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'36","last_scrub_stamp":"2024-04-25T05:55:15.821567+0000","last_deep_scrub":"614'36","last_deep_scrub_stamp":"2024-04-25T05:55:15.821567+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.821567+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.944313+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1220,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244050+0000","last_change":"2024-04-25T05:54:50.243860+0000","last_active":"2024-04-25T05:55:33.244050+0000","last_peered":"2024-04-25T05:55:33.244050+0000","last_clean":"2024-04-25T05:55:33.244050+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:55:33.244050+0000","last_undegraded":"2024-04-25T05:55:33.244050+0000","last_fullsized":"2024-04-25T05:55:33.244050+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:50.243784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.243784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.382174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":277,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371137+0000","last_change":"2024-04-25T05:55:20.569637+0000","last_active":"2024-04-25T05:55:32.371137+0000","last_peered":"2024-04-25T05:55:32.371137+0000","last_clean":"2024-04-25T05:55:32.371137+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:55:32.371137+0000","last_undegraded":"2024-04-25T05:55:32.371137+0000","last_fullsized":"2024-04-25T05:55:32.371137+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:20.569509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.569509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:44.237659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1176,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373437+0000","last_change":"2024-04-25T05:54:57.108076+0000","last_active":"2024-04-25T05:55:32.373437+0000","last_peered":"2024-04-25T05:55:32.373437+0000","last_clean":"2024-04-25T05:55:32.373437+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:55:32.373437+0000","last_undegraded":"2024-04-25T05:55:32.373437+0000","last_fullsized":"2024-04-25T05:55:32.373437+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:57.107977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:54:57.107977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.554767+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"581'46","reported_seq":568,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585753+0000","last_change":"2024-04-25T05:54:26.442804+0000","last_active":"2024-04-25T05:55:25.585753+0000","last_peered":"2024-04-25T05:55:25.585753+0000","last_clean":"2024-04-25T05:55:25.585753+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:55:25.585753+0000","last_undegraded":"2024-04-25T05:55:25.585753+0000","last_fullsized":"2024-04-25T05:55:25.585753+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'46","last_scrub_stamp":"2024-04-25T05:54:26.442689+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.442689+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:36.083886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778732+0000","last_change":"2024-04-25T05:54:34.255892+0000","last_active":"2024-04-25T05:55:32.778732+0000","last_peered":"2024-04-25T05:55:32.778732+0000","last_clean":"2024-04-25T05:55:32.778732+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:55:32.778732+0000","last_undegraded":"2024-04-25T05:55:32.778732+0000","last_fullsized":"2024-04-25T05:55:32.778732+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.255784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.255784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:40.426091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1199,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372976+0000","last_change":"2024-04-25T05:54:34.454920+0000","last_active":"2024-04-25T05:55:32.372976+0000","last_peered":"2024-04-25T05:55:32.372976+0000","last_clean":"2024-04-25T05:55:32.372976+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:55:32.372976+0000","last_undegraded":"2024-04-25T05:55:32.372976+0000","last_fullsized":"2024-04-25T05:55:32.372976+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.454834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.454834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:42.196066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1216,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244380+0000","last_change":"2024-04-25T05:54:44.330594+0000","last_active":"2024-04-25T05:55:33.244380+0000","last_peered":"2024-04-25T05:55:33.244380+0000","last_clean":"2024-04-25T05:55:33.244380+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:55:33.244380+0000","last_undegraded":"2024-04-25T05:55:33.244380+0000","last_fullsized":"2024-04-25T05:55:33.244380+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.330464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.330464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:58.127089+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"581'15","reported_seq":697,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244718+0000","last_change":"2024-04-25T05:55:22.128467+0000","last_active":"2024-04-25T05:55:33.244718+0000","last_peered":"2024-04-25T05:55:33.244718+0000","last_clean":"2024-04-25T05:55:33.244718+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:55:33.244718+0000","last_undegraded":"2024-04-25T05:55:33.244718+0000","last_fullsized":"2024-04-25T05:55:33.244718+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'15","last_scrub_stamp":"2024-04-25T05:55:22.128353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:22.128353+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.768433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756167+0000","last_change":"2024-04-25T05:54:59.609213+0000","last_active":"2024-04-25T05:55:32.756167+0000","last_peered":"2024-04-25T05:55:32.756167+0000","last_clean":"2024-04-25T05:55:32.756167+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:55:32.756167+0000","last_undegraded":"2024-04-25T05:55:32.756167+0000","last_fullsized":"2024-04-25T05:55:32.756167+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.609135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.609135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.406325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1174,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588645+0000","last_change":"2024-04-25T05:54:30.693524+0000","last_active":"2024-04-25T05:55:32.588645+0000","last_peered":"2024-04-25T05:55:32.588645+0000","last_clean":"2024-04-25T05:55:32.588645+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:55:32.588645+0000","last_undegraded":"2024-04-25T05:55:32.588645+0000","last_fullsized":"2024-04-25T05:55:32.588645+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:30.693428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.693428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755518+0000","last_change":"2024-04-25T05:55:10.771336+0000","last_active":"2024-04-25T05:55:32.755518+0000","last_peered":"2024-04-25T05:55:32.755518+0000","last_clean":"2024-04-25T05:55:32.755518+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:55:32.755518+0000","last_undegraded":"2024-04-25T05:55:32.755518+0000","last_fullsized":"2024-04-25T05:55:32.755518+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:10.771219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:55:10.771219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:39.669662+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"581'40","reported_seq":1017,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244063+0000","last_change":"2024-04-25T05:55:20.110162+0000","last_active":"2024-04-25T05:55:33.244063+0000","last_peered":"2024-04-25T05:55:33.244063+0000","last_clean":"2024-04-25T05:55:33.244063+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:55:33.244063+0000","last_undegraded":"2024-04-25T05:55:33.244063+0000","last_fullsized":"2024-04-25T05:55:33.244063+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'40","last_scrub_stamp":"2024-04-25T05:55:20.110035+0000","last_deep_scrub":"581'40","last_deep_scrub_stamp":"2024-04-25T05:55:20.110035+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.110035+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.879437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756138+0000","last_change":"2024-04-25T05:54:41.775664+0000","last_active":"2024-04-25T05:55:32.756138+0000","last_peered":"2024-04-25T05:55:32.756138+0000","last_clean":"2024-04-25T05:55:32.756138+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:55:32.756138+0000","last_undegraded":"2024-04-25T05:55:32.756138+0000","last_fullsized":"2024-04-25T05:55:32.756138+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:41.775569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.775569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:44.061066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755657+0000","last_change":"2024-04-25T05:55:11.701662+0000","last_active":"2024-04-25T05:55:32.755657+0000","last_peered":"2024-04-25T05:55:32.755657+0000","last_clean":"2024-04-25T05:55:32.755657+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:55:32.755657+0000","last_undegraded":"2024-04-25T05:55:32.755657+0000","last_fullsized":"2024-04-25T05:55:32.755657+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:11.701557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:11.701557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:13.957487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778119+0000","last_change":"2024-04-25T05:53:54.281982+0000","last_active":"2024-04-25T05:55:32.778119+0000","last_peered":"2024-04-25T05:55:32.778119+0000","last_clean":"2024-04-25T05:55:32.778119+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:55:32.778119+0000","last_undegraded":"2024-04-25T05:55:32.778119+0000","last_fullsized":"2024-04-25T05:55:32.778119+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:54.281861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:53:54.281861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"582'63","reported_seq":1507,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371983+0000","last_change":"2024-04-25T05:54:59.857846+0000","last_active":"2024-04-25T05:55:32.371983+0000","last_peered":"2024-04-25T05:55:32.371983+0000","last_clean":"2024-04-25T05:55:32.371983+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:55:32.371983+0000","last_undegraded":"2024-04-25T05:55:32.371983+0000","last_fullsized":"2024-04-25T05:55:32.371983+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'63","last_scrub_stamp":"2024-04-25T05:54:59.857725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.857725+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:14.283169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372595+0000","last_change":"2024-04-25T05:53:59.778523+0000","last_active":"2024-04-25T05:55:32.372595+0000","last_peered":"2024-04-25T05:55:32.372595+0000","last_clean":"2024-04-25T05:55:32.372595+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:55:32.372595+0000","last_undegraded":"2024-04-25T05:55:32.372595+0000","last_fullsized":"2024-04-25T05:55:32.372595+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:59.778392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:53:59.778392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":749,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243337+0000","last_change":"2024-04-25T05:55:32.210629+0000","last_active":"2024-04-25T05:55:33.243337+0000","last_peered":"2024-04-25T05:55:33.243337+0000","last_clean":"2024-04-25T05:55:33.243337+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:55:33.243337+0000","last_undegraded":"2024-04-25T05:55:33.243337+0000","last_fullsized":"2024-04-25T05:55:33.243337+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.210503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.210503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:49.628442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1162,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372289+0000","last_change":"2024-04-25T05:54:34.080973+0000","last_active":"2024-04-25T05:55:32.372289+0000","last_peered":"2024-04-25T05:55:32.372289+0000","last_clean":"2024-04-25T05:55:32.372289+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:55:32.372289+0000","last_undegraded":"2024-04-25T05:55:32.372289+0000","last_fullsized":"2024-04-25T05:55:32.372289+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.080845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.080845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.146065+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"581'32","reported_seq":1285,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243193+0000","last_change":"2024-04-25T05:54:40.264714+0000","last_active":"2024-04-25T05:55:33.243193+0000","last_peered":"2024-04-25T05:55:33.243193+0000","last_clean":"2024-04-25T05:55:33.243193+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:55:33.243193+0000","last_undegraded":"2024-04-25T05:55:33.243193+0000","last_fullsized":"2024-04-25T05:55:33.243193+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:54:40.264562+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:54:40.264562+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:07.392246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779037+0000","last_change":"2024-04-25T05:55:03.284944+0000","last_active":"2024-04-25T05:55:32.779037+0000","last_peered":"2024-04-25T05:55:32.779037+0000","last_clean":"2024-04-25T05:55:32.779037+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:55:32.779037+0000","last_undegraded":"2024-04-25T05:55:32.779037+0000","last_fullsized":"2024-04-25T05:55:32.779037+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:03.284841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:03.284841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:29.572890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1187,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371268+0000","last_change":"2024-04-25T05:54:29.444628+0000","last_active":"2024-04-25T05:55:32.371268+0000","last_peered":"2024-04-25T05:55:32.371268+0000","last_clean":"2024-04-25T05:55:32.371268+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:55:32.371268+0000","last_undegraded":"2024-04-25T05:55:32.371268+0000","last_fullsized":"2024-04-25T05:55:32.371268+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:29.444552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.444552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.015362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":725,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243603+0000","last_change":"2024-04-25T05:55:25.211610+0000","last_active":"2024-04-25T05:55:33.243603+0000","last_peered":"2024-04-25T05:55:33.243603+0000","last_clean":"2024-04-25T05:55:33.243603+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:55:33.243603+0000","last_undegraded":"2024-04-25T05:55:33.243603+0000","last_fullsized":"2024-04-25T05:55:33.243603+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:25.211471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:25.211471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:25.450911+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":1270,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1818,"num_read_kb":1543,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070729,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931784,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92813368,"statfs":{"total":95995035648,"available":95040888832,"internally_reserved":0,"allocated":2424832,"data_stored":1492566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":951712202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365069,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867580,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92877572,"statfs":{"total":95995035648,"available":95106633728,"internally_reserved":0,"allocated":2080768,"data_stored":1170401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":886307270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129376,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":1127294,"kb_avail":92615776,"statfs":{"total":95995035648,"available":94838554624,"internally_reserved":0,"allocated":2121728,"data_stored":1170317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154349509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769036,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92976116,"statfs":{"total":95995035648,"available":95207542784,"internally_reserved":0,"allocated":2097152,"data_stored":1169766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727600,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017552,"statfs":{"total":95995035648,"available":95249973248,"internally_reserved":0,"allocated":2334720,"data_stored":1425318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149979,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1219928,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":1217278,"kb_avail":92525224,"statfs":{"total":95995035648,"available":94745829376,"internally_reserved":0,"allocated":2703360,"data_stored":1760231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1246493127},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986454,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827336,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92917816,"statfs":{"total":95995035648,"available":95147843584,"internally_reserved":0,"allocated":2097152,"data_stored":1163860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692114,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637684,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93107468,"statfs":{"total":95995035648,"available":95342047232,"internally_reserved":0,"allocated":2666496,"data_stored":1753522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:55:36.188 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T05:55:36.441 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:36.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[37492]: 3.b scrub starts 2024-04-25T05:55:36.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[37492]: 3.b scrub ok 2024-04-25T05:55:36.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[37492]: 6.18 scrub starts 2024-04-25T05:55:36.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[37492]: 6.18 scrub ok 2024-04-25T05:55:36.564 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[37492]: from='client.16995 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:36.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[27010]: 3.b scrub starts 2024-04-25T05:55:36.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[27010]: 3.b scrub ok 2024-04-25T05:55:36.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[27010]: 6.18 scrub starts 2024-04-25T05:55:36.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[27010]: 6.18 scrub ok 2024-04-25T05:55:36.565 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:36 smithi043 ceph-mon[27010]: from='client.16995 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:36 smithi060 ceph-mon[30898]: 3.b scrub starts 2024-04-25T05:55:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:36 smithi060 ceph-mon[30898]: 3.b scrub ok 2024-04-25T05:55:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:36 smithi060 ceph-mon[30898]: 6.18 scrub starts 2024-04-25T05:55:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:36 smithi060 ceph-mon[30898]: 6.18 scrub ok 2024-04-25T05:55:36.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:36 smithi060 ceph-mon[30898]: from='client.16995 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:37.201 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-25T05:55:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-25T05:55:37.482 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:37 smithi043 ceph-mon[27010]: pgmap v1177: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:37.483 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:37 smithi043 ceph-mon[37492]: pgmap v1177: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:37.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:37 smithi060 ceph-mon[30898]: pgmap v1177: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-25T05:55:38.148 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1177,"stamp":"2024-04-25T05:55:36.150635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3117,"num_read_kb":2837,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":544337,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1804,"ondisk_log_size":1804,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":427,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7110324,"kb_used_data":18092,"kb_used_omap":12,"kb_used_meta":7092147,"kb_avail":742850892,"statfs":{"total":767960285184,"available":760679313408,"internally_reserved":0,"allocated":18526208,"data_stored":11105981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":7262359088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002598"},"pg_stats":[{"pgid":"4.c","version":"581'30","reported_seq":686,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244055+0000","last_change":"2024-04-25T05:54:26.189530+0000","last_active":"2024-04-25T05:55:33.244055+0000","last_peered":"2024-04-25T05:55:33.244055+0000","last_clean":"2024-04-25T05:55:33.244055+0000","last_became_active":"2024-04-25T05:49:09.580395+0000","last_became_peered":"2024-04-25T05:49:09.580395+0000","last_unstale":"2024-04-25T05:55:33.244055+0000","last_undegraded":"2024-04-25T05:55:33.244055+0000","last_fullsized":"2024-04-25T05:55:33.244055+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'30","last_scrub_stamp":"2024-04-25T05:54:26.186309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.186309+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:45.156742+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1187,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372196+0000","last_change":"2024-04-25T05:53:57.738227+0000","last_active":"2024-04-25T05:55:32.372196+0000","last_peered":"2024-04-25T05:55:32.372196+0000","last_clean":"2024-04-25T05:55:32.372196+0000","last_became_active":"2024-04-25T05:51:40.099662+0000","last_became_peered":"2024-04-25T05:51:40.099662+0000","last_unstale":"2024-04-25T05:55:32.372196+0000","last_undegraded":"2024-04-25T05:55:32.372196+0000","last_fullsized":"2024-04-25T05:55:32.372196+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:57.736955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:53:57.736955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"388'14","reported_seq":659,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586052+0000","last_change":"2024-04-25T05:55:00.529284+0000","last_active":"2024-04-25T05:55:25.586052+0000","last_peered":"2024-04-25T05:55:25.586052+0000","last_clean":"2024-04-25T05:55:25.586052+0000","last_became_active":"2024-04-25T05:46:34.971483+0000","last_became_peered":"2024-04-25T05:46:34.971483+0000","last_unstale":"2024-04-25T05:55:25.586052+0000","last_undegraded":"2024-04-25T05:55:25.586052+0000","last_fullsized":"2024-04-25T05:55:25.586052+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'14","last_scrub_stamp":"2024-04-25T05:55:00.529198+0000","last_deep_scrub":"388'14","last_deep_scrub_stamp":"2024-04-25T05:53:57.528618+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.529198+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:03.248967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":756,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244584+0000","last_change":"2024-04-25T05:55:12.203013+0000","last_active":"2024-04-25T05:55:33.244584+0000","last_peered":"2024-04-25T05:55:33.244584+0000","last_clean":"2024-04-25T05:55:33.244584+0000","last_became_active":"2024-04-25T05:49:09.559034+0000","last_became_peered":"2024-04-25T05:49:09.559034+0000","last_unstale":"2024-04-25T05:55:33.244584+0000","last_undegraded":"2024-04-25T05:55:33.244584+0000","last_fullsized":"2024-04-25T05:55:33.244584+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:12.202870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:28.676660+0000","last_clean_scrub_stamp":"2024-04-25T05:55:12.202870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.010806+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1536,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588999+0000","last_change":"2024-04-25T05:55:15.450232+0000","last_active":"2024-04-25T05:55:32.588999+0000","last_peered":"2024-04-25T05:55:32.588999+0000","last_clean":"2024-04-25T05:55:32.588999+0000","last_became_active":"2024-04-25T05:46:23.148289+0000","last_became_peered":"2024-04-25T05:46:23.148289+0000","last_unstale":"2024-04-25T05:55:32.588999+0000","last_undegraded":"2024-04-25T05:55:32.588999+0000","last_fullsized":"2024-04-25T05:55:32.588999+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:15.450125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.450125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.386702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"581'53","reported_seq":701,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244535+0000","last_change":"2024-04-25T05:55:00.259864+0000","last_active":"2024-04-25T05:55:33.244535+0000","last_peered":"2024-04-25T05:55:33.244535+0000","last_clean":"2024-04-25T05:55:33.244535+0000","last_became_active":"2024-04-25T05:49:09.556522+0000","last_became_peered":"2024-04-25T05:49:09.556522+0000","last_unstale":"2024-04-25T05:55:33.244535+0000","last_undegraded":"2024-04-25T05:55:33.244535+0000","last_fullsized":"2024-04-25T05:55:33.244535+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'53","last_scrub_stamp":"2024-04-25T05:55:00.259736+0000","last_deep_scrub":"581'53","last_deep_scrub_stamp":"2024-04-25T05:55:00.259736+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.259736+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:05.573152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":909,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371257+0000","last_change":"2024-04-25T05:54:59.495532+0000","last_active":"2024-04-25T05:55:32.371257+0000","last_peered":"2024-04-25T05:55:32.371257+0000","last_clean":"2024-04-25T05:55:32.371257+0000","last_became_active":"2024-04-25T05:39:29.796907+0000","last_became_peered":"2024-04-25T05:39:29.796907+0000","last_unstale":"2024-04-25T05:55:32.371257+0000","last_undegraded":"2024-04-25T05:55:32.371257+0000","last_fullsized":"2024-04-25T05:55:32.371257+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.495451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:59.495451+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.495451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.576932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1200,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372760+0000","last_change":"2024-04-25T05:54:28.469911+0000","last_active":"2024-04-25T05:55:32.372760+0000","last_peered":"2024-04-25T05:55:32.372760+0000","last_clean":"2024-04-25T05:55:32.372760+0000","last_became_active":"2024-04-25T05:51:40.108927+0000","last_became_peered":"2024-04-25T05:51:40.108927+0000","last_unstale":"2024-04-25T05:55:32.372760+0000","last_undegraded":"2024-04-25T05:55:32.372760+0000","last_fullsized":"2024-04-25T05:55:32.372760+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:28.469809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:28.469809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:52.028889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":535,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586330+0000","last_change":"2024-04-25T05:54:50.675033+0000","last_active":"2024-04-25T05:55:25.586330+0000","last_peered":"2024-04-25T05:55:25.586330+0000","last_clean":"2024-04-25T05:55:25.586330+0000","last_became_active":"2024-04-25T05:46:20.011387+0000","last_became_peered":"2024-04-25T05:46:20.011387+0000","last_unstale":"2024-04-25T05:55:25.586330+0000","last_undegraded":"2024-04-25T05:55:25.586330+0000","last_fullsized":"2024-04-25T05:55:25.586330+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:50.674927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.674927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:51.561974+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"405'129","reported_seq":1135,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755345+0000","last_change":"2024-04-25T05:55:29.769300+0000","last_active":"2024-04-25T05:55:32.755345+0000","last_peered":"2024-04-25T05:55:32.755345+0000","last_clean":"2024-04-25T05:55:32.755345+0000","last_became_active":"2024-04-25T05:46:37.980838+0000","last_became_peered":"2024-04-25T05:46:37.980838+0000","last_unstale":"2024-04-25T05:55:32.755345+0000","last_undegraded":"2024-04-25T05:55:32.755345+0000","last_fullsized":"2024-04-25T05:55:32.755345+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"405'129","last_scrub_stamp":"2024-04-25T05:55:29.769188+0000","last_deep_scrub":"405'129","last_deep_scrub_stamp":"2024-04-25T05:47:25.988268+0000","last_clean_scrub_stamp":"2024-04-25T05:55:29.769188+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:30.899875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"581'33","reported_seq":1000,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.245992+0000","last_change":"2024-04-25T05:55:33.246042+0000","last_active":"2024-04-25T05:55:33.245992+0000","last_peered":"2024-04-25T05:55:33.245992+0000","last_clean":"2024-04-25T05:55:33.245992+0000","last_became_active":"2024-04-25T05:49:09.560844+0000","last_became_peered":"2024-04-25T05:49:09.560844+0000","last_unstale":"2024-04-25T05:55:33.245992+0000","last_undegraded":"2024-04-25T05:55:33.245992+0000","last_fullsized":"2024-04-25T05:55:33.245992+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'33","last_scrub_stamp":"2024-04-25T05:55:33.245892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:33.245892+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:57:02.767474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243451+0000","last_change":"2024-04-25T05:54:58.287691+0000","last_active":"2024-04-25T05:55:33.243451+0000","last_peered":"2024-04-25T05:55:33.243451+0000","last_clean":"2024-04-25T05:55:33.243451+0000","last_became_active":"2024-04-25T05:46:19.962418+0000","last_became_peered":"2024-04-25T05:46:19.962418+0000","last_unstale":"2024-04-25T05:55:33.243451+0000","last_undegraded":"2024-04-25T05:55:33.243451+0000","last_fullsized":"2024-04-25T05:55:33.243451+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.287555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:53:29.887008+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.287555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.327665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756547+0000","last_change":"2024-04-25T05:55:07.677768+0000","last_active":"2024-04-25T05:55:32.756547+0000","last_peered":"2024-04-25T05:55:32.756547+0000","last_clean":"2024-04-25T05:55:32.756547+0000","last_became_active":"2024-04-25T05:46:34.979731+0000","last_became_peered":"2024-04-25T05:46:34.979731+0000","last_unstale":"2024-04-25T05:55:32.756547+0000","last_undegraded":"2024-04-25T05:55:32.756547+0000","last_fullsized":"2024-04-25T05:55:32.756547+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:07.677664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:44:57.457842+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.677664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:17.643226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1175,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372465+0000","last_change":"2024-04-25T05:54:44.855689+0000","last_active":"2024-04-25T05:55:32.372465+0000","last_peered":"2024-04-25T05:55:32.372465+0000","last_clean":"2024-04-25T05:55:32.372465+0000","last_became_active":"2024-04-25T05:46:34.984674+0000","last_became_peered":"2024-04-25T05:46:34.984674+0000","last_unstale":"2024-04-25T05:55:32.372465+0000","last_undegraded":"2024-04-25T05:55:32.372465+0000","last_fullsized":"2024-04-25T05:55:32.372465+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.855611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.855611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.985647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"582'46","reported_seq":1242,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371435+0000","last_change":"2024-04-25T05:55:01.543726+0000","last_active":"2024-04-25T05:55:32.371435+0000","last_peered":"2024-04-25T05:55:32.371435+0000","last_clean":"2024-04-25T05:55:32.371435+0000","last_became_active":"2024-04-25T05:34:00.691181+0000","last_became_peered":"2024-04-25T05:34:00.691181+0000","last_unstale":"2024-04-25T05:55:32.371435+0000","last_undegraded":"2024-04-25T05:55:32.371435+0000","last_fullsized":"2024-04-25T05:55:32.371435+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'46","last_scrub_stamp":"2024-04-25T05:55:01.543612+0000","last_deep_scrub":"582'46","last_deep_scrub_stamp":"2024-04-25T05:55:01.543612+0000","last_clean_scrub_stamp":"2024-04-25T05:55:01.543612+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:06.221133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1171,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371395+0000","last_change":"2024-04-25T05:55:09.783532+0000","last_active":"2024-04-25T05:55:32.371395+0000","last_peered":"2024-04-25T05:55:32.371395+0000","last_clean":"2024-04-25T05:55:32.371395+0000","last_became_active":"2024-04-25T05:33:58.681355+0000","last_became_peered":"2024-04-25T05:33:58.681355+0000","last_unstale":"2024-04-25T05:55:32.371395+0000","last_undegraded":"2024-04-25T05:55:32.371395+0000","last_fullsized":"2024-04-25T05:55:32.371395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:09.783445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:09.783445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.266015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"501'17","reported_seq":909,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244532+0000","last_change":"2024-04-25T05:55:07.198140+0000","last_active":"2024-04-25T05:55:33.244532+0000","last_peered":"2024-04-25T05:55:33.244532+0000","last_clean":"2024-04-25T05:55:33.244532+0000","last_became_active":"2024-04-25T05:51:40.098101+0000","last_became_peered":"2024-04-25T05:51:40.098101+0000","last_unstale":"2024-04-25T05:55:33.244532+0000","last_undegraded":"2024-04-25T05:55:33.244532+0000","last_fullsized":"2024-04-25T05:55:33.244532+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:55:07.198024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.198024+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.769365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756412+0000","last_change":"2024-04-25T05:55:00.668485+0000","last_active":"2024-04-25T05:55:32.756412+0000","last_peered":"2024-04-25T05:55:32.756412+0000","last_clean":"2024-04-25T05:55:32.756412+0000","last_became_active":"2024-04-25T05:51:40.104917+0000","last_became_peered":"2024-04-25T05:51:40.104917+0000","last_unstale":"2024-04-25T05:55:32.756412+0000","last_undegraded":"2024-04-25T05:55:32.756412+0000","last_fullsized":"2024-04-25T05:55:32.756412+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.668373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:31.796754+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.668373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:21.496320+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":1091,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755853+0000","last_change":"2024-04-25T05:54:40.775438+0000","last_active":"2024-04-25T05:55:32.755853+0000","last_peered":"2024-04-25T05:55:32.755853+0000","last_clean":"2024-04-25T05:55:32.755853+0000","last_became_active":"2024-04-25T05:46:37.981912+0000","last_became_peered":"2024-04-25T05:46:37.981912+0000","last_unstale":"2024-04-25T05:55:32.755853+0000","last_undegraded":"2024-04-25T05:55:32.755853+0000","last_fullsized":"2024-04-25T05:55:32.755853+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:40.775348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:54:40.775348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.684559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243439+0000","last_change":"2024-04-25T05:54:42.286100+0000","last_active":"2024-04-25T05:55:33.243439+0000","last_peered":"2024-04-25T05:55:33.243439+0000","last_clean":"2024-04-25T05:55:33.243439+0000","last_became_active":"2024-04-25T05:34:02.699170+0000","last_became_peered":"2024-04-25T05:34:02.699170+0000","last_unstale":"2024-04-25T05:55:33.243439+0000","last_undegraded":"2024-04-25T05:55:33.243439+0000","last_fullsized":"2024-04-25T05:55:33.243439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:42.286019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:42.286019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.929826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"581'54","reported_seq":1372,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371484+0000","last_change":"2024-04-25T05:55:16.816884+0000","last_active":"2024-04-25T05:55:32.371484+0000","last_peered":"2024-04-25T05:55:32.371484+0000","last_clean":"2024-04-25T05:55:32.371484+0000","last_became_active":"2024-04-25T05:51:40.082843+0000","last_became_peered":"2024-04-25T05:51:40.082843+0000","last_unstale":"2024-04-25T05:55:32.371484+0000","last_undegraded":"2024-04-25T05:55:32.371484+0000","last_fullsized":"2024-04-25T05:55:32.371484+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'54","last_scrub_stamp":"2024-04-25T05:55:16.816759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:16.816759+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.887708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":74,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373276+0000","last_change":"2024-04-25T05:55:21.685763+0000","last_active":"2024-04-25T05:55:32.373276+0000","last_peered":"2024-04-25T05:55:32.373276+0000","last_clean":"2024-04-25T05:55:32.373276+0000","last_became_active":"2024-04-25T05:51:40.083844+0000","last_became_peered":"2024-04-25T05:51:40.083844+0000","last_unstale":"2024-04-25T05:55:32.373276+0000","last_undegraded":"2024-04-25T05:55:32.373276+0000","last_fullsized":"2024-04-25T05:55:32.373276+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:21.685639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:21.685639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.568034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":743,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243626+0000","last_change":"2024-04-25T05:54:28.228531+0000","last_active":"2024-04-25T05:55:33.243626+0000","last_peered":"2024-04-25T05:55:33.243626+0000","last_clean":"2024-04-25T05:55:33.243626+0000","last_became_active":"2024-04-25T05:49:09.553392+0000","last_became_peered":"2024-04-25T05:49:09.553392+0000","last_unstale":"2024-04-25T05:55:33.243626+0000","last_undegraded":"2024-04-25T05:55:33.243626+0000","last_fullsized":"2024-04-25T05:55:33.243626+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:28.228445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:28.228445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.356012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":1466,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243785+0000","last_change":"2024-04-25T05:54:41.265026+0000","last_active":"2024-04-25T05:55:33.243785+0000","last_peered":"2024-04-25T05:55:33.243785+0000","last_clean":"2024-04-25T05:55:33.243785+0000","last_became_active":"2024-04-25T05:52:47.115419+0000","last_became_peered":"2024-04-25T05:52:47.115419+0000","last_unstale":"2024-04-25T05:55:33.243785+0000","last_undegraded":"2024-04-25T05:55:33.243785+0000","last_fullsized":"2024-04-25T05:55:33.243785+0000","mapping_epoch":379,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:41.264918+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:44:14.223066+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.264918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.305989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"353'9","reported_seq":1442,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371962+0000","last_change":"2024-04-25T05:55:27.750052+0000","last_active":"2024-04-25T05:55:32.371962+0000","last_peered":"2024-04-25T05:55:32.371962+0000","last_clean":"2024-04-25T05:55:32.371962+0000","last_became_active":"2024-04-25T05:45:36.198733+0000","last_became_peered":"2024-04-25T05:45:36.198733+0000","last_unstale":"2024-04-25T05:55:32.371962+0000","last_undegraded":"2024-04-25T05:55:32.371962+0000","last_fullsized":"2024-04-25T05:55:32.371962+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'9","last_scrub_stamp":"2024-04-25T05:55:27.749966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:27.749966+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.544075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"582'48","reported_seq":894,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243711+0000","last_change":"2024-04-25T05:54:32.250833+0000","last_active":"2024-04-25T05:55:33.243711+0000","last_peered":"2024-04-25T05:55:33.243711+0000","last_clean":"2024-04-25T05:55:33.243711+0000","last_became_active":"2024-04-25T05:49:09.598095+0000","last_became_peered":"2024-04-25T05:49:09.598095+0000","last_unstale":"2024-04-25T05:55:33.243711+0000","last_undegraded":"2024-04-25T05:55:33.243711+0000","last_fullsized":"2024-04-25T05:55:33.243711+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'48","last_scrub_stamp":"2024-04-25T05:54:32.250707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.250707+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.911453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":63,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"98'2","reported_seq":273,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778682+0000","last_change":"2024-04-25T05:55:05.201807+0000","last_active":"2024-04-25T05:55:32.778682+0000","last_peered":"2024-04-25T05:55:32.778682+0000","last_clean":"2024-04-25T05:55:32.778682+0000","last_became_active":"2024-04-25T05:51:40.102231+0000","last_became_peered":"2024-04-25T05:51:40.102231+0000","last_unstale":"2024-04-25T05:55:32.778682+0000","last_undegraded":"2024-04-25T05:55:32.778682+0000","last_fullsized":"2024-04-25T05:55:32.778682+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'2","last_scrub_stamp":"2024-04-25T05:55:05.201722+0000","last_deep_scrub":"98'2","last_deep_scrub_stamp":"2024-04-25T05:54:01.147559+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.201722+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:23.179419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":561,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372208+0000","last_change":"2024-04-25T05:55:00.772872+0000","last_active":"2024-04-25T05:55:32.372208+0000","last_peered":"2024-04-25T05:55:32.372208+0000","last_clean":"2024-04-25T05:55:32.372208+0000","last_became_active":"2024-04-25T05:46:19.991290+0000","last_became_peered":"2024-04-25T05:46:19.991290+0000","last_unstale":"2024-04-25T05:55:32.372208+0000","last_undegraded":"2024-04-25T05:55:32.372208+0000","last_fullsized":"2024-04-25T05:55:32.372208+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.772758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.772758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.545490+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":1432,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372297+0000","last_change":"2024-04-25T05:55:31.750049+0000","last_active":"2024-04-25T05:55:32.372297+0000","last_peered":"2024-04-25T05:55:32.372297+0000","last_clean":"2024-04-25T05:55:32.372297+0000","last_became_active":"2024-04-25T05:52:51.531591+0000","last_became_peered":"2024-04-25T05:52:51.531591+0000","last_unstale":"2024-04-25T05:55:32.372297+0000","last_undegraded":"2024-04-25T05:55:32.372297+0000","last_fullsized":"2024-04-25T05:55:32.372297+0000","mapping_epoch":387,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:31.749899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:55:31.749899+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.988148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"501'17","reported_seq":319,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778834+0000","last_change":"2024-04-25T05:55:22.205572+0000","last_active":"2024-04-25T05:55:32.778834+0000","last_peered":"2024-04-25T05:55:32.778834+0000","last_clean":"2024-04-25T05:55:32.778834+0000","last_became_active":"2024-04-25T05:51:40.086132+0000","last_became_peered":"2024-04-25T05:51:40.086132+0000","last_unstale":"2024-04-25T05:55:32.778834+0000","last_undegraded":"2024-04-25T05:55:32.778834+0000","last_fullsized":"2024-04-25T05:55:32.778834+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'17","last_scrub_stamp":"2024-04-25T05:55:22.205480+0000","last_deep_scrub":"388'16","last_deep_scrub_stamp":"2024-04-25T05:49:06.550555+0000","last_clean_scrub_stamp":"2024-04-25T05:55:22.205480+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:39.004832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"581'42","reported_seq":936,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371436+0000","last_change":"2024-04-25T05:54:36.487946+0000","last_active":"2024-04-25T05:55:32.371436+0000","last_peered":"2024-04-25T05:55:32.371436+0000","last_clean":"2024-04-25T05:55:32.371436+0000","last_became_active":"2024-04-25T05:46:34.974499+0000","last_became_peered":"2024-04-25T05:46:34.974499+0000","last_unstale":"2024-04-25T05:55:32.371436+0000","last_undegraded":"2024-04-25T05:55:32.371436+0000","last_fullsized":"2024-04-25T05:55:32.371436+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'42","last_scrub_stamp":"2024-04-25T05:54:36.487842+0000","last_deep_scrub":"195'22","last_deep_scrub_stamp":"2024-04-25T05:42:18.092050+0000","last_clean_scrub_stamp":"2024-04-25T05:54:36.487842+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:42.043257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":512,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756421+0000","last_change":"2024-04-25T05:54:34.810815+0000","last_active":"2024-04-25T05:55:32.756421+0000","last_peered":"2024-04-25T05:55:32.756421+0000","last_clean":"2024-04-25T05:55:32.756421+0000","last_became_active":"2024-04-25T05:46:34.985938+0000","last_became_peered":"2024-04-25T05:46:34.985938+0000","last_unstale":"2024-04-25T05:55:32.756421+0000","last_undegraded":"2024-04-25T05:55:32.756421+0000","last_fullsized":"2024-04-25T05:55:32.756421+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.810673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.810673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:39.599925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778279+0000","last_change":"2024-04-25T05:55:15.149927+0000","last_active":"2024-04-25T05:55:32.778279+0000","last_peered":"2024-04-25T05:55:32.778279+0000","last_clean":"2024-04-25T05:55:32.778279+0000","last_became_active":"2024-04-25T05:51:40.093687+0000","last_became_peered":"2024-04-25T05:51:40.093687+0000","last_unstale":"2024-04-25T05:55:32.778279+0000","last_undegraded":"2024-04-25T05:55:32.778279+0000","last_fullsized":"2024-04-25T05:55:32.778279+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.149819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.149819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:36.742334+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":1421,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371865+0000","last_change":"2024-04-25T05:54:52.483978+0000","last_active":"2024-04-25T05:55:32.371865+0000","last_peered":"2024-04-25T05:55:32.371865+0000","last_clean":"2024-04-25T05:55:32.371865+0000","last_became_active":"2024-04-25T05:52:39.109573+0000","last_became_peered":"2024-04-25T05:52:39.109573+0000","last_unstale":"2024-04-25T05:55:32.371865+0000","last_undegraded":"2024-04-25T05:55:32.371865+0000","last_fullsized":"2024-04-25T05:55:32.371865+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:52.483888+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:54:52.483888+0000","last_clean_scrub_stamp":"2024-04-25T05:54:52.483888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:56.751854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":945,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371378+0000","last_change":"2024-04-25T05:54:55.473987+0000","last_active":"2024-04-25T05:55:32.371378+0000","last_peered":"2024-04-25T05:55:32.371378+0000","last_clean":"2024-04-25T05:55:32.371378+0000","last_became_active":"2024-04-25T05:51:40.104708+0000","last_became_peered":"2024-04-25T05:51:40.104708+0000","last_unstale":"2024-04-25T05:55:32.371378+0000","last_undegraded":"2024-04-25T05:55:32.371378+0000","last_fullsized":"2024-04-25T05:55:32.371378+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:55.473897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:21.964139+0000","last_clean_scrub_stamp":"2024-04-25T05:54:55.473897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:07.847727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"581'45","reported_seq":304,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778769+0000","last_change":"2024-04-25T05:54:47.240397+0000","last_active":"2024-04-25T05:55:32.778769+0000","last_peered":"2024-04-25T05:55:32.778769+0000","last_clean":"2024-04-25T05:55:32.778769+0000","last_became_active":"2024-04-25T05:51:40.099400+0000","last_became_peered":"2024-04-25T05:51:40.099400+0000","last_unstale":"2024-04-25T05:55:32.778769+0000","last_undegraded":"2024-04-25T05:55:32.778769+0000","last_fullsized":"2024-04-25T05:55:32.778769+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'45","last_scrub_stamp":"2024-04-25T05:54:47.240244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:47.240244+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:12.021917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1217,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372600+0000","last_change":"2024-04-25T05:55:14.679679+0000","last_active":"2024-04-25T05:55:32.372600+0000","last_peered":"2024-04-25T05:55:32.372600+0000","last_clean":"2024-04-25T05:55:32.372600+0000","last_became_active":"2024-04-25T05:46:34.974119+0000","last_became_peered":"2024-04-25T05:46:34.974119+0000","last_unstale":"2024-04-25T05:55:32.372600+0000","last_undegraded":"2024-04-25T05:55:32.372600+0000","last_fullsized":"2024-04-25T05:55:32.372600+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:14.679597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:14.679597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.036407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1191,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372158+0000","last_change":"2024-04-25T05:54:41.480708+0000","last_active":"2024-04-25T05:55:32.372158+0000","last_peered":"2024-04-25T05:55:32.372158+0000","last_clean":"2024-04-25T05:55:32.372158+0000","last_became_active":"2024-04-25T05:46:19.973957+0000","last_became_peered":"2024-04-25T05:46:19.973957+0000","last_unstale":"2024-04-25T05:55:32.372158+0000","last_undegraded":"2024-04-25T05:55:32.372158+0000","last_fullsized":"2024-04-25T05:55:32.372158+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:41.480591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.480591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.057402+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":1448,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371737+0000","last_change":"2024-04-25T05:55:08.557057+0000","last_active":"2024-04-25T05:55:32.371737+0000","last_peered":"2024-04-25T05:55:32.371737+0000","last_clean":"2024-04-25T05:55:32.371737+0000","last_became_active":"2024-04-25T05:52:43.436401+0000","last_became_peered":"2024-04-25T05:52:43.436401+0000","last_unstale":"2024-04-25T05:55:32.371737+0000","last_undegraded":"2024-04-25T05:55:32.371737+0000","last_fullsized":"2024-04-25T05:55:32.371737+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:55:08.556938+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:55:08.556938+0000","last_clean_scrub_stamp":"2024-04-25T05:55:08.556938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.347835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":518,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756594+0000","last_change":"2024-04-25T05:54:44.773940+0000","last_active":"2024-04-25T05:55:32.756594+0000","last_peered":"2024-04-25T05:55:32.756594+0000","last_clean":"2024-04-25T05:55:32.756594+0000","last_became_active":"2024-04-25T05:51:40.105034+0000","last_became_peered":"2024-04-25T05:51:40.105034+0000","last_unstale":"2024-04-25T05:55:32.756594+0000","last_undegraded":"2024-04-25T05:55:32.756594+0000","last_fullsized":"2024-04-25T05:55:32.756594+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.773835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:44.773835+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.773835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.593004+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"581'36","reported_seq":215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778311+0000","last_change":"2024-04-25T05:54:41.168240+0000","last_active":"2024-04-25T05:55:32.778311+0000","last_peered":"2024-04-25T05:55:32.778311+0000","last_clean":"2024-04-25T05:55:32.778311+0000","last_became_active":"2024-04-25T05:52:43.437539+0000","last_became_peered":"2024-04-25T05:52:43.437539+0000","last_unstale":"2024-04-25T05:55:32.778311+0000","last_undegraded":"2024-04-25T05:55:32.778311+0000","last_fullsized":"2024-04-25T05:55:32.778311+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'36","last_scrub_stamp":"2024-04-25T05:54:41.168118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.168118+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.078238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2515,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588390+0000","last_change":"2024-04-25T05:53:53.683465+0000","last_active":"2024-04-25T05:55:32.588390+0000","last_peered":"2024-04-25T05:55:32.588390+0000","last_clean":"2024-04-25T05:55:32.588390+0000","last_became_active":"2024-04-25T05:51:40.081920+0000","last_became_peered":"2024-04-25T05:51:40.081920+0000","last_unstale":"2024-04-25T05:55:32.588390+0000","last_undegraded":"2024-04-25T05:55:32.588390+0000","last_fullsized":"2024-04-25T05:55:32.588390+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-04-25T05:53:53.683339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:53:53.683339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0011947920000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1275,"num_read_kb":1270,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778546+0000","last_change":"2024-04-25T05:54:48.225902+0000","last_active":"2024-04-25T05:55:32.778546+0000","last_peered":"2024-04-25T05:55:32.778546+0000","last_clean":"2024-04-25T05:55:32.778546+0000","last_became_active":"2024-04-25T05:51:40.097535+0000","last_became_peered":"2024-04-25T05:51:40.097535+0000","last_unstale":"2024-04-25T05:55:32.778546+0000","last_undegraded":"2024-04-25T05:55:32.778546+0000","last_fullsized":"2024-04-25T05:55:32.778546+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:48.225753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:25.731370+0000","last_clean_scrub_stamp":"2024-04-25T05:54:48.225753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.650917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1179,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372376+0000","last_change":"2024-04-25T05:54:35.488312+0000","last_active":"2024-04-25T05:55:32.372376+0000","last_peered":"2024-04-25T05:55:32.372376+0000","last_clean":"2024-04-25T05:55:32.372376+0000","last_became_active":"2024-04-25T05:46:34.974280+0000","last_became_peered":"2024-04-25T05:46:34.974280+0000","last_unstale":"2024-04-25T05:55:32.372376+0000","last_undegraded":"2024-04-25T05:55:32.372376+0000","last_fullsized":"2024-04-25T05:55:32.372376+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:35.488226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:35.488226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.587910+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":1428,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372102+0000","last_change":"2024-04-25T05:54:38.471994+0000","last_active":"2024-04-25T05:55:32.372102+0000","last_peered":"2024-04-25T05:55:32.372102+0000","last_clean":"2024-04-25T05:55:32.372102+0000","last_became_active":"2024-04-25T05:52:55.109763+0000","last_became_peered":"2024-04-25T05:52:55.109763+0000","last_unstale":"2024-04-25T05:55:32.372102+0000","last_undegraded":"2024-04-25T05:55:32.372102+0000","last_fullsized":"2024-04-25T05:55:32.372102+0000","mapping_epoch":206,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":552,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:38.471909+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:28.990556+0000","last_clean_scrub_stamp":"2024-04-25T05:54:38.471909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:51.162526+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1208,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589202+0000","last_change":"2024-04-25T05:55:04.510991+0000","last_active":"2024-04-25T05:55:32.589202+0000","last_peered":"2024-04-25T05:55:32.589202+0000","last_clean":"2024-04-25T05:55:32.589202+0000","last_became_active":"2024-04-25T05:46:34.981209+0000","last_became_peered":"2024-04-25T05:46:34.981209+0000","last_unstale":"2024-04-25T05:55:32.589202+0000","last_undegraded":"2024-04-25T05:55:32.589202+0000","last_fullsized":"2024-04-25T05:55:32.589202+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:04.510872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:03.680381+0000","last_clean_scrub_stamp":"2024-04-25T05:55:04.510872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:22.322702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"581'51","reported_seq":1155,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371839+0000","last_change":"2024-04-25T05:54:26.423017+0000","last_active":"2024-04-25T05:55:32.371839+0000","last_peered":"2024-04-25T05:55:32.371839+0000","last_clean":"2024-04-25T05:55:32.371839+0000","last_became_active":"2024-04-25T05:39:29.802535+0000","last_became_peered":"2024-04-25T05:39:29.802535+0000","last_unstale":"2024-04-25T05:55:32.371839+0000","last_undegraded":"2024-04-25T05:55:32.371839+0000","last_fullsized":"2024-04-25T05:55:32.371839+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'51","last_scrub_stamp":"2024-04-25T05:54:26.422816+0000","last_deep_scrub":"457'39","last_deep_scrub_stamp":"2024-04-25T05:50:13.027901+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.422816+0000","objects_scrubbed":8,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:44.329991+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779141+0000","last_change":"2024-04-25T05:55:32.779181+0000","last_active":"2024-04-25T05:55:32.779141+0000","last_peered":"2024-04-25T05:55:32.779141+0000","last_clean":"2024-04-25T05:55:32.779141+0000","last_became_active":"2024-04-25T05:46:34.970259+0000","last_became_peered":"2024-04-25T05:46:34.970259+0000","last_unstale":"2024-04-25T05:55:32.779141+0000","last_undegraded":"2024-04-25T05:55:32.779141+0000","last_fullsized":"2024-04-25T05:55:32.779141+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.779056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.779056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.628666+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1553,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588440+0000","last_change":"2024-04-25T05:54:38.761794+0000","last_active":"2024-04-25T05:55:32.588440+0000","last_peered":"2024-04-25T05:55:32.588440+0000","last_clean":"2024-04-25T05:55:32.588440+0000","last_became_active":"2024-04-25T05:52:35.091072+0000","last_became_peered":"2024-04-25T05:52:35.091072+0000","last_unstale":"2024-04-25T05:55:32.588440+0000","last_undegraded":"2024-04-25T05:55:32.588440+0000","last_fullsized":"2024-04-25T05:55:32.588440+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":19,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:38.761693+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:42:18.235819+0000","last_clean_scrub_stamp":"2024-04-25T05:54:38.761693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.195038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":485,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373738+0000","last_change":"2024-04-25T05:55:29.127381+0000","last_active":"2024-04-25T05:55:32.373738+0000","last_peered":"2024-04-25T05:55:32.373738+0000","last_clean":"2024-04-25T05:55:32.373738+0000","last_became_active":"2024-04-25T05:47:02.157170+0000","last_became_peered":"2024-04-25T05:47:02.157170+0000","last_unstale":"2024-04-25T05:55:32.373738+0000","last_undegraded":"2024-04-25T05:55:32.373738+0000","last_fullsized":"2024-04-25T05:55:32.373738+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:29.127262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:55:29.127262+0000","last_clean_scrub_stamp":"2024-04-25T05:55:29.127262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:42.796405+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":541,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.586234+0000","last_change":"2024-04-25T05:55:17.794404+0000","last_active":"2024-04-25T05:55:25.586234+0000","last_peered":"2024-04-25T05:55:25.586234+0000","last_clean":"2024-04-25T05:55:25.586234+0000","last_became_active":"2024-04-25T05:46:34.970756+0000","last_became_peered":"2024-04-25T05:46:34.970756+0000","last_unstale":"2024-04-25T05:55:25.586234+0000","last_undegraded":"2024-04-25T05:55:25.586234+0000","last_fullsized":"2024-04-25T05:55:25.586234+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:17.794316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:17.794316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:37.671449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"581'33","reported_seq":1351,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371318+0000","last_change":"2024-04-25T05:55:20.756691+0000","last_active":"2024-04-25T05:55:32.371318+0000","last_peered":"2024-04-25T05:55:32.371318+0000","last_clean":"2024-04-25T05:55:32.371318+0000","last_became_active":"2024-04-25T05:51:40.111823+0000","last_became_peered":"2024-04-25T05:51:40.111823+0000","last_unstale":"2024-04-25T05:55:32.371318+0000","last_undegraded":"2024-04-25T05:55:32.371318+0000","last_fullsized":"2024-04-25T05:55:32.371318+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'33","last_scrub_stamp":"2024-04-25T05:55:20.756598+0000","last_deep_scrub":"98'25","last_deep_scrub_stamp":"2024-04-25T05:40:14.592911+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.756598+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.540095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":541,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585008+0000","last_change":"2024-04-25T05:54:43.418290+0000","last_active":"2024-04-25T05:55:25.585008+0000","last_peered":"2024-04-25T05:55:25.585008+0000","last_clean":"2024-04-25T05:55:25.585008+0000","last_became_active":"2024-04-25T05:51:40.106282+0000","last_became_peered":"2024-04-25T05:51:40.106282+0000","last_unstale":"2024-04-25T05:55:25.585008+0000","last_undegraded":"2024-04-25T05:55:25.585008+0000","last_fullsized":"2024-04-25T05:55:25.585008+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:43.418207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:56.651479+0000","last_clean_scrub_stamp":"2024-04-25T05:54:43.418207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.125299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010861829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1215,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371196+0000","last_change":"2024-04-25T05:54:27.494562+0000","last_active":"2024-04-25T05:55:32.371196+0000","last_peered":"2024-04-25T05:55:32.371196+0000","last_clean":"2024-04-25T05:55:32.371196+0000","last_became_active":"2024-04-25T05:46:19.988161+0000","last_became_peered":"2024-04-25T05:46:19.988161+0000","last_unstale":"2024-04-25T05:55:32.371196+0000","last_undegraded":"2024-04-25T05:55:32.371196+0000","last_fullsized":"2024-04-25T05:55:32.371196+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:27.494478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:27.494478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:39.511823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1186,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589652+0000","last_change":"2024-04-25T05:54:44.788203+0000","last_active":"2024-04-25T05:55:32.589652+0000","last_peered":"2024-04-25T05:55:32.589652+0000","last_clean":"2024-04-25T05:55:32.589652+0000","last_became_active":"2024-04-25T05:39:29.799070+0000","last_became_peered":"2024-04-25T05:39:29.799070+0000","last_unstale":"2024-04-25T05:55:32.589652+0000","last_undegraded":"2024-04-25T05:55:32.589652+0000","last_fullsized":"2024-04-25T05:55:32.589652+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.788114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:51.540285+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.788114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.659567+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":1561,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588680+0000","last_change":"2024-04-25T05:54:37.771913+0000","last_active":"2024-04-25T05:55:32.588680+0000","last_peered":"2024-04-25T05:55:32.588680+0000","last_clean":"2024-04-25T05:55:32.588680+0000","last_became_active":"2024-04-25T05:52:27.138282+0000","last_became_peered":"2024-04-25T05:52:27.138282+0000","last_unstale":"2024-04-25T05:55:32.588680+0000","last_undegraded":"2024-04-25T05:55:32.588680+0000","last_fullsized":"2024-04-25T05:55:32.588680+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:37.771809+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-04-25T05:51:34.121865+0000","last_clean_scrub_stamp":"2024-04-25T05:54:37.771809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:41.922250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778744+0000","last_change":"2024-04-25T05:54:26.260460+0000","last_active":"2024-04-25T05:55:32.778744+0000","last_peered":"2024-04-25T05:55:32.778744+0000","last_clean":"2024-04-25T05:55:32.778744+0000","last_became_active":"2024-04-25T05:51:40.097191+0000","last_became_peered":"2024-04-25T05:51:40.097191+0000","last_unstale":"2024-04-25T05:55:32.778744+0000","last_undegraded":"2024-04-25T05:55:32.778744+0000","last_fullsized":"2024-04-25T05:55:32.778744+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:26.260375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.260375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:49.182432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"581'71","reported_seq":1603,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.995345+0000","last_change":"2024-04-25T05:54:46.506499+0000","last_active":"2024-04-25T05:55:33.995345+0000","last_peered":"2024-04-25T05:55:33.995345+0000","last_clean":"2024-04-25T05:55:33.995345+0000","last_became_active":"2024-04-25T05:46:19.987784+0000","last_became_peered":"2024-04-25T05:46:19.987784+0000","last_unstale":"2024-04-25T05:55:33.995345+0000","last_undegraded":"2024-04-25T05:55:33.995345+0000","last_fullsized":"2024-04-25T05:55:33.995345+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'71","last_scrub_stamp":"2024-04-25T05:54:46.506365+0000","last_deep_scrub":"581'71","last_deep_scrub_stamp":"2024-04-25T05:54:46.506365+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.506365+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:53.945333+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":762,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243653+0000","last_change":"2024-04-25T05:55:15.108209+0000","last_active":"2024-04-25T05:55:33.243653+0000","last_peered":"2024-04-25T05:55:33.243653+0000","last_clean":"2024-04-25T05:55:33.243653+0000","last_became_active":"2024-04-25T05:51:40.103871+0000","last_became_peered":"2024-04-25T05:51:40.103871+0000","last_unstale":"2024-04-25T05:55:33.243653+0000","last_undegraded":"2024-04-25T05:55:33.243653+0000","last_fullsized":"2024-04-25T05:55:33.243653+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.108110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:37:43.613376+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.108110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:32.957117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"506'3","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755594+0000","last_change":"2024-04-25T05:54:27.940078+0000","last_active":"2024-04-25T05:55:32.755594+0000","last_peered":"2024-04-25T05:55:32.755594+0000","last_clean":"2024-04-25T05:55:32.755594+0000","last_became_active":"2024-04-25T05:46:34.969653+0000","last_became_peered":"2024-04-25T05:46:34.969653+0000","last_unstale":"2024-04-25T05:55:32.755594+0000","last_undegraded":"2024-04-25T05:55:32.755594+0000","last_fullsized":"2024-04-25T05:55:32.755594+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'3","last_scrub_stamp":"2024-04-25T05:54:27.939936+0000","last_deep_scrub":"506'3","last_deep_scrub_stamp":"2024-04-25T05:52:01.064292+0000","last_clean_scrub_stamp":"2024-04-25T05:54:27.939936+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00052709800000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1151,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372903+0000","last_change":"2024-04-25T05:54:32.917114+0000","last_active":"2024-04-25T05:55:32.372903+0000","last_peered":"2024-04-25T05:55:32.372903+0000","last_clean":"2024-04-25T05:55:32.372903+0000","last_became_active":"2024-04-25T05:34:04.721956+0000","last_became_peered":"2024-04-25T05:34:04.721956+0000","last_unstale":"2024-04-25T05:55:32.372903+0000","last_undegraded":"2024-04-25T05:55:32.372903+0000","last_fullsized":"2024-04-25T05:55:32.372903+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:32.917028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:32.917028+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.917028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:40.343262+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":1262,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755496+0000","last_change":"2024-04-25T05:54:39.795686+0000","last_active":"2024-04-25T05:55:32.755496+0000","last_peered":"2024-04-25T05:55:32.755496+0000","last_clean":"2024-04-25T05:55:32.755496+0000","last_became_active":"2024-04-25T05:52:31.177637+0000","last_became_peered":"2024-04-25T05:52:31.177637+0000","last_unstale":"2024-04-25T05:55:32.755496+0000","last_undegraded":"2024-04-25T05:55:32.755496+0000","last_fullsized":"2024-04-25T05:55:32.755496+0000","mapping_epoch":500,"log_start":"79'128","ondisk_log_start":"79'128","created":171,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-04-25T05:54:39.795598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:31:45.191968+0000","last_clean_scrub_stamp":"2024-04-25T05:54:39.795598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:48.695810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"581'32","reported_seq":290,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778592+0000","last_change":"2024-04-25T05:55:00.330495+0000","last_active":"2024-04-25T05:55:32.778592+0000","last_peered":"2024-04-25T05:55:32.778592+0000","last_clean":"2024-04-25T05:55:32.778592+0000","last_became_active":"2024-04-25T05:51:40.098092+0000","last_became_peered":"2024-04-25T05:51:40.098092+0000","last_unstale":"2024-04-25T05:55:32.778592+0000","last_undegraded":"2024-04-25T05:55:32.778592+0000","last_fullsized":"2024-04-25T05:55:32.778592+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:55:00.330385+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:41.766685+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.330385+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.895940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756173+0000","last_change":"2024-04-25T05:54:51.735918+0000","last_active":"2024-04-25T05:55:32.756173+0000","last_peered":"2024-04-25T05:55:32.756173+0000","last_clean":"2024-04-25T05:55:32.756173+0000","last_became_active":"2024-04-25T05:46:34.977475+0000","last_became_peered":"2024-04-25T05:46:34.977475+0000","last_unstale":"2024-04-25T05:55:32.756173+0000","last_undegraded":"2024-04-25T05:55:32.756173+0000","last_fullsized":"2024-04-25T05:55:32.756173+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:51.735807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:41:28.560585+0000","last_clean_scrub_stamp":"2024-04-25T05:54:51.735807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:03.296810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":535,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585142+0000","last_change":"2024-04-25T05:55:05.716998+0000","last_active":"2024-04-25T05:55:25.585142+0000","last_peered":"2024-04-25T05:55:25.585142+0000","last_clean":"2024-04-25T05:55:25.585142+0000","last_became_active":"2024-04-25T05:46:19.995957+0000","last_became_peered":"2024-04-25T05:46:19.995957+0000","last_unstale":"2024-04-25T05:55:25.585142+0000","last_undegraded":"2024-04-25T05:55:25.585142+0000","last_fullsized":"2024-04-25T05:55:25.585142+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:05.716901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:32.512005+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.716901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.752558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778179+0000","last_change":"2024-04-25T05:54:25.300412+0000","last_active":"2024-04-25T05:55:32.778179+0000","last_peered":"2024-04-25T05:55:32.778179+0000","last_clean":"2024-04-25T05:55:32.778179+0000","last_became_active":"2024-04-25T05:51:40.098679+0000","last_became_peered":"2024-04-25T05:51:40.098679+0000","last_unstale":"2024-04-25T05:55:32.778179+0000","last_undegraded":"2024-04-25T05:55:32.778179+0000","last_fullsized":"2024-04-25T05:55:32.778179+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:25.300308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:25.300308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.594574+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"582'67","reported_seq":997,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.592469+0000","last_change":"2024-04-25T05:55:32.592521+0000","last_active":"2024-04-25T05:55:32.592469+0000","last_peered":"2024-04-25T05:55:32.592469+0000","last_clean":"2024-04-25T05:55:32.592469+0000","last_became_active":"2024-04-25T05:39:29.802392+0000","last_became_peered":"2024-04-25T05:39:29.802392+0000","last_unstale":"2024-04-25T05:55:32.592469+0000","last_undegraded":"2024-04-25T05:55:32.592469+0000","last_fullsized":"2024-04-25T05:55:32.592469+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'67","last_scrub_stamp":"2024-04-25T05:55:32.592394+0000","last_deep_scrub":"458'51","last_deep_scrub_stamp":"2024-04-25T05:50:15.941896+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.592394+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:57:01.099599+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"472'2","reported_seq":1232,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589400+0000","last_change":"2024-04-25T05:54:53.582781+0000","last_active":"2024-04-25T05:55:32.589400+0000","last_peered":"2024-04-25T05:55:32.589400+0000","last_clean":"2024-04-25T05:55:32.589400+0000","last_became_active":"2024-04-25T05:46:34.971641+0000","last_became_peered":"2024-04-25T05:46:34.971641+0000","last_unstale":"2024-04-25T05:55:32.589400+0000","last_undegraded":"2024-04-25T05:55:32.589400+0000","last_fullsized":"2024-04-25T05:55:32.589400+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'2","last_scrub_stamp":"2024-04-25T05:54:53.582673+0000","last_deep_scrub":"472'2","last_deep_scrub_stamp":"2024-04-25T05:50:47.124554+0000","last_clean_scrub_stamp":"2024-04-25T05:54:53.582673+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.091409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":547,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585612+0000","last_change":"2024-04-25T05:55:15.729050+0000","last_active":"2024-04-25T05:55:25.585612+0000","last_peered":"2024-04-25T05:55:25.585612+0000","last_clean":"2024-04-25T05:55:25.585612+0000","last_became_active":"2024-04-25T05:51:40.105319+0000","last_became_peered":"2024-04-25T05:51:40.105319+0000","last_unstale":"2024-04-25T05:55:25.585612+0000","last_undegraded":"2024-04-25T05:55:25.585612+0000","last_fullsized":"2024-04-25T05:55:25.585612+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:15.728949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.728949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:34.671139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588299+0000","last_change":"2024-04-25T05:54:59.518039+0000","last_active":"2024-04-25T05:55:32.588299+0000","last_peered":"2024-04-25T05:55:32.588299+0000","last_clean":"2024-04-25T05:55:32.588299+0000","last_became_active":"2024-04-25T05:46:19.986682+0000","last_became_peered":"2024-04-25T05:46:19.986682+0000","last_unstale":"2024-04-25T05:55:32.588299+0000","last_undegraded":"2024-04-25T05:55:32.588299+0000","last_fullsized":"2024-04-25T05:55:32.588299+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.517922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.517922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.907823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":547,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.584789+0000","last_change":"2024-04-25T05:54:47.398323+0000","last_active":"2024-04-25T05:55:25.584789+0000","last_peered":"2024-04-25T05:55:25.584789+0000","last_clean":"2024-04-25T05:55:25.584789+0000","last_became_active":"2024-04-25T05:51:40.105868+0000","last_became_peered":"2024-04-25T05:51:40.105868+0000","last_unstale":"2024-04-25T05:55:25.584789+0000","last_undegraded":"2024-04-25T05:55:25.584789+0000","last_fullsized":"2024-04-25T05:55:25.584789+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:47.398225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:47.398225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:04.378970+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"600'43","reported_seq":618,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244327+0000","last_change":"2024-04-25T05:54:45.296651+0000","last_active":"2024-04-25T05:55:33.244327+0000","last_peered":"2024-04-25T05:55:33.244327+0000","last_clean":"2024-04-25T05:55:33.244327+0000","last_became_active":"2024-04-25T05:49:09.558838+0000","last_became_peered":"2024-04-25T05:49:09.558838+0000","last_unstale":"2024-04-25T05:55:33.244327+0000","last_undegraded":"2024-04-25T05:55:33.244327+0000","last_fullsized":"2024-04-25T05:55:33.244327+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'43","last_scrub_stamp":"2024-04-25T05:54:45.296530+0000","last_deep_scrub":"600'43","last_deep_scrub_stamp":"2024-04-25T05:54:45.296530+0000","last_clean_scrub_stamp":"2024-04-25T05:54:45.296530+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.109602+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":49,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589467+0000","last_change":"2024-04-25T05:54:54.552769+0000","last_active":"2024-04-25T05:55:32.589467+0000","last_peered":"2024-04-25T05:55:32.589467+0000","last_clean":"2024-04-25T05:55:32.589467+0000","last_became_active":"2024-04-25T05:33:58.687010+0000","last_became_peered":"2024-04-25T05:33:58.687010+0000","last_unstale":"2024-04-25T05:55:32.589467+0000","last_undegraded":"2024-04-25T05:55:32.589467+0000","last_fullsized":"2024-04-25T05:55:32.589467+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:54.552681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:51.060223+0000","last_clean_scrub_stamp":"2024-04-25T05:54:54.552681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.031257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1163,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372386+0000","last_change":"2024-04-25T05:55:06.051836+0000","last_active":"2024-04-25T05:55:32.372386+0000","last_peered":"2024-04-25T05:55:32.372386+0000","last_clean":"2024-04-25T05:55:32.372386+0000","last_became_active":"2024-04-25T05:40:55.987999+0000","last_became_peered":"2024-04-25T05:40:55.987999+0000","last_unstale":"2024-04-25T05:55:32.372386+0000","last_undegraded":"2024-04-25T05:55:32.372386+0000","last_fullsized":"2024-04-25T05:55:32.372386+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:06.051728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:06.051728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:22.918905+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"208'11","reported_seq":1471,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588417+0000","last_change":"2024-04-25T05:54:45.736450+0000","last_active":"2024-04-25T05:55:32.588417+0000","last_peered":"2024-04-25T05:55:32.588417+0000","last_clean":"2024-04-25T05:55:32.588417+0000","last_became_active":"2024-04-25T05:39:29.798454+0000","last_became_peered":"2024-04-25T05:39:29.798454+0000","last_unstale":"2024-04-25T05:55:32.588417+0000","last_undegraded":"2024-04-25T05:55:32.588417+0000","last_fullsized":"2024-04-25T05:55:32.588417+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'11","last_scrub_stamp":"2024-04-25T05:54:45.736320+0000","last_deep_scrub":"208'11","last_deep_scrub_stamp":"2024-04-25T05:52:50.387023+0000","last_clean_scrub_stamp":"2024-04-25T05:54:45.736320+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.710562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"581'51","reported_seq":550,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755339+0000","last_change":"2024-04-25T05:54:30.886889+0000","last_active":"2024-04-25T05:55:32.755339+0000","last_peered":"2024-04-25T05:55:32.755339+0000","last_clean":"2024-04-25T05:55:32.755339+0000","last_became_active":"2024-04-25T05:46:34.978298+0000","last_became_peered":"2024-04-25T05:46:34.978298+0000","last_unstale":"2024-04-25T05:55:32.755339+0000","last_undegraded":"2024-04-25T05:55:32.755339+0000","last_fullsized":"2024-04-25T05:55:32.755339+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'51","last_scrub_stamp":"2024-04-25T05:54:30.886719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.886719+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:33.869436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1245,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589577+0000","last_change":"2024-04-25T05:55:03.505601+0000","last_active":"2024-04-25T05:55:32.589577+0000","last_peered":"2024-04-25T05:55:32.589577+0000","last_clean":"2024-04-25T05:55:32.589577+0000","last_became_active":"2024-04-25T05:51:40.097994+0000","last_became_peered":"2024-04-25T05:51:40.097994+0000","last_unstale":"2024-04-25T05:55:32.589577+0000","last_undegraded":"2024-04-25T05:55:32.589577+0000","last_fullsized":"2024-04-25T05:55:32.589577+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-04-25T05:55:03.505497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:03.505497+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:21.467173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":518,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.777051+0000","last_change":"2024-04-25T05:55:32.777078+0000","last_active":"2024-04-25T05:55:32.777051+0000","last_peered":"2024-04-25T05:55:32.777051+0000","last_clean":"2024-04-25T05:55:32.777051+0000","last_became_active":"2024-04-25T05:51:40.083785+0000","last_became_peered":"2024-04-25T05:51:40.083785+0000","last_unstale":"2024-04-25T05:55:32.777051+0000","last_undegraded":"2024-04-25T05:55:32.777051+0000","last_fullsized":"2024-04-25T05:55:32.777051+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.776959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.776959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.153078+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"613'28","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373560+0000","last_change":"2024-04-25T05:55:14.809995+0000","last_active":"2024-04-25T05:55:32.373560+0000","last_peered":"2024-04-25T05:55:32.373560+0000","last_clean":"2024-04-25T05:55:32.373560+0000","last_became_active":"2024-04-25T05:34:00.999976+0000","last_became_peered":"2024-04-25T05:34:00.999976+0000","last_unstale":"2024-04-25T05:55:32.373560+0000","last_undegraded":"2024-04-25T05:55:32.373560+0000","last_fullsized":"2024-04-25T05:55:32.373560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'28","last_scrub_stamp":"2024-04-25T05:55:14.809911+0000","last_deep_scrub":"613'28","last_deep_scrub_stamp":"2024-04-25T05:55:14.809911+0000","last_clean_scrub_stamp":"2024-04-25T05:55:14.809911+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:19.085723+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":944,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.370891+0000","last_change":"2024-04-25T05:54:24.388418+0000","last_active":"2024-04-25T05:55:32.370891+0000","last_peered":"2024-04-25T05:55:32.370891+0000","last_clean":"2024-04-25T05:55:32.370891+0000","last_became_active":"2024-04-25T05:51:40.104959+0000","last_became_peered":"2024-04-25T05:51:40.104959+0000","last_unstale":"2024-04-25T05:55:32.370891+0000","last_undegraded":"2024-04-25T05:55:32.370891+0000","last_fullsized":"2024-04-25T05:55:32.370891+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:24.388316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:59.516638+0000","last_clean_scrub_stamp":"2024-04-25T05:54:24.388316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:53.660202+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756389+0000","last_change":"2024-04-25T05:54:46.752748+0000","last_active":"2024-04-25T05:55:32.756389+0000","last_peered":"2024-04-25T05:55:32.756389+0000","last_clean":"2024-04-25T05:55:32.756389+0000","last_became_active":"2024-04-25T05:46:34.985696+0000","last_became_peered":"2024-04-25T05:46:34.985696+0000","last_unstale":"2024-04-25T05:55:32.756389+0000","last_undegraded":"2024-04-25T05:55:32.756389+0000","last_fullsized":"2024-04-25T05:55:32.756389+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:46.752658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:46.752658+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.752658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:01.381614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589044+0000","last_change":"2024-04-25T05:55:00.655656+0000","last_active":"2024-04-25T05:55:32.589044+0000","last_peered":"2024-04-25T05:55:32.589044+0000","last_clean":"2024-04-25T05:55:32.589044+0000","last_became_active":"2024-04-25T05:39:29.798470+0000","last_became_peered":"2024-04-25T05:39:29.798470+0000","last_unstale":"2024-04-25T05:55:32.589044+0000","last_undegraded":"2024-04-25T05:55:32.589044+0000","last_fullsized":"2024-04-25T05:55:32.589044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:00.655554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:00.655554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.183884+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"602'39","reported_seq":298,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778686+0000","last_change":"2024-04-25T05:54:50.269023+0000","last_active":"2024-04-25T05:55:32.778686+0000","last_peered":"2024-04-25T05:55:32.778686+0000","last_clean":"2024-04-25T05:55:32.778686+0000","last_became_active":"2024-04-25T05:51:40.103887+0000","last_became_peered":"2024-04-25T05:51:40.103887+0000","last_unstale":"2024-04-25T05:55:32.778686+0000","last_undegraded":"2024-04-25T05:55:32.778686+0000","last_fullsized":"2024-04-25T05:55:32.778686+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'39","last_scrub_stamp":"2024-04-25T05:54:50.268891+0000","last_deep_scrub":"602'39","last_deep_scrub_stamp":"2024-04-25T05:54:50.268891+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.268891+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:57.400862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1203,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588567+0000","last_change":"2024-04-25T05:54:56.527212+0000","last_active":"2024-04-25T05:55:32.588567+0000","last_peered":"2024-04-25T05:55:32.588567+0000","last_clean":"2024-04-25T05:55:32.588567+0000","last_became_active":"2024-04-25T05:39:29.798114+0000","last_became_peered":"2024-04-25T05:39:29.798114+0000","last_unstale":"2024-04-25T05:55:32.588567+0000","last_undegraded":"2024-04-25T05:55:32.588567+0000","last_fullsized":"2024-04-25T05:55:32.588567+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:56.527124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:47:31.715261+0000","last_clean_scrub_stamp":"2024-04-25T05:54:56.527124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:18.197169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1154,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373482+0000","last_change":"2024-04-25T05:55:18.784318+0000","last_active":"2024-04-25T05:55:32.373482+0000","last_peered":"2024-04-25T05:55:32.373482+0000","last_clean":"2024-04-25T05:55:32.373482+0000","last_became_active":"2024-04-25T05:34:02.697753+0000","last_became_peered":"2024-04-25T05:34:02.697753+0000","last_unstale":"2024-04-25T05:55:32.373482+0000","last_undegraded":"2024-04-25T05:55:32.373482+0000","last_fullsized":"2024-04-25T05:55:32.373482+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:18.784225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:18.784225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.971233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":526,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585911+0000","last_change":"2024-04-25T05:54:46.431180+0000","last_active":"2024-04-25T05:55:25.585911+0000","last_peered":"2024-04-25T05:55:25.585911+0000","last_clean":"2024-04-25T05:55:25.585911+0000","last_became_active":"2024-04-25T05:46:19.984203+0000","last_became_peered":"2024-04-25T05:46:19.984203+0000","last_unstale":"2024-04-25T05:55:25.585911+0000","last_undegraded":"2024-04-25T05:55:25.585911+0000","last_fullsized":"2024-04-25T05:55:25.585911+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:46.431061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.431061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.757044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"581'32","reported_seq":532,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756066+0000","last_change":"2024-04-25T05:55:02.624343+0000","last_active":"2024-04-25T05:55:32.756066+0000","last_peered":"2024-04-25T05:55:32.756066+0000","last_clean":"2024-04-25T05:55:32.756066+0000","last_became_active":"2024-04-25T05:46:34.981765+0000","last_became_peered":"2024-04-25T05:46:34.981765+0000","last_unstale":"2024-04-25T05:55:32.756066+0000","last_undegraded":"2024-04-25T05:55:32.756066+0000","last_fullsized":"2024-04-25T05:55:32.756066+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:55:02.624225+0000","last_deep_scrub":"100'20","last_deep_scrub_stamp":"2024-04-25T05:35:40.845036+0000","last_clean_scrub_stamp":"2024-04-25T05:55:02.624225+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:04.850542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778401+0000","last_change":"2024-04-25T05:54:53.256101+0000","last_active":"2024-04-25T05:55:32.778401+0000","last_peered":"2024-04-25T05:55:32.778401+0000","last_clean":"2024-04-25T05:55:32.778401+0000","last_became_active":"2024-04-25T05:51:40.098809+0000","last_became_peered":"2024-04-25T05:51:40.098809+0000","last_unstale":"2024-04-25T05:55:32.778401+0000","last_undegraded":"2024-04-25T05:55:32.778401+0000","last_fullsized":"2024-04-25T05:55:32.778401+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:53.256012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:36:31.117874+0000","last_clean_scrub_stamp":"2024-04-25T05:54:53.256012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:11.289454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":1443,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373139+0000","last_change":"2024-04-25T05:55:07.811741+0000","last_active":"2024-04-25T05:55:32.373139+0000","last_peered":"2024-04-25T05:55:32.373139+0000","last_clean":"2024-04-25T05:55:32.373139+0000","last_became_active":"2024-04-25T05:46:19.999850+0000","last_became_peered":"2024-04-25T05:46:19.999850+0000","last_unstale":"2024-04-25T05:55:32.373139+0000","last_undegraded":"2024-04-25T05:55:32.373139+0000","last_fullsized":"2024-04-25T05:55:32.373139+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-04-25T05:55:07.811600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:07.811600+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:13.028382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":725,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244731+0000","last_change":"2024-04-25T05:54:39.440135+0000","last_active":"2024-04-25T05:55:33.244731+0000","last_peered":"2024-04-25T05:55:33.244731+0000","last_clean":"2024-04-25T05:55:33.244731+0000","last_became_active":"2024-04-25T05:49:09.556677+0000","last_became_peered":"2024-04-25T05:49:09.556677+0000","last_unstale":"2024-04-25T05:55:33.244731+0000","last_undegraded":"2024-04-25T05:55:33.244731+0000","last_fullsized":"2024-04-25T05:55:33.244731+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:39.440013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:39.440013+0000","last_clean_scrub_stamp":"2024-04-25T05:54:39.440013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.436716+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"581'42","reported_seq":1236,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372624+0000","last_change":"2024-04-25T05:55:19.744864+0000","last_active":"2024-04-25T05:55:32.372624+0000","last_peered":"2024-04-25T05:55:32.372624+0000","last_clean":"2024-04-25T05:55:32.372624+0000","last_became_active":"2024-04-25T05:34:00.999360+0000","last_became_peered":"2024-04-25T05:34:00.999360+0000","last_unstale":"2024-04-25T05:55:32.372624+0000","last_undegraded":"2024-04-25T05:55:32.372624+0000","last_fullsized":"2024-04-25T05:55:32.372624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'42","last_scrub_stamp":"2024-04-25T05:55:19.744753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:19.744753+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.237215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1197,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588751+0000","last_change":"2024-04-25T05:54:58.515557+0000","last_active":"2024-04-25T05:55:32.588751+0000","last_peered":"2024-04-25T05:55:32.588751+0000","last_clean":"2024-04-25T05:55:32.588751+0000","last_became_active":"2024-04-25T05:46:19.985591+0000","last_became_peered":"2024-04-25T05:46:19.985591+0000","last_unstale":"2024-04-25T05:55:32.588751+0000","last_undegraded":"2024-04-25T05:55:32.588751+0000","last_fullsized":"2024-04-25T05:55:32.588751+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.515470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.515470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:27.679494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"501'14","reported_seq":624,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756273+0000","last_change":"2024-04-25T05:54:32.801763+0000","last_active":"2024-04-25T05:55:32.756273+0000","last_peered":"2024-04-25T05:55:32.756273+0000","last_clean":"2024-04-25T05:55:32.756273+0000","last_became_active":"2024-04-25T05:51:40.104608+0000","last_became_peered":"2024-04-25T05:51:40.104608+0000","last_unstale":"2024-04-25T05:55:32.756273+0000","last_undegraded":"2024-04-25T05:55:32.756273+0000","last_fullsized":"2024-04-25T05:55:32.756273+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'14","last_scrub_stamp":"2024-04-25T05:54:32.801648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:32.801648+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.540836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778909+0000","last_change":"2024-04-25T05:54:29.432405+0000","last_active":"2024-04-25T05:55:32.778909+0000","last_peered":"2024-04-25T05:55:32.778909+0000","last_clean":"2024-04-25T05:55:32.778909+0000","last_became_active":"2024-04-25T05:51:40.103146+0000","last_became_peered":"2024-04-25T05:51:40.103146+0000","last_unstale":"2024-04-25T05:55:32.778909+0000","last_undegraded":"2024-04-25T05:55:32.778909+0000","last_fullsized":"2024-04-25T05:55:32.778909+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:29.432303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.432303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:59.154796+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"581'32","reported_seq":1337,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243085+0000","last_change":"2024-04-25T05:54:46.258177+0000","last_active":"2024-04-25T05:55:33.243085+0000","last_peered":"2024-04-25T05:55:33.243085+0000","last_clean":"2024-04-25T05:55:33.243085+0000","last_became_active":"2024-04-25T05:46:36.981225+0000","last_became_peered":"2024-04-25T05:46:36.981225+0000","last_unstale":"2024-04-25T05:55:33.243085+0000","last_undegraded":"2024-04-25T05:55:33.243085+0000","last_fullsized":"2024-04-25T05:55:33.243085+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:54:46.258082+0000","last_deep_scrub":"320'24","last_deep_scrub_stamp":"2024-04-25T05:51:23.777815+0000","last_clean_scrub_stamp":"2024-04-25T05:54:46.258082+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.893531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"67'1","reported_seq":779,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243775+0000","last_change":"2024-04-25T05:55:17.118213+0000","last_active":"2024-04-25T05:55:33.243775+0000","last_peered":"2024-04-25T05:55:33.243775+0000","last_clean":"2024-04-25T05:55:33.243775+0000","last_became_active":"2024-04-25T05:49:09.580380+0000","last_became_peered":"2024-04-25T05:49:09.580380+0000","last_unstale":"2024-04-25T05:55:33.243775+0000","last_undegraded":"2024-04-25T05:55:33.243775+0000","last_fullsized":"2024-04-25T05:55:33.243775+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:55:17.118107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:17.118107+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:44.384424+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371266+0000","last_change":"2024-04-25T05:55:05.572752+0000","last_active":"2024-04-25T05:55:32.371266+0000","last_peered":"2024-04-25T05:55:32.371266+0000","last_clean":"2024-04-25T05:55:32.371266+0000","last_became_active":"2024-04-25T05:46:34.973468+0000","last_became_peered":"2024-04-25T05:46:34.973468+0000","last_unstale":"2024-04-25T05:55:32.371266+0000","last_undegraded":"2024-04-25T05:55:32.371266+0000","last_fullsized":"2024-04-25T05:55:32.371266+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:05.572644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.572644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:33.809245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1201,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371613+0000","last_change":"2024-04-25T05:54:58.863337+0000","last_active":"2024-04-25T05:55:32.371613+0000","last_peered":"2024-04-25T05:55:32.371613+0000","last_clean":"2024-04-25T05:55:32.371613+0000","last_became_active":"2024-04-25T05:51:40.083559+0000","last_became_peered":"2024-04-25T05:51:40.083559+0000","last_unstale":"2024-04-25T05:55:32.371613+0000","last_undegraded":"2024-04-25T05:55:32.371613+0000","last_fullsized":"2024-04-25T05:55:32.371613+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.863207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.863207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:15.678236+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"581'25","reported_seq":741,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371864+0000","last_change":"2024-04-25T05:54:30.437415+0000","last_active":"2024-04-25T05:55:32.371864+0000","last_peered":"2024-04-25T05:55:32.371864+0000","last_clean":"2024-04-25T05:55:32.371864+0000","last_became_active":"2024-04-25T05:46:20.001869+0000","last_became_peered":"2024-04-25T05:46:20.001869+0000","last_unstale":"2024-04-25T05:55:32.371864+0000","last_undegraded":"2024-04-25T05:55:32.371864+0000","last_fullsized":"2024-04-25T05:55:32.371864+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'25","last_scrub_stamp":"2024-04-25T05:54:30.437264+0000","last_deep_scrub":"257'13","last_deep_scrub_stamp":"2024-04-25T05:41:30.532371+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.437264+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.390257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1184,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589573+0000","last_change":"2024-04-25T05:54:48.642072+0000","last_active":"2024-04-25T05:55:32.589573+0000","last_peered":"2024-04-25T05:55:32.589573+0000","last_clean":"2024-04-25T05:55:32.589573+0000","last_became_active":"2024-04-25T05:33:58.687094+0000","last_became_peered":"2024-04-25T05:33:58.687094+0000","last_unstale":"2024-04-25T05:55:32.589573+0000","last_undegraded":"2024-04-25T05:55:32.589573+0000","last_fullsized":"2024-04-25T05:55:32.589573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:48.641984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:42.725737+0000","last_clean_scrub_stamp":"2024-04-25T05:54:48.641984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:12.941861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1185,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372578+0000","last_change":"2024-04-25T05:55:02.893757+0000","last_active":"2024-04-25T05:55:32.372578+0000","last_peered":"2024-04-25T05:55:32.372578+0000","last_clean":"2024-04-25T05:55:32.372578+0000","last_became_active":"2024-04-25T05:51:40.100205+0000","last_became_peered":"2024-04-25T05:51:40.100205+0000","last_unstale":"2024-04-25T05:55:32.372578+0000","last_undegraded":"2024-04-25T05:55:32.372578+0000","last_fullsized":"2024-04-25T05:55:32.372578+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:02.893671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:55:02.893671+0000","last_clean_scrub_stamp":"2024-04-25T05:55:02.893671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:31.595330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778143+0000","last_change":"2024-04-25T05:55:19.230360+0000","last_active":"2024-04-25T05:55:32.778143+0000","last_peered":"2024-04-25T05:55:32.778143+0000","last_clean":"2024-04-25T05:55:32.778143+0000","last_became_active":"2024-04-25T05:51:40.094474+0000","last_became_peered":"2024-04-25T05:51:40.094474+0000","last_unstale":"2024-04-25T05:55:32.778143+0000","last_undegraded":"2024-04-25T05:55:32.778143+0000","last_fullsized":"2024-04-25T05:55:32.778143+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:19.230225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:19.230225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:35.604414+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"582'39","reported_seq":1268,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371940+0000","last_change":"2024-04-25T05:55:08.765847+0000","last_active":"2024-04-25T05:55:32.371940+0000","last_peered":"2024-04-25T05:55:32.371940+0000","last_clean":"2024-04-25T05:55:32.371940+0000","last_became_active":"2024-04-25T05:39:29.797260+0000","last_became_peered":"2024-04-25T05:39:29.797260+0000","last_unstale":"2024-04-25T05:55:32.371940+0000","last_undegraded":"2024-04-25T05:55:32.371940+0000","last_fullsized":"2024-04-25T05:55:32.371940+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'39","last_scrub_stamp":"2024-04-25T05:55:08.765698+0000","last_deep_scrub":"406'31","last_deep_scrub_stamp":"2024-04-25T05:47:30.478462+0000","last_clean_scrub_stamp":"2024-04-25T05:55:08.765698+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.183048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756183+0000","last_change":"2024-04-25T05:54:35.811475+0000","last_active":"2024-04-25T05:55:32.756183+0000","last_peered":"2024-04-25T05:55:32.756183+0000","last_clean":"2024-04-25T05:55:32.756183+0000","last_became_active":"2024-04-25T05:46:34.970048+0000","last_became_peered":"2024-04-25T05:46:34.970048+0000","last_unstale":"2024-04-25T05:55:32.756183+0000","last_undegraded":"2024-04-25T05:55:32.756183+0000","last_fullsized":"2024-04-25T05:55:32.756183+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:35.811379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:35.811379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:54.744260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1195,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588652+0000","last_change":"2024-04-25T05:54:57.536648+0000","last_active":"2024-04-25T05:55:32.588652+0000","last_peered":"2024-04-25T05:55:32.588652+0000","last_clean":"2024-04-25T05:55:32.588652+0000","last_became_active":"2024-04-25T05:39:29.797180+0000","last_became_peered":"2024-04-25T05:39:29.797180+0000","last_unstale":"2024-04-25T05:55:32.588652+0000","last_undegraded":"2024-04-25T05:55:32.588652+0000","last_fullsized":"2024-04-25T05:55:32.588652+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:57.536536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:57.536536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:08.582347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372485+0000","last_change":"2024-04-25T05:54:06.026590+0000","last_active":"2024-04-25T05:55:32.372485+0000","last_peered":"2024-04-25T05:55:32.372485+0000","last_clean":"2024-04-25T05:55:32.372485+0000","last_became_active":"2024-04-25T05:51:40.100768+0000","last_became_peered":"2024-04-25T05:51:40.100768+0000","last_unstale":"2024-04-25T05:55:32.372485+0000","last_undegraded":"2024-04-25T05:55:32.372485+0000","last_fullsized":"2024-04-25T05:55:32.372485+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:06.026456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:54:06.026456+0000","last_clean_scrub_stamp":"2024-04-25T05:54:06.026456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"581'12","reported_seq":1209,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.779003+0000","last_change":"2024-04-25T05:55:33.779033+0000","last_active":"2024-04-25T05:55:33.779003+0000","last_peered":"2024-04-25T05:55:33.779003+0000","last_clean":"2024-04-25T05:55:33.779003+0000","last_became_active":"2024-04-25T05:39:29.798750+0000","last_became_peered":"2024-04-25T05:39:29.798750+0000","last_unstale":"2024-04-25T05:55:33.779003+0000","last_undegraded":"2024-04-25T05:55:33.779003+0000","last_fullsized":"2024-04-25T05:55:33.779003+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'12","last_scrub_stamp":"2024-04-25T05:55:33.778925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:33.778925+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.063264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779031+0000","last_change":"2024-04-25T05:54:58.287258+0000","last_active":"2024-04-25T05:55:32.779031+0000","last_peered":"2024-04-25T05:55:32.779031+0000","last_clean":"2024-04-25T05:55:32.779031+0000","last_became_active":"2024-04-25T05:51:40.096935+0000","last_became_peered":"2024-04-25T05:51:40.096935+0000","last_unstale":"2024-04-25T05:55:32.779031+0000","last_undegraded":"2024-04-25T05:55:32.779031+0000","last_fullsized":"2024-04-25T05:55:32.779031+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:58.287131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:58.287131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:01.930630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":900,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.370804+0000","last_change":"2024-04-25T05:54:31.535891+0000","last_active":"2024-04-25T05:55:32.370804+0000","last_peered":"2024-04-25T05:55:32.370804+0000","last_clean":"2024-04-25T05:55:32.370804+0000","last_became_active":"2024-04-25T05:39:29.802364+0000","last_became_peered":"2024-04-25T05:39:29.802364+0000","last_unstale":"2024-04-25T05:55:32.370804+0000","last_undegraded":"2024-04-25T05:55:32.370804+0000","last_fullsized":"2024-04-25T05:55:32.370804+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:31.535778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:45:29.822937+0000","last_clean_scrub_stamp":"2024-04-25T05:54:31.535778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:00.619524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"610'2","reported_seq":1195,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588782+0000","last_change":"2024-04-25T05:55:05.525112+0000","last_active":"2024-04-25T05:55:32.588782+0000","last_peered":"2024-04-25T05:55:32.588782+0000","last_clean":"2024-04-25T05:55:32.588782+0000","last_became_active":"2024-04-25T05:46:19.987198+0000","last_became_peered":"2024-04-25T05:46:19.987198+0000","last_unstale":"2024-04-25T05:55:32.588782+0000","last_undegraded":"2024-04-25T05:55:32.588782+0000","last_fullsized":"2024-04-25T05:55:32.588782+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'2","last_scrub_stamp":"2024-04-25T05:55:05.524993+0000","last_deep_scrub":"610'2","last_deep_scrub_stamp":"2024-04-25T05:55:05.524993+0000","last_clean_scrub_stamp":"2024-04-25T05:55:05.524993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:19.201505+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"582'39","reported_seq":1033,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372059+0000","last_change":"2024-04-25T05:54:49.441851+0000","last_active":"2024-04-25T05:55:32.372059+0000","last_peered":"2024-04-25T05:55:32.372059+0000","last_clean":"2024-04-25T05:55:32.372059+0000","last_became_active":"2024-04-25T05:46:36.981735+0000","last_became_peered":"2024-04-25T05:46:36.981735+0000","last_unstale":"2024-04-25T05:55:32.372059+0000","last_undegraded":"2024-04-25T05:55:32.372059+0000","last_fullsized":"2024-04-25T05:55:32.372059+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'39","last_scrub_stamp":"2024-04-25T05:54:49.441705+0000","last_deep_scrub":"292'23","last_deep_scrub_stamp":"2024-04-25T05:42:52.491617+0000","last_clean_scrub_stamp":"2024-04-25T05:54:49.441705+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:50.417754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1200,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371196+0000","last_change":"2024-04-25T05:55:30.766563+0000","last_active":"2024-04-25T05:55:32.371196+0000","last_peered":"2024-04-25T05:55:32.371196+0000","last_clean":"2024-04-25T05:55:32.371196+0000","last_became_active":"2024-04-25T05:51:40.084849+0000","last_became_peered":"2024-04-25T05:51:40.084849+0000","last_unstale":"2024-04-25T05:55:32.371196+0000","last_undegraded":"2024-04-25T05:55:32.371196+0000","last_fullsized":"2024-04-25T05:55:32.371196+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:30.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:36.324465+0000","last_clean_scrub_stamp":"2024-04-25T05:55:30.766478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:46.739046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778565+0000","last_change":"2024-04-25T05:54:54.305334+0000","last_active":"2024-04-25T05:55:32.778565+0000","last_peered":"2024-04-25T05:55:32.778565+0000","last_clean":"2024-04-25T05:55:32.778565+0000","last_became_active":"2024-04-25T05:51:40.102601+0000","last_became_peered":"2024-04-25T05:51:40.102601+0000","last_unstale":"2024-04-25T05:55:32.778565+0000","last_undegraded":"2024-04-25T05:55:32.778565+0000","last_fullsized":"2024-04-25T05:55:32.778565+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:54.305250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:54.305250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:10.261701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1183,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372879+0000","last_change":"2024-04-25T05:55:24.017478+0000","last_active":"2024-04-25T05:55:32.372879+0000","last_peered":"2024-04-25T05:55:32.372879+0000","last_clean":"2024-04-25T05:55:32.372879+0000","last_became_active":"2024-04-25T05:46:34.974485+0000","last_became_peered":"2024-04-25T05:46:34.974485+0000","last_unstale":"2024-04-25T05:55:32.372879+0000","last_undegraded":"2024-04-25T05:55:32.372879+0000","last_fullsized":"2024-04-25T05:55:32.372879+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:24.017335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:17.140516+0000","last_clean_scrub_stamp":"2024-04-25T05:55:24.017335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.356248+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"582'42","reported_seq":290,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778798+0000","last_change":"2024-04-25T05:54:06.134244+0000","last_active":"2024-04-25T05:55:32.778798+0000","last_peered":"2024-04-25T05:55:32.778798+0000","last_clean":"2024-04-25T05:55:32.778798+0000","last_became_active":"2024-04-25T05:51:40.101220+0000","last_became_peered":"2024-04-25T05:51:40.101220+0000","last_unstale":"2024-04-25T05:55:32.778798+0000","last_undegraded":"2024-04-25T05:55:32.778798+0000","last_fullsized":"2024-04-25T05:55:32.778798+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"320'32","last_scrub_stamp":"2024-04-25T05:54:06.134150+0000","last_deep_scrub":"320'32","last_deep_scrub_stamp":"2024-04-25T05:50:48.098124+0000","last_clean_scrub_stamp":"2024-04-25T05:54:06.134150+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1186,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371144+0000","last_change":"2024-04-25T05:55:26.803181+0000","last_active":"2024-04-25T05:55:32.371144+0000","last_peered":"2024-04-25T05:55:32.371144+0000","last_clean":"2024-04-25T05:55:32.371144+0000","last_became_active":"2024-04-25T05:33:58.677888+0000","last_became_peered":"2024-04-25T05:33:58.677888+0000","last_unstale":"2024-04-25T05:55:32.371144+0000","last_undegraded":"2024-04-25T05:55:32.371144+0000","last_fullsized":"2024-04-25T05:55:32.371144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-04-25T05:55:26.803087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:26.803087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:41.832618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"208'10","reported_seq":1105,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371824+0000","last_change":"2024-04-25T05:54:29.519271+0000","last_active":"2024-04-25T05:55:32.371824+0000","last_peered":"2024-04-25T05:55:32.371824+0000","last_clean":"2024-04-25T05:55:32.371824+0000","last_became_active":"2024-04-25T05:39:29.802556+0000","last_became_peered":"2024-04-25T05:39:29.802556+0000","last_unstale":"2024-04-25T05:55:32.371824+0000","last_undegraded":"2024-04-25T05:55:32.371824+0000","last_fullsized":"2024-04-25T05:55:32.371824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-04-25T05:54:29.519168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.519168+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:55.489245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"507'2","reported_seq":1214,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.589165+0000","last_change":"2024-04-25T05:54:33.756679+0000","last_active":"2024-04-25T05:55:32.589165+0000","last_peered":"2024-04-25T05:55:32.589165+0000","last_clean":"2024-04-25T05:55:32.589165+0000","last_became_active":"2024-04-25T05:46:34.971175+0000","last_became_peered":"2024-04-25T05:46:34.971175+0000","last_unstale":"2024-04-25T05:55:32.589165+0000","last_undegraded":"2024-04-25T05:55:32.589165+0000","last_fullsized":"2024-04-25T05:55:32.589165+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-04-25T05:54:33.756467+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-04-25T05:52:08.192693+0000","last_clean_scrub_stamp":"2024-04-25T05:54:33.756467+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:47.408397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"614'36","reported_seq":1217,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372393+0000","last_change":"2024-04-25T05:55:15.821674+0000","last_active":"2024-04-25T05:55:32.372393+0000","last_peered":"2024-04-25T05:55:32.372393+0000","last_clean":"2024-04-25T05:55:32.372393+0000","last_became_active":"2024-04-25T05:34:00.684698+0000","last_became_peered":"2024-04-25T05:34:00.684698+0000","last_unstale":"2024-04-25T05:55:32.372393+0000","last_undegraded":"2024-04-25T05:55:32.372393+0000","last_fullsized":"2024-04-25T05:55:32.372393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"614'36","last_scrub_stamp":"2024-04-25T05:55:15.821567+0000","last_deep_scrub":"614'36","last_deep_scrub_stamp":"2024-04-25T05:55:15.821567+0000","last_clean_scrub_stamp":"2024-04-25T05:55:15.821567+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:20.944313+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1220,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244050+0000","last_change":"2024-04-25T05:54:50.243860+0000","last_active":"2024-04-25T05:55:33.244050+0000","last_peered":"2024-04-25T05:55:33.244050+0000","last_clean":"2024-04-25T05:55:33.244050+0000","last_became_active":"2024-04-25T05:46:19.974251+0000","last_became_peered":"2024-04-25T05:46:19.974251+0000","last_unstale":"2024-04-25T05:55:33.244050+0000","last_undegraded":"2024-04-25T05:55:33.244050+0000","last_fullsized":"2024-04-25T05:55:33.244050+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:50.243784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:50.243784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:09.382174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":277,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371137+0000","last_change":"2024-04-25T05:55:20.569637+0000","last_active":"2024-04-25T05:55:32.371137+0000","last_peered":"2024-04-25T05:55:32.371137+0000","last_clean":"2024-04-25T05:55:32.371137+0000","last_became_active":"2024-04-25T05:51:40.104867+0000","last_became_peered":"2024-04-25T05:51:40.104867+0000","last_unstale":"2024-04-25T05:55:32.371137+0000","last_undegraded":"2024-04-25T05:55:32.371137+0000","last_fullsized":"2024-04-25T05:55:32.371137+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:20.569509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:48.415168+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.569509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:44.237659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1176,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.373437+0000","last_change":"2024-04-25T05:54:57.108076+0000","last_active":"2024-04-25T05:55:32.373437+0000","last_peered":"2024-04-25T05:55:32.373437+0000","last_clean":"2024-04-25T05:55:32.373437+0000","last_became_active":"2024-04-25T05:46:34.972481+0000","last_became_peered":"2024-04-25T05:46:34.972481+0000","last_unstale":"2024-04-25T05:55:32.373437+0000","last_undegraded":"2024-04-25T05:55:32.373437+0000","last_fullsized":"2024-04-25T05:55:32.373437+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:57.107977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:52.519375+0000","last_clean_scrub_stamp":"2024-04-25T05:54:57.107977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:26.554767+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"581'46","reported_seq":568,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-04-25T05:55:25.585753+0000","last_change":"2024-04-25T05:54:26.442804+0000","last_active":"2024-04-25T05:55:25.585753+0000","last_peered":"2024-04-25T05:55:25.585753+0000","last_clean":"2024-04-25T05:55:25.585753+0000","last_became_active":"2024-04-25T05:46:19.995835+0000","last_became_peered":"2024-04-25T05:46:19.995835+0000","last_unstale":"2024-04-25T05:55:25.585753+0000","last_undegraded":"2024-04-25T05:55:25.585753+0000","last_fullsized":"2024-04-25T05:55:25.585753+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'46","last_scrub_stamp":"2024-04-25T05:54:26.442689+0000","last_deep_scrub":"150'22","last_deep_scrub_stamp":"2024-04-25T05:37:50.872688+0000","last_clean_scrub_stamp":"2024-04-25T05:54:26.442689+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:36.083886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778732+0000","last_change":"2024-04-25T05:54:34.255892+0000","last_active":"2024-04-25T05:55:32.778732+0000","last_peered":"2024-04-25T05:55:32.778732+0000","last_clean":"2024-04-25T05:55:32.778732+0000","last_became_active":"2024-04-25T05:51:40.098893+0000","last_became_peered":"2024-04-25T05:51:40.098893+0000","last_unstale":"2024-04-25T05:55:32.778732+0000","last_undegraded":"2024-04-25T05:55:32.778732+0000","last_fullsized":"2024-04-25T05:55:32.778732+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.255784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.255784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:40.426091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1199,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372976+0000","last_change":"2024-04-25T05:54:34.454920+0000","last_active":"2024-04-25T05:55:32.372976+0000","last_peered":"2024-04-25T05:55:32.372976+0000","last_clean":"2024-04-25T05:55:32.372976+0000","last_became_active":"2024-04-25T05:51:40.109212+0000","last_became_peered":"2024-04-25T05:51:40.109212+0000","last_unstale":"2024-04-25T05:55:32.372976+0000","last_undegraded":"2024-04-25T05:55:32.372976+0000","last_fullsized":"2024-04-25T05:55:32.372976+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.454834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.454834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:42.196066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1216,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244380+0000","last_change":"2024-04-25T05:54:44.330594+0000","last_active":"2024-04-25T05:55:33.244380+0000","last_peered":"2024-04-25T05:55:33.244380+0000","last_clean":"2024-04-25T05:55:33.244380+0000","last_became_active":"2024-04-25T05:46:34.968090+0000","last_became_peered":"2024-04-25T05:46:34.968090+0000","last_unstale":"2024-04-25T05:55:33.244380+0000","last_undegraded":"2024-04-25T05:55:33.244380+0000","last_fullsized":"2024-04-25T05:55:33.244380+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:44.330464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:42:24.062132+0000","last_clean_scrub_stamp":"2024-04-25T05:54:44.330464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:58.127089+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"581'15","reported_seq":697,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244718+0000","last_change":"2024-04-25T05:55:22.128467+0000","last_active":"2024-04-25T05:55:33.244718+0000","last_peered":"2024-04-25T05:55:33.244718+0000","last_clean":"2024-04-25T05:55:33.244718+0000","last_became_active":"2024-04-25T05:49:09.553768+0000","last_became_peered":"2024-04-25T05:49:09.553768+0000","last_unstale":"2024-04-25T05:55:33.244718+0000","last_undegraded":"2024-04-25T05:55:33.244718+0000","last_fullsized":"2024-04-25T05:55:33.244718+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'15","last_scrub_stamp":"2024-04-25T05:55:22.128353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:55:22.128353+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:47.768433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756167+0000","last_change":"2024-04-25T05:54:59.609213+0000","last_active":"2024-04-25T05:55:32.756167+0000","last_peered":"2024-04-25T05:55:32.756167+0000","last_clean":"2024-04-25T05:55:32.756167+0000","last_became_active":"2024-04-25T05:46:34.986530+0000","last_became_peered":"2024-04-25T05:46:34.986530+0000","last_unstale":"2024-04-25T05:55:32.756167+0000","last_undegraded":"2024-04-25T05:55:32.756167+0000","last_fullsized":"2024-04-25T05:55:32.756167+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:59.609135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:52:47.057728+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.609135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:24.406325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1174,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.588645+0000","last_change":"2024-04-25T05:54:30.693524+0000","last_active":"2024-04-25T05:55:32.588645+0000","last_peered":"2024-04-25T05:55:32.588645+0000","last_clean":"2024-04-25T05:55:32.588645+0000","last_became_active":"2024-04-25T05:45:36.201276+0000","last_became_peered":"2024-04-25T05:45:36.201276+0000","last_unstale":"2024-04-25T05:55:32.588645+0000","last_undegraded":"2024-04-25T05:55:32.588645+0000","last_fullsized":"2024-04-25T05:55:32.588645+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:30.693428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:30.693428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755518+0000","last_change":"2024-04-25T05:55:10.771336+0000","last_active":"2024-04-25T05:55:32.755518+0000","last_peered":"2024-04-25T05:55:32.755518+0000","last_clean":"2024-04-25T05:55:32.755518+0000","last_became_active":"2024-04-25T05:46:34.985082+0000","last_became_peered":"2024-04-25T05:46:34.985082+0000","last_unstale":"2024-04-25T05:55:32.755518+0000","last_undegraded":"2024-04-25T05:55:32.755518+0000","last_fullsized":"2024-04-25T05:55:32.755518+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:10.771219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:51:31.994196+0000","last_clean_scrub_stamp":"2024-04-25T05:55:10.771219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:39.669662+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"581'40","reported_seq":1017,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.244063+0000","last_change":"2024-04-25T05:55:20.110162+0000","last_active":"2024-04-25T05:55:33.244063+0000","last_peered":"2024-04-25T05:55:33.244063+0000","last_clean":"2024-04-25T05:55:33.244063+0000","last_became_active":"2024-04-25T05:51:40.097822+0000","last_became_peered":"2024-04-25T05:51:40.097822+0000","last_unstale":"2024-04-25T05:55:33.244063+0000","last_undegraded":"2024-04-25T05:55:33.244063+0000","last_fullsized":"2024-04-25T05:55:33.244063+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'40","last_scrub_stamp":"2024-04-25T05:55:20.110035+0000","last_deep_scrub":"581'40","last_deep_scrub_stamp":"2024-04-25T05:55:20.110035+0000","last_clean_scrub_stamp":"2024-04-25T05:55:20.110035+0000","objects_scrubbed":6,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:28.879437+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":511,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.756138+0000","last_change":"2024-04-25T05:54:41.775664+0000","last_active":"2024-04-25T05:55:32.756138+0000","last_peered":"2024-04-25T05:55:32.756138+0000","last_clean":"2024-04-25T05:55:32.756138+0000","last_became_active":"2024-04-25T05:46:34.978583+0000","last_became_peered":"2024-04-25T05:46:34.978583+0000","last_unstale":"2024-04-25T05:55:32.756138+0000","last_undegraded":"2024-04-25T05:55:32.756138+0000","last_fullsized":"2024-04-25T05:55:32.756138+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:41.775569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:35:00.716191+0000","last_clean_scrub_stamp":"2024-04-25T05:54:41.775569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:44.061066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":517,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.755657+0000","last_change":"2024-04-25T05:55:11.701662+0000","last_active":"2024-04-25T05:55:32.755657+0000","last_peered":"2024-04-25T05:55:32.755657+0000","last_clean":"2024-04-25T05:55:32.755657+0000","last_became_active":"2024-04-25T05:51:40.083680+0000","last_became_peered":"2024-04-25T05:51:40.083680+0000","last_unstale":"2024-04-25T05:55:32.755657+0000","last_undegraded":"2024-04-25T05:55:32.755657+0000","last_fullsized":"2024-04-25T05:55:32.755657+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:11.701557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:11.701557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:13.957487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":261,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.778119+0000","last_change":"2024-04-25T05:53:54.281982+0000","last_active":"2024-04-25T05:55:32.778119+0000","last_peered":"2024-04-25T05:55:32.778119+0000","last_clean":"2024-04-25T05:55:32.778119+0000","last_became_active":"2024-04-25T05:51:40.110038+0000","last_became_peered":"2024-04-25T05:51:40.110038+0000","last_unstale":"2024-04-25T05:55:32.778119+0000","last_undegraded":"2024-04-25T05:55:32.778119+0000","last_fullsized":"2024-04-25T05:55:32.778119+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:54.281861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:50:08.944412+0000","last_clean_scrub_stamp":"2024-04-25T05:53:54.281861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"582'63","reported_seq":1507,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371983+0000","last_change":"2024-04-25T05:54:59.857846+0000","last_active":"2024-04-25T05:55:32.371983+0000","last_peered":"2024-04-25T05:55:32.371983+0000","last_clean":"2024-04-25T05:55:32.371983+0000","last_became_active":"2024-04-25T05:51:40.100412+0000","last_became_peered":"2024-04-25T05:51:40.100412+0000","last_unstale":"2024-04-25T05:55:32.371983+0000","last_undegraded":"2024-04-25T05:55:32.371983+0000","last_fullsized":"2024-04-25T05:55:32.371983+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"582'63","last_scrub_stamp":"2024-04-25T05:54:59.857725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:59.667161+0000","last_clean_scrub_stamp":"2024-04-25T05:54:59.857725+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:14.283169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":98,"num_read_kb":85,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372595+0000","last_change":"2024-04-25T05:53:59.778523+0000","last_active":"2024-04-25T05:55:32.372595+0000","last_peered":"2024-04-25T05:55:32.372595+0000","last_clean":"2024-04-25T05:55:32.372595+0000","last_became_active":"2024-04-25T05:46:19.997260+0000","last_became_peered":"2024-04-25T05:46:19.997260+0000","last_unstale":"2024-04-25T05:55:32.372595+0000","last_undegraded":"2024-04-25T05:55:32.372595+0000","last_fullsized":"2024-04-25T05:55:32.372595+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:53:59.778392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:40:47.485053+0000","last_clean_scrub_stamp":"2024-04-25T05:53:59.778392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":749,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243337+0000","last_change":"2024-04-25T05:55:32.210629+0000","last_active":"2024-04-25T05:55:33.243337+0000","last_peered":"2024-04-25T05:55:33.243337+0000","last_clean":"2024-04-25T05:55:33.243337+0000","last_became_active":"2024-04-25T05:49:09.556177+0000","last_became_peered":"2024-04-25T05:49:09.556177+0000","last_unstale":"2024-04-25T05:55:33.243337+0000","last_undegraded":"2024-04-25T05:55:33.243337+0000","last_fullsized":"2024-04-25T05:55:33.243337+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:32.210503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:55:32.210503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:49.628442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1162,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.372289+0000","last_change":"2024-04-25T05:54:34.080973+0000","last_active":"2024-04-25T05:55:32.372289+0000","last_peered":"2024-04-25T05:55:32.372289+0000","last_clean":"2024-04-25T05:55:32.372289+0000","last_became_active":"2024-04-25T05:39:29.798734+0000","last_became_peered":"2024-04-25T05:39:29.798734+0000","last_unstale":"2024-04-25T05:55:32.372289+0000","last_undegraded":"2024-04-25T05:55:32.372289+0000","last_fullsized":"2024-04-25T05:55:32.372289+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:34.080845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:54:34.080845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.146065+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"581'32","reported_seq":1285,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243193+0000","last_change":"2024-04-25T05:54:40.264714+0000","last_active":"2024-04-25T05:55:33.243193+0000","last_peered":"2024-04-25T05:55:33.243193+0000","last_clean":"2024-04-25T05:55:33.243193+0000","last_became_active":"2024-04-25T05:39:29.786841+0000","last_became_peered":"2024-04-25T05:39:29.786841+0000","last_unstale":"2024-04-25T05:55:33.243193+0000","last_undegraded":"2024-04-25T05:55:33.243193+0000","last_fullsized":"2024-04-25T05:55:33.243193+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'32","last_scrub_stamp":"2024-04-25T05:54:40.264562+0000","last_deep_scrub":"211'20","last_deep_scrub_stamp":"2024-04-25T05:39:42.695422+0000","last_clean_scrub_stamp":"2024-04-25T05:54:40.264562+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:07.392246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":269,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.779037+0000","last_change":"2024-04-25T05:55:03.284944+0000","last_active":"2024-04-25T05:55:32.779037+0000","last_peered":"2024-04-25T05:55:32.779037+0000","last_clean":"2024-04-25T05:55:32.779037+0000","last_became_active":"2024-04-25T05:51:40.091231+0000","last_became_peered":"2024-04-25T05:51:40.091231+0000","last_unstale":"2024-04-25T05:55:32.779037+0000","last_undegraded":"2024-04-25T05:55:32.779037+0000","last_fullsized":"2024-04-25T05:55:32.779037+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:03.284841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:33:57.656996+0000","last_clean_scrub_stamp":"2024-04-25T05:55:03.284841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:29.572890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1187,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:32.371268+0000","last_change":"2024-04-25T05:54:29.444628+0000","last_active":"2024-04-25T05:55:32.371268+0000","last_peered":"2024-04-25T05:55:32.371268+0000","last_clean":"2024-04-25T05:55:32.371268+0000","last_became_active":"2024-04-25T05:46:34.974087+0000","last_became_peered":"2024-04-25T05:46:34.974087+0000","last_unstale":"2024-04-25T05:55:32.371268+0000","last_undegraded":"2024-04-25T05:55:32.371268+0000","last_fullsized":"2024-04-25T05:55:32.371268+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:54:29.444552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:01.680110+0000","last_clean_scrub_stamp":"2024-04-25T05:54:29.444552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:55:46.015362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":725,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-04-25T05:55:33.243603+0000","last_change":"2024-04-25T05:55:25.211610+0000","last_active":"2024-04-25T05:55:33.243603+0000","last_peered":"2024-04-25T05:55:33.243603+0000","last_clean":"2024-04-25T05:55:33.243603+0000","last_became_active":"2024-04-25T05:49:09.556916+0000","last_became_peered":"2024-04-25T05:49:09.556916+0000","last_unstale":"2024-04-25T05:55:33.243603+0000","last_undegraded":"2024-04-25T05:55:33.243603+0000","last_fullsized":"2024-04-25T05:55:33.243603+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T05:55:25.211471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T05:34:03.694746+0000","last_clean_scrub_stamp":"2024-04-25T05:55:25.211471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:56:25.450911+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1275,"num_read_kb":1270,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1818,"num_read_kb":1543,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":11474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070729,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931784,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92813368,"statfs":{"total":95995035648,"available":95040888832,"internally_reserved":0,"allocated":2424832,"data_stored":1492566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":951712202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365069,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":867580,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92877572,"statfs":{"total":95995035648,"available":95106633728,"internally_reserved":0,"allocated":2080768,"data_stored":1170401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":886307270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129376,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":1127294,"kb_avail":92615776,"statfs":{"total":95995035648,"available":94838554624,"internally_reserved":0,"allocated":2121728,"data_stored":1170317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1154349509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769036,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92976116,"statfs":{"total":95995035648,"available":95207542784,"internally_reserved":0,"allocated":2097152,"data_stored":1169766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727600,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017552,"statfs":{"total":95995035648,"available":95249973248,"internally_reserved":0,"allocated":2334720,"data_stored":1425318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149979,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1219928,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":1217278,"kb_avail":92525224,"statfs":{"total":95995035648,"available":94745829376,"internally_reserved":0,"allocated":2703360,"data_stored":1760231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1246493127},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986454,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827336,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92917816,"statfs":{"total":95995035648,"available":95147843584,"internally_reserved":0,"allocated":2097152,"data_stored":1163860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692114,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637684,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93107468,"statfs":{"total":95995035648,"available":95342047232,"internally_reserved":0,"allocated":2666496,"data_stored":1753522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T05:55:38.154 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-25T05:55:38.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737 shell --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-25T05:55:38.405 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T05:55:38.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[37492]: 2.2 scrub starts 2024-04-25T05:55:38.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[37492]: 2.2 scrub ok 2024-04-25T05:55:38.641 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[37492]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[27010]: 2.2 scrub starts 2024-04-25T05:55:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[27010]: 2.2 scrub ok 2024-04-25T05:55:38.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:38 smithi043 ceph-mon[27010]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:38 smithi060 ceph-mon[30898]: 2.2 scrub starts 2024-04-25T05:55:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:38 smithi060 ceph-mon[30898]: 2.2 scrub ok 2024-04-25T05:55:38.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:38 smithi060 ceph-mon[30898]: from='client.26882 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T05:55:39.311 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE Bus STOPPING 2024-04-25T05:55:39.311 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:55:39.311 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE Bus STOPPED 2024-04-25T05:55:39.311 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE Bus STARTING 2024-04-25T05:55:39.311 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE Serving on http://:::9283 2024-04-25T05:55:39.641 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:39] ENGINE Bus STARTED 2024-04-25T05:55:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 2.0 scrub starts 2024-04-25T05:55:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 2.0 scrub ok 2024-04-25T05:55:39.641 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.8 scrub starts 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.8 scrub ok 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 6.13 scrub starts 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 6.13 scrub ok 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.1c scrub starts 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.1c scrub ok 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.1e deep-scrub starts 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: 4.1e deep-scrub ok 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:55:39.642 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[27010]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 2.0 scrub starts 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 2.0 scrub ok 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.8 scrub starts 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.8 scrub ok 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 6.13 scrub starts 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 6.13 scrub ok 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.1c scrub starts 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.1c scrub ok 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:39.643 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.1e deep-scrub starts 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: 4.1e deep-scrub ok 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:55:39.644 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-mon[37492]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 2.0 scrub starts 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 2.0 scrub ok 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.8 scrub starts 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.8 scrub ok 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 6.13 scrub starts 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 6.13 scrub ok 2024-04-25T05:55:39.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.1c scrub starts 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.1c scrub ok 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: pgmap v1178: 142 pgs: 142 active+clean; 583 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.1e deep-scrub starts 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: 4.1e deep-scrub ok 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='client.? 172.21.15.43:0/3791107044' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T05:55:39.750 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:39 smithi060 ceph-mon[30898]: from='mgr.24509 172.21.15.43:0/416043969' entity='mgr.y' 2024-04-25T05:55:39.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:39 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: ::ffff:172.21.15.60 - - [25/Apr/2024:05:55:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-25T05:55:40.025 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T05:55:40.035 INFO:tasks.cephadm:Teardown begin 2024-04-25T05:55:40.035 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:55:40.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:55:40.134 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T05:55:40.134 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T05:55:40.164 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T05:55:40.190 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T05:55:40.190 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T05:55:40.190 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a 2024-04-25T05:55:40.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: 3.1e deep-scrub starts 2024-04-25T05:55:40.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: 3.1e deep-scrub ok 2024-04-25T05:55:40.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: 5.1a scrub starts 2024-04-25T05:55:40.551 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: 5.1a scrub ok 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: purged_snaps scrub starts 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[37492]: purged_snaps scrub ok 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 systemd[1]: Stopping Ceph mon.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: 3.1e deep-scrub starts 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: 3.1e deep-scrub ok 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: 5.1a scrub starts 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: 5.1a scrub ok 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: purged_snaps scrub starts 2024-04-25T05:55:40.552 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-mon[27010]: purged_snaps scrub ok 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: 3.1e deep-scrub starts 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: 3.1e deep-scrub ok 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: 5.1a scrub starts 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: 5.1a scrub ok 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: purged_snaps scrub starts 2024-04-25T05:55:40.749 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:40 smithi060 ceph-mon[30898]: purged_snaps scrub ok 2024-04-25T05:55:40.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:55:40.550+0000 7f1744473700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:40.891 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a[27005]: 2024-04-25T05:55:40.550+0000 7f1744473700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:40 smithi043 podman[210725]: 2024-04-25 05:55:40.61243057 +0000 UTC m=+0.196119624 container died 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE Bus STOPPING 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE Bus STOPPED 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE Bus STARTING 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE Serving on http://:::9283 2024-04-25T05:55:40.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:40 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: [25/Apr/2024:05:55:40] ENGINE Bus STARTED 2024-04-25T05:55:41.175 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:41 smithi043 podman[210725]: 2024-04-25 05:55:41.159588045 +0000 UTC m=+0.743277096 container cleanup 49b51fc8c9a1420ef50f6eb50a44aab4cdb0c85b2c68eb5e075d76dc16a6616e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:55:41.175 INFO:journalctl@ceph.mon.a.smithi043.stdout:Apr 25 05:55:41 smithi043 bash[210725]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-a 2024-04-25T05:55:41.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T05:55:41.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:41.698 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T05:55:41.698 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-25T05:55:41.698 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.c 2024-04-25T05:55:42.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:41 smithi043 systemd[1]: Stopping Ceph mon.c for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c[37487]: 2024-04-25T05:55:42.014+0000 7f61c322d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:42 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c[37487]: 2024-04-25T05:55:42.014+0000 7f61c322d700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T05:55:42.391 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:42 smithi043 podman[210857]: 2024-04-25 05:55:42.229142495 +0000 UTC m=+0.349464680 container died 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T05:55:42.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:42 smithi043 podman[210857]: 2024-04-25 05:55:42.815313565 +0000 UTC m=+0.935635757 container cleanup 3731e521cc92d959acd53dd79dbddec3160c3be19a3f1171a1f573249a83d531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:55:42.891 INFO:journalctl@ceph.mon.c.smithi043.stdout:Apr 25 05:55:42 smithi043 bash[210857]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-c 2024-04-25T05:55:43.250 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T05:55:43.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:43.285 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-25T05:55:43.285 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-25T05:55:43.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.b 2024-04-25T05:55:43.617 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:43 smithi060 systemd[1]: Stopping Ceph mon.b for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:43.974 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b[30880]: 2024-04-25T05:55:43.616+0000 7fae5ba9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:43.974 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b[30880]: 2024-04-25T05:55:43.616+0000 7fae5ba9d700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T05:55:43.974 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:43 smithi060 podman[151213]: 2024-04-25 05:55:43.729624483 +0000 UTC m=+0.214980373 container died 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T05:55:44.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:55:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug there is no tcmu-runner data available 2024-04-25T05:55:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 podman[151213]: 2024-04-25 05:55:44.159320479 +0000 UTC m=+0.644676386 container cleanup 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T05:55:44.249 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 bash[151213]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b 2024-04-25T05:55:44.630 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T05:55:44.654 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 podman[151227]: 2024-04-25 05:55:44.402878748 +0000 UTC m=+0.676459153 container remove 041addbd0a3d65e276fb48e82a97896e35742a587b5815e41af4ff82f89558c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T05:55:44.654 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-25T05:55:44.654 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 systemd[1]: Stopped Ceph mon.b for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:55:44.654 INFO:journalctl@ceph.mon.b.smithi060.stdout:Apr 25 05:55:44 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mon.b.service: Consumed 23.744s CPU time. 2024-04-25T05:55:44.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:44.681 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-25T05:55:44.681 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T05:55:44.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y 2024-04-25T05:55:44.994 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:44 smithi043 systemd[1]: Stopping Ceph mgr.y for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:45.345 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:55:44.993+0000 7fa190002700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:45.345 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:44 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y[27238]: 2024-04-25T05:55:44.993+0000 7fa190002700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T05:55:45.345 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:45 smithi043 podman[210978]: 2024-04-25 05:55:45.025394939 +0000 UTC m=+0.144566925 container died 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T05:55:45.596 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:45 smithi043 podman[210978]: 2024-04-25 05:55:45.57053138 +0000 UTC m=+0.689703359 container cleanup 082b909b97197ba55a2c06c82dbd48067c2227b3476ad6f311a73d73f5c3e023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d6f5f4e5ec08db21b2896cb71b4cb907b7fb4737, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T05:55:45.596 INFO:journalctl@ceph.mgr.y.smithi043.stdout:Apr 25 05:55:45 smithi043 bash[210978]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-y 2024-04-25T05:55:46.170 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T05:55:46.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:46.205 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T05:55:46.205 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T05:55:46.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.x 2024-04-25T05:55:46.499 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:55:46 smithi060 systemd[1]: Stopping Ceph mgr.x for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:46.814 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:55:46 smithi060 podman[151335]: 2024-04-25 05:55:46.562397429 +0000 UTC m=+0.157587107 container died 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T05:55:47.232 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:55:46 smithi060 podman[151335]: 2024-04-25 05:55:46.914006067 +0000 UTC m=+0.509195747 container cleanup 81c55a78bdd3a67bfc2af9715df66253693a0cd7c1212282f3389ab5b76f755b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True) 2024-04-25T05:55:47.232 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Apr 25 05:55:46 smithi060 bash[151335]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-mgr-x 2024-04-25T05:55:47.356 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T05:55:47.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:47.390 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T05:55:47.390 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T05:55:47.390 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.0 2024-04-25T05:55:47.725 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:47 smithi043 systemd[1]: Stopping Ceph osd.0 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:48.141 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0[41306]: 2024-04-25T05:55:47.724+0000 7fba0358c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:48.141 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0[41306]: 2024-04-25T05:55:47.724+0000 7fba0358c700 -1 osd.0 621 *** Got signal Terminated *** 2024-04-25T05:55:48.141 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:47 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0[41306]: 2024-04-25T05:55:47.724+0000 7fba0358c700 -1 osd.0 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:55:53.634 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:53 smithi043 podman[211098]: 2024-04-25 05:55:53.28699387 +0000 UTC m=+5.691506553 container died caad4115e9e1900395b266e0da05eaa4616680cddd6673ac76253230ee5cd8bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:55:53.891 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:53 smithi043 podman[211098]: 2024-04-25 05:55:53.800156021 +0000 UTC m=+6.204668702 container cleanup caad4115e9e1900395b266e0da05eaa4616680cddd6673ac76253230ee5cd8bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T05:55:53.891 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:53 smithi043 bash[211098]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0 2024-04-25T05:55:54.391 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211115]: 2024-04-25 05:55:54.068602195 +0000 UTC m=+0.775185009 container remove caad4115e9e1900395b266e0da05eaa4616680cddd6673ac76253230ee5cd8bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:55:54.789 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211205]: 2024-04-25 05:55:54.398622079 +0000 UTC m=+0.018419948 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:55:54.789 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211205]: 2024-04-25 05:55:54.511379524 +0000 UTC m=+0.131177426 container create f3522ceda644126923ab135f2abc0010af56b13dd46a9590bd4ad681b5451794 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS) 2024-04-25T05:55:54.789 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211205]: 2024-04-25 05:55:54.789092522 +0000 UTC m=+0.408890388 container init f3522ceda644126923ab135f2abc0010af56b13dd46a9590bd4ad681b5451794 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:55:55.141 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211205]: 2024-04-25 05:55:54.792870541 +0000 UTC m=+0.412668403 container start f3522ceda644126923ab135f2abc0010af56b13dd46a9590bd4ad681b5451794 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:55:55.141 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:54 smithi043 podman[211205]: 2024-04-25 05:55:54.947289461 +0000 UTC m=+0.567087318 container attach f3522ceda644126923ab135f2abc0010af56b13dd46a9590bd4ad681b5451794 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:55:55.802 INFO:journalctl@ceph.osd.0.smithi043.stdout:Apr 25 05:55:55 smithi043 podman[211205]: 2024-04-25 05:55:55.437085814 +0000 UTC m=+1.056883676 container died f3522ceda644126923ab135f2abc0010af56b13dd46a9590bd4ad681b5451794 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T05:55:56.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T05:55:56.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:55:56.180 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T05:55:56.180 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T05:55:56.181 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.1 2024-04-25T05:55:56.391 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:55:56 smithi043 systemd[1]: Stopping Ceph osd.1 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:55:56.891 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:55:56 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1[47378]: 2024-04-25T05:55:56.545+0000 7f62fb227700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:55:56.892 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:55:56 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1[47378]: 2024-04-25T05:55:56.545+0000 7f62fb227700 -1 osd.1 621 *** Got signal Terminated *** 2024-04-25T05:55:56.892 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:55:56 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1[47378]: 2024-04-25T05:55:56.545+0000 7f62fb227700 -1 osd.1 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:02.357 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:02 smithi043 podman[211323]: 2024-04-25 05:56:02.021189412 +0000 UTC m=+5.608028823 container died 849d24592cedba32b46247f51b7adb5375d9b1e800e84a3c147e2ae6da50d71c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:56:02.641 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:02 smithi043 podman[211323]: 2024-04-25 05:56:02.554382975 +0000 UTC m=+6.141222407 container cleanup 849d24592cedba32b46247f51b7adb5375d9b1e800e84a3c147e2ae6da50d71c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T05:56:02.641 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:02 smithi043 bash[211323]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1 2024-04-25T05:56:03.141 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:02 smithi043 podman[211341]: 2024-04-25 05:56:02.822585848 +0000 UTC m=+0.801166048 container remove 849d24592cedba32b46247f51b7adb5375d9b1e800e84a3c147e2ae6da50d71c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T05:56:03.526 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.14509627 +0000 UTC m=+0.018625663 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:03.526 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.281707964 +0000 UTC m=+0.155237357 container create 1374328241de4389f2c499573ab48d54ad21a92617a75aa72be821b58b297097 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T05:56:03.526 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.525723978 +0000 UTC m=+0.399253373 container init 1374328241de4389f2c499573ab48d54ad21a92617a75aa72be821b58b297097 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T05:56:03.830 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.529510692 +0000 UTC m=+0.403040094 container start 1374328241de4389f2c499573ab48d54ad21a92617a75aa72be821b58b297097 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T05:56:03.830 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.634138756 +0000 UTC m=+0.507668162 container attach 1374328241de4389f2c499573ab48d54ad21a92617a75aa72be821b58b297097 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T05:56:04.141 INFO:journalctl@ceph.osd.1.smithi043.stdout:Apr 25 05:56:03 smithi043 podman[211430]: 2024-04-25 05:56:03.831243505 +0000 UTC m=+0.704772901 container died 1374328241de4389f2c499573ab48d54ad21a92617a75aa72be821b58b297097 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T05:56:04.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T05:56:04.582 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:04.582 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T05:56:04.583 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T05:56:04.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.2 2024-04-25T05:56:04.891 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:04 smithi043 systemd[1]: Stopping Ceph osd.2 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:05.391 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:04 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2[53089]: 2024-04-25T05:56:04.924+0000 7fc499a1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:05.391 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:04 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2[53089]: 2024-04-25T05:56:04.924+0000 7fc499a1a700 -1 osd.2 621 *** Got signal Terminated *** 2024-04-25T05:56:05.392 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:04 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2[53089]: 2024-04-25T05:56:04.924+0000 7fc499a1a700 -1 osd.2 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:10.892 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:10 smithi043 podman[211535]: 2024-04-25 05:56:10.59391316 +0000 UTC m=+5.786266555 container died 0542d077cfee86f2e9846e9ef6a306cb7366cb838f856c6fc1c3621819534802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T05:56:11.373 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211535]: 2024-04-25 05:56:11.106208123 +0000 UTC m=+6.298561516 container cleanup 0542d077cfee86f2e9846e9ef6a306cb7366cb838f856c6fc1c3621819534802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T05:56:11.373 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 bash[211535]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2 2024-04-25T05:56:11.641 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211554]: 2024-04-25 05:56:11.374275603 +0000 UTC m=+0.773394710 container remove 0542d077cfee86f2e9846e9ef6a306cb7366cb838f856c6fc1c3621819534802 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T05:56:11.985 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211642]: 2024-04-25 05:56:11.584560007 +0000 UTC m=+0.018624730 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:11.985 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211642]: 2024-04-25 05:56:11.733406384 +0000 UTC m=+0.167471094 container create ff97480de2ea2b819bab5106d22398e11bd001d29c5d6501011ab83408e34c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T05:56:11.985 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211642]: 2024-04-25 05:56:11.985621465 +0000 UTC m=+0.419686177 container init ff97480de2ea2b819bab5106d22398e11bd001d29c5d6501011ab83408e34c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T05:56:12.295 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:11 smithi043 podman[211642]: 2024-04-25 05:56:11.988893506 +0000 UTC m=+0.422958220 container start ff97480de2ea2b819bab5106d22398e11bd001d29c5d6501011ab83408e34c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T05:56:12.296 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:12 smithi043 podman[211642]: 2024-04-25 05:56:12.09411013 +0000 UTC m=+0.528174850 container attach ff97480de2ea2b819bab5106d22398e11bd001d29c5d6501011ab83408e34c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T05:56:12.641 INFO:journalctl@ceph.osd.2.smithi043.stdout:Apr 25 05:56:12 smithi043 podman[211642]: 2024-04-25 05:56:12.29705902 +0000 UTC m=+0.731123732 container died ff97480de2ea2b819bab5106d22398e11bd001d29c5d6501011ab83408e34c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:56:12.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T05:56:12.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:12.980 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T05:56:12.980 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T05:56:12.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.3 2024-04-25T05:56:13.300 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:13 smithi043 systemd[1]: Stopping Ceph osd.3 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:13.641 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:13 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:13.299+0000 7f8906efd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:13.641 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:13 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:13.299+0000 7f8906efd700 -1 osd.3 621 *** Got signal Terminated *** 2024-04-25T05:56:13.641 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:13 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:13.299+0000 7f8906efd700 -1 osd.3 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:14.056 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:13.854+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:14.056 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:13.759+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:14.999 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:14 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:14.892+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:14.999 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:14 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:14.746+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:15.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:15 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:15.317+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:15.891 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:15 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:15.456+0000 7f89034f1700 -1 osd.3 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.628108+0000 front 2024-04-25T05:55:49.628312+0000 (oldest deadline 2024-04-25T05:56:14.928367+0000) 2024-04-25T05:56:16.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:15 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:15.872+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:16.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:15 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:15.785+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:16.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:16 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:16.319+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:16.891 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:16 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:16.443+0000 7f89034f1700 -1 osd.3 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.628108+0000 front 2024-04-25T05:55:49.628312+0000 (oldest deadline 2024-04-25T05:56:14.928367+0000) 2024-04-25T05:56:17.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:16 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:16.833+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:17.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:16 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:16.918+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:17.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:17 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:17.341+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:17.891 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:17 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3[59209]: 2024-04-25T05:56:17.449+0000 7f89034f1700 -1 osd.3 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.628108+0000 front 2024-04-25T05:55:49.628312+0000 (oldest deadline 2024-04-25T05:56:14.928367+0000) 2024-04-25T05:56:18.214 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:17 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:17.887+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:18.214 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:17 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:17.859+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:18.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:18.346+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:18.499 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:18.213+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:19.168 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:18.888+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:19.168 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:18 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:18.871+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:19.208 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:18 smithi043 podman[211746]: 2024-04-25 05:56:18.909879479 +0000 UTC m=+5.731622094 container died 9a285bea549d41efd5ecaef0fc247b60078bb5197486eb69d0a2fa2f87ac57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T05:56:19.499 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:19 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:19.167+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:19.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:19 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:19.384+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:19.641 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:19 smithi043 podman[211746]: 2024-04-25 05:56:19.407974356 +0000 UTC m=+6.229716979 container cleanup 9a285bea549d41efd5ecaef0fc247b60078bb5197486eb69d0a2fa2f87ac57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T05:56:19.642 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:19 smithi043 bash[211746]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3 2024-04-25T05:56:19.991 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:19 smithi043 podman[211765]: 2024-04-25 05:56:19.659218626 +0000 UTC m=+0.745413949 container remove 9a285bea549d41efd5ecaef0fc247b60078bb5197486eb69d0a2fa2f87ac57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T05:56:20.198 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:19 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:19.862+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:20.198 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:19 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:19.877+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:20.311 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:19 smithi043 podman[211860]: 2024-04-25 05:56:19.894574775 +0000 UTC m=+0.018375545 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:20.312 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:20 smithi043 podman[211860]: 2024-04-25 05:56:20.020235564 +0000 UTC m=+0.144036363 container create f7fa24b2be17b4355cea24f15780ca5b0e8b207ce05bb95c6938edf79e856c10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:56:20.312 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:20 smithi043 podman[211860]: 2024-04-25 05:56:20.228905686 +0000 UTC m=+0.352706446 container init f7fa24b2be17b4355cea24f15780ca5b0e8b207ce05bb95c6938edf79e856c10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:56:20.312 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:20 smithi043 podman[211860]: 2024-04-25 05:56:20.232157731 +0000 UTC m=+0.355958491 container start f7fa24b2be17b4355cea24f15780ca5b0e8b207ce05bb95c6938edf79e856c10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T05:56:20.312 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:20 smithi043 podman[211860]: 2024-04-25 05:56:20.312288276 +0000 UTC m=+0.436089046 container attach f7fa24b2be17b4355cea24f15780ca5b0e8b207ce05bb95c6938edf79e856c10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:56:20.449 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:20.197+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:20.450 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:20.411+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:20.450 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:56:20.447Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-04-25T05:56:20.450 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:56:20.448Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-04-25T05:56:20.450 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:56:20.450Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-04-25T05:56:20.641 INFO:journalctl@ceph.osd.3.smithi043.stdout:Apr 25 05:56:20 smithi043 podman[211860]: 2024-04-25 05:56:20.530177444 +0000 UTC m=+0.653978204 container died f7fa24b2be17b4355cea24f15780ca5b0e8b207ce05bb95c6938edf79e856c10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:56:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:56:20.451Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-04-25T05:56:20.749 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:56:20.453Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.43:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.43:8765: connect: connection refused" 2024-04-25T05:56:21.178 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:20.859+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:21.178 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:20 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:20.830+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:21.183 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T05:56:21.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:21.227 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T05:56:21.227 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T05:56:21.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.4 2024-04-25T05:56:21.452 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:21.178+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:21.453 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:21 smithi060 systemd[1]: Stopping Ceph osd.4 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:21.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:21.534+0000 7f368ecc5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:21.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:21.534+0000 7f368ecc5700 -1 osd.4 621 *** Got signal Terminated *** 2024-04-25T05:56:21.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:21.534+0000 7f368ecc5700 -1 osd.4 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:21.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:21.452+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:22.149 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:21.895+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:22.149 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:21 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:21.869+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:22.431 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:22.148+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:22.748 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:22.430+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:23.153 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:22.898+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:23.153 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:22 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:22.862+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:23.456 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:23.152+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:23.456 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:23.152+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.781890+0000 front 2024-04-25T05:55:57.781792+0000 (oldest deadline 2024-04-25T05:56:22.982424+0000) 2024-04-25T05:56:23.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:23.456+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:23.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:23.456+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:24.161 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:23.880+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:24.161 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:23 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:23.853+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:24.447 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:24.160+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:24.447 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:24.160+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.781890+0000 front 2024-04-25T05:55:57.781792+0000 (oldest deadline 2024-04-25T05:56:22.982424+0000) 2024-04-25T05:56:24.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:24.446+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:24.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:24.446+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:25.211 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:24.922+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:25.211 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:24.922+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:25.211 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:24.892+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:25.211 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:24 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:24.892+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:25.469 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:25.210+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:25.469 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:25.210+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.781890+0000 front 2024-04-25T05:55:57.781792+0000 (oldest deadline 2024-04-25T05:56:22.982424+0000) 2024-04-25T05:56:25.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:25.468+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:25.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:25.468+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:26.233 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:25.933+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:26.234 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:25.933+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:26.234 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:25.914+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:26.234 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:25 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:25.914+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:26.499 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:26.233+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:51.881290+0000 front 2024-04-25T05:55:51.881303+0000 (oldest deadline 2024-04-25T05:56:17.781167+0000) 2024-04-25T05:56:26.499 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4[35852]: 2024-04-25T05:56:26.233+0000 7f368b2b9700 -1 osd.4 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.781890+0000 front 2024-04-25T05:55:57.781792+0000 (oldest deadline 2024-04-25T05:56:22.982424+0000) 2024-04-25T05:56:26.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:26.469+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:26.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:26.469+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:27.137 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:26.938+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:27.138 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:26.938+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:27.138 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:26.872+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:27.138 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:26.872+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:27.440 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:27 smithi060 podman[151474]: 2024-04-25 05:56:27.138100173 +0000 UTC m=+5.703545868 container died dbf1f9cf96e3ace99b4bd42a5eca4c574d7b470d2dd0ec10b0a7d99ac3ddffd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T05:56:27.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:27.440+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:27.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:27.440+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:27.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:27 smithi060 podman[151474]: 2024-04-25 05:56:27.609119868 +0000 UTC m=+6.174565559 container cleanup dbf1f9cf96e3ace99b4bd42a5eca4c574d7b470d2dd0ec10b0a7d99ac3ddffd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T05:56:27.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:27 smithi060 bash[151474]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4 2024-04-25T05:56:28.182 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:27 smithi060 podman[151491]: 2024-04-25 05:56:27.835801728 +0000 UTC m=+0.702189465 container remove dbf1f9cf96e3ace99b4bd42a5eca4c574d7b470d2dd0ec10b0a7d99ac3ddffd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T05:56:28.182 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:27.983+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:28.182 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:27.983+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:28.183 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:27.887+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:28.183 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:27 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:27.887+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:28.436 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.086003811 +0000 UTC m=+0.018928808 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:28.437 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.198887717 +0000 UTC m=+0.131812757 container create f613689ed092004f7ed47f58489d9b925229aec7baf47dcf617e06c78b9008ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T05:56:28.437 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.414379167 +0000 UTC m=+0.347304155 container init f613689ed092004f7ed47f58489d9b925229aec7baf47dcf617e06c78b9008ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:56:28.437 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.417501723 +0000 UTC m=+0.350426719 container start f613689ed092004f7ed47f58489d9b925229aec7baf47dcf617e06c78b9008ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T05:56:28.749 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.539634292 +0000 UTC m=+0.472559277 container attach f613689ed092004f7ed47f58489d9b925229aec7baf47dcf617e06c78b9008ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T05:56:28.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:28.436+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:28.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:28.436+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:29.150 INFO:journalctl@ceph.osd.4.smithi060.stdout:Apr 25 05:56:28 smithi060 podman[151581]: 2024-04-25 05:56:28.855644289 +0000 UTC m=+0.788569274 container died f613689ed092004f7ed47f58489d9b925229aec7baf47dcf617e06c78b9008ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T05:56:29.151 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:29.028+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:29.151 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:29.028+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:29.151 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:28.873+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:29.151 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:28 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:28.873+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:29.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T05:56:29.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:29.535 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T05:56:29.535 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T05:56:29.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.5 2024-04-25T05:56:29.749 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 systemd[1]: Stopping Ceph osd.5 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:29.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:29.427+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:29.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:29.427+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:30.061 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:29.865+0000 7f65defee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:30.061 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:29.865+0000 7f65defee700 -1 osd.5 621 *** Got signal Terminated *** 2024-04-25T05:56:30.061 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:29.865+0000 7f65defee700 -1 osd.5 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:30.062 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:29.868+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:30.062 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:29 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:29.868+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:30.404 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:30.060+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:30.404 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:30.060+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:30.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:30.403+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:30.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:30.403+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:31.090 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:30.908+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:31.090 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:30 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:30.908+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:31.365 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:31.089+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:31.365 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:31.089+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:31.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:31.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:31.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:31.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:31.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:31.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:32.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:32.061+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:32.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:32.061+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:32.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:31.921+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:32.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:31.921+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:32.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:31 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:31.921+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:32.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:32.369+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:32.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:32.369+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:32.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:32.369+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:33.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:33.057+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:33.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:33.057+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:33.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:32.919+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:33.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:32.919+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:33.250 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:32 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:32.919+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:33.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:33.363+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:33.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:33.363+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:33.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:33.363+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:34.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:34.029+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.487199+0000 front 2024-04-25T05:55:49.487184+0000 (oldest deadline 2024-04-25T05:56:13.586940+0000) 2024-04-25T05:56:34.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5[41345]: 2024-04-25T05:56:34.029+0000 7f65db5e2700 -1 osd.5 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:59.388744+0000 front 2024-04-25T05:55:59.388731+0000 (oldest deadline 2024-04-25T05:56:24.688652+0000) 2024-04-25T05:56:34.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:33.968+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:34.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:33.968+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:34.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:33 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:33.968+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:34.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:34.368+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:34.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:34.368+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:34.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:34 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:34.368+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:35.330 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.011+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:35.331 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.011+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:35.331 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.011+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:35 smithi060 podman[151687]: 2024-04-25 05:56:35.371162643 +0000 UTC m=+5.618377097 container stop 7b74ca1df748879ac186e8a73f5a029ebccf5027f43a58c0e060ff24ab68ba92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:35 smithi060 podman[151687]: 2024-04-25 05:56:35.396984473 +0000 UTC m=+5.644198944 container died 7b74ca1df748879ac186e8a73f5a029ebccf5027f43a58c0e060ff24ab68ba92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:35.330+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:35.330+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:35.330+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:35.667 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:35.330+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:35.963 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:35 smithi060 podman[151687]: 2024-04-25 05:56:35.841028705 +0000 UTC m=+6.088243155 container cleanup 7b74ca1df748879ac186e8a73f5a029ebccf5027f43a58c0e060ff24ab68ba92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T05:56:35.963 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:35 smithi060 bash[151687]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5 2024-04-25T05:56:36.248 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151715]: 2024-04-25 05:56:36.02585512 +0000 UTC m=+0.648664500 container remove 7b74ca1df748879ac186e8a73f5a029ebccf5027f43a58c0e060ff24ab68ba92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T05:56:36.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.962+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:36.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.962+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:36.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:35 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:35.962+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:36.662 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:36.341+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:36.663 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:36.341+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:36.663 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:36.341+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:36.663 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:36.342+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:36.663 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.353063301 +0000 UTC m=+0.018584487 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:36.663 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.451738735 +0000 UTC m=+0.117259905 container create 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux ) 2024-04-25T05:56:36.939 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.662656099 +0000 UTC m=+0.328177270 container init 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T05:56:36.939 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.666657168 +0000 UTC m=+0.332178339 container start 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T05:56:36.939 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.746231701 +0000 UTC m=+0.411752874 container attach 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T05:56:37.249 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:36 smithi060 podman[151805]: 2024-04-25 05:56:36.963102337 +0000 UTC m=+0.628623515 container died 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-25T05:56:37.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:36.938+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:37.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:36.938+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:37.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:36 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:36.938+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:37.509 INFO:journalctl@ceph.osd.5.smithi060.stdout:Apr 25 05:56:37 smithi060 podman[151831]: 2024-04-25 05:56:37.508051896 +0000 UTC m=+0.534772929 container remove 2511280c6e4638fff9597d25ee120ed78a5ed2b8e7d1d81954997d3f9f17a830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T05:56:37.510 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:37.386+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:37.510 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:37.386+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:37.510 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:37.386+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:37.510 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:37.386+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:37.530 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T05:56:37.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:37.570 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T05:56:37.570 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T05:56:37.570 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.6 2024-04-25T05:56:37.885 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 systemd[1]: Stopping Ceph osd.6 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:37.886 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.885+0000 7f51d8dc1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:38.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.885+0000 7f51d8dc1700 -1 osd.6 621 *** Got signal Terminated *** 2024-04-25T05:56:38.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.885+0000 7f51d8dc1700 -1 osd.6 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:38.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.959+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:38.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.959+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:38.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:37 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:37.959+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:38.748 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:38 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:38.362+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:38.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:38 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:38.362+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:38.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:38 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:38.362+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:38.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:38 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:38.362+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:39.381 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.005+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:39.382 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.005+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:39.382 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.005+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:39.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:39.381+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:39.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:39.381+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:39.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:39.381+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:39.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:39.381+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:40.247 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.976+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:40.247 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.976+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:40.247 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:39 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:39.976+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:40.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:40 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:40.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:40.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:40 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:40.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:40.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:40 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:40.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:40.499 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:40 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:40.364+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:40.500 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:40 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:40.365+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:41.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.017+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:41.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.017+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:41.249 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.017+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:41.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:41.377+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:41.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:41.377+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:41.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:41.377+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:41.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:41.377+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:41.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:41 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:41.378+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:42.352 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.999+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.526464+0000 front 2024-04-25T05:55:49.526499+0000 (oldest deadline 2024-04-25T05:56:13.626716+0000) 2024-04-25T05:56:42.352 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.999+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:58.928038+0000 front 2024-04-25T05:55:58.928113+0000 (oldest deadline 2024-04-25T05:56:24.227922+0000) 2024-04-25T05:56:42.352 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6[47290]: 2024-04-25T05:56:41.999+0000 7f51d13ad700 -1 osd.6 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.129034+0000 front 2024-04-25T05:56:07.129077+0000 (oldest deadline 2024-04-25T05:56:31.228958+0000) 2024-04-25T05:56:42.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:42.351+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:42.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:42.351+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:42.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:42.351+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:42.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:42.351+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:42.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:42 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:42.351+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:43.650 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:43 smithi060 podman[151909]: 2024-04-25 05:56:43.436691019 +0000 UTC m=+5.643484571 container died 82873dc228c8d9f3d28f35cc746a151c69b4a56b1bf26c1f4d1b321b421cc330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-25T05:56:43.651 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:43.395+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:43.651 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:43.395+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:43.651 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:43.395+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:43.651 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:43.395+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:43.651 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:43 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:43.395+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:43.999 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:43 smithi060 podman[151909]: 2024-04-25 05:56:43.818816564 +0000 UTC m=+6.025610118 container cleanup 82873dc228c8d9f3d28f35cc746a151c69b4a56b1bf26c1f4d1b321b421cc330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T05:56:43.999 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:43 smithi060 bash[151909]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6 2024-04-25T05:56:44.371 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[151929]: 2024-04-25 05:56:44.070739936 +0000 UTC m=+0.632906151 container remove 82873dc228c8d9f3d28f35cc746a151c69b4a56b1bf26c1f4d1b321b421cc330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T05:56:44.674 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:44 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:44.370+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:44.674 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:44 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:44.370+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:44.674 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:44 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:44.370+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:44.675 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:44 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:44.370+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:44.675 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:44 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:44.371+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:44.675 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.304096479 +0000 UTC m=+0.018603934 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:44.675 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.405059238 +0000 UTC m=+0.119566689 container create 7547ed0fcad656043ded777608a3518a5c814b84c250ea55cca0b03c40f80480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T05:56:44.675 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.674950561 +0000 UTC m=+0.389458010 container init 7547ed0fcad656043ded777608a3518a5c814b84c250ea55cca0b03c40f80480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T05:56:44.988 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.678546793 +0000 UTC m=+0.393054230 container start 7547ed0fcad656043ded777608a3518a5c814b84c250ea55cca0b03c40f80480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T05:56:44.989 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.766201783 +0000 UTC m=+0.480709237 container attach 7547ed0fcad656043ded777608a3518a5c814b84c250ea55cca0b03c40f80480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T05:56:44.989 INFO:journalctl@ceph.osd.6.smithi060.stdout:Apr 25 05:56:44 smithi060 podman[152024]: 2024-04-25 05:56:44.98907547 +0000 UTC m=+0.703582925 container died 7547ed0fcad656043ded777608a3518a5c814b84c250ea55cca0b03c40f80480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T05:56:45.644 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:45.414+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:45.645 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:45.414+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:45.645 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:45.414+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:45.645 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:45.414+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:45.645 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:45.414+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:45.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T05:56:45.715 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:45.715 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T05:56:45.715 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T05:56:45.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.7 2024-04-25T05:56:45.969 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:45 smithi060 systemd[1]: Stopping Ceph osd.7 for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:46.249 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.073+0000 7f2b277b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:46.249 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.073+0000 7f2b277b6700 -1 osd.7 621 *** Got signal Terminated *** 2024-04-25T05:56:46.249 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.073+0000 7f2b277b6700 -1 osd.7 621 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T05:56:46.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.422+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:46.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.422+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:46.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.422+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:46.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.422+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:46.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:46 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:46.423+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:47.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:47 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:47.405+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:47.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:47 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:47.405+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:47.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:47 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:47.405+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:47.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:47 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:47.405+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:47.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:47 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:47.405+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:48.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:48 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:48.411+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:48.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:48 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:48.411+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:48.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:48 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:48.411+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:48.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:48 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:48.411+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:48.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:48 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:48.412+0000 7f2b23daa700 -1 osd.7 621 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24397.0:1311 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e621) 2024-04-25T05:56:49.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:49 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:49.387+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:49.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:49 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:49.387+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:49.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:49 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:49.387+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:49.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:49 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:49.387+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:50.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:50 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:50.359+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6803 osd.0 since back 2024-04-25T05:55:49.775176+0000 front 2024-04-25T05:55:49.775101+0000 (oldest deadline 2024-04-25T05:56:14.475020+0000) 2024-04-25T05:56:50.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:50 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:50.359+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6807 osd.1 since back 2024-04-25T05:55:57.976135+0000 front 2024-04-25T05:55:57.976003+0000 (oldest deadline 2024-04-25T05:56:23.275899+0000) 2024-04-25T05:56:50.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:50 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:50.359+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6811 osd.2 since back 2024-04-25T05:56:07.177911+0000 front 2024-04-25T05:56:07.177849+0000 (oldest deadline 2024-04-25T05:56:31.277744+0000) 2024-04-25T05:56:50.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:50 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7[52996]: 2024-04-25T05:56:50.359+0000 7f2b23daa700 -1 osd.7 621 heartbeat_check: no reply from 172.21.15.43:6815 osd.3 since back 2024-04-25T05:56:15.879272+0000 front 2024-04-25T05:56:15.879253+0000 (oldest deadline 2024-04-25T05:56:39.979171+0000) 2024-04-25T05:56:51.999 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:51 smithi060 podman[152131]: 2024-04-25 05:56:51.621066918 +0000 UTC m=+5.689955042 container died 6ab4a1279b3073523358a0b23d57d42dd60f30be91f04f3c944961e6d76de23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T05:56:52.474 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:52 smithi060 podman[152131]: 2024-04-25 05:56:52.190610779 +0000 UTC m=+6.259498956 container cleanup 6ab4a1279b3073523358a0b23d57d42dd60f30be91f04f3c944961e6d76de23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T05:56:52.475 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:52 smithi060 bash[152131]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7 2024-04-25T05:56:52.749 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:52 smithi060 podman[152159]: 2024-04-25 05:56:52.475159135 +0000 UTC m=+0.850405176 container remove 6ab4a1279b3073523358a0b23d57d42dd60f30be91f04f3c944961e6d76de23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T05:56:53.136 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:52 smithi060 podman[152250]: 2024-04-25 05:56:52.692840456 +0000 UTC m=+0.018948385 image pull 3d0c579d3b62546f09ea44dc23ec09e0c00d5c2184768532a3c2aae78479c2f1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72 2024-04-25T05:56:53.137 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:52 smithi060 podman[152250]: 2024-04-25 05:56:52.792754394 +0000 UTC m=+0.118862346 container create 96ee8eb7fc30fbf7e8f5d1167606421edc6c1080c64589603fe291e334e59d6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T05:56:53.137 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:53 smithi060 podman[152250]: 2024-04-25 05:56:53.137327164 +0000 UTC m=+0.463435073 container init 96ee8eb7fc30fbf7e8f5d1167606421edc6c1080c64589603fe291e334e59d6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_BRANCH=HEAD) 2024-04-25T05:56:53.457 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:53 smithi060 podman[152250]: 2024-04-25 05:56:53.140844255 +0000 UTC m=+0.466952166 container start 96ee8eb7fc30fbf7e8f5d1167606421edc6c1080c64589603fe291e334e59d6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T05:56:53.457 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:53 smithi060 podman[152250]: 2024-04-25 05:56:53.228773236 +0000 UTC m=+0.554881160 container attach 96ee8eb7fc30fbf7e8f5d1167606421edc6c1080c64589603fe291e334e59d6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T05:56:53.748 INFO:journalctl@ceph.osd.7.smithi060.stdout:Apr 25 05:56:53 smithi060 podman[152250]: 2024-04-25 05:56:53.458061094 +0000 UTC m=+0.784169009 container died 96ee8eb7fc30fbf7e8f5d1167606421edc6c1080c64589603fe291e334e59d6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T05:56:54.039 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T05:56:54.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:56:54.079 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T05:56:54.079 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-25T05:56:54.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@rgw.foo.a 2024-04-25T05:56:54.389 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:56:54 smithi043 systemd[1]: Stopping Ceph rgw.foo.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:56:54.641 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:56:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a[64036]: 2024-04-25T05:56:54.389+0000 7eff8c38c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T05:56:54.641 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:56:54 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a[64036]: 2024-04-25T05:56:54.389+0000 7eff9ac15ac0 -1 shutting down 2024-04-25T05:57:04.806 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:04 smithi043 bash[211966]: time="2024-04-25T05:57:04Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-04-25T05:57:04.806 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:04 smithi043 podman[211966]: 2024-04-25 05:57:04.409234176 +0000 UTC m=+10.132817995 container died f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T05:57:05.101 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:04 smithi043 podman[211966]: 2024-04-25 05:57:04.99677347 +0000 UTC m=+10.720357334 container cleanup f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T05:57:05.102 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:04 smithi043 bash[211966]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a 2024-04-25T05:57:05.362 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:05 smithi043 podman[211993]: 2024-04-25 05:57:05.264749085 +0000 UTC m=+0.850065144 container remove f54681f61880842d78ebbb7d2440f6cb5402c7adaecc18717e26517fae56f457 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T05:57:05.363 INFO:journalctl@ceph.rgw.foo.a.smithi043.stdout:Apr 25 05:57:05 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-04-25T05:57:05.378 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@rgw.foo.a.service' 2024-04-25T05:57:05.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:57:05.414 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-25T05:57:05.414 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-25T05:57:05.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-84884b5e-02c4-11ef-bc93-c7b262605968@prometheus.a 2024-04-25T05:57:05.709 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 systemd[1]: Stopping Ceph prometheus.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:05.968 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-25T05:57:05.968 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-25T05:57:05.968 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-25T05:57:05.968 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.709Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.716Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.716Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.716Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a[66975]: ts=2024-04-25T05:57:05.716Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-25T05:57:05.969 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:05 smithi060 podman[152357]: 2024-04-25 05:57:05.719641025 +0000 UTC m=+0.106325063 container died 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:06.243 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:06 smithi060 podman[152357]: 2024-04-25 05:57:06.067415937 +0000 UTC m=+0.454099975 container cleanup 988f046836bc540e904dd30634289d25932cfe9f7b2e665503917d059661fb8a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:06.243 INFO:journalctl@ceph.prometheus.a.smithi060.stdout:Apr 25 05:57:06 smithi060 bash[152357]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-prometheus-a 2024-04-25T05:57:06.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-84884b5e-02c4-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-25T05:57:06.493 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T05:57:06.493 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-25T05:57:06.493 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T05:57:06.685 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:57:08.618 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:08 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:08.891 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:08 smithi043 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a[80893]: ts=2024-04-25T05:57:08.618Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-25T05:57:08.891 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:08 smithi043 podman[212468]: 2024-04-25 05:57:08.669764749 +0000 UTC m=+0.166377325 container died a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:09.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 podman[212468]: 2024-04-25 05:57:09.15212248 +0000 UTC m=+0.648735066 container cleanup a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:09.391 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 bash[212468]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a 2024-04-25T05:57:09.822 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 podman[212482]: 2024-04-25 05:57:09.403150956 +0000 UTC m=+0.734979341 container remove a27ec0255f23895b3b3cbf75363f93a6895c6ffb097a031d85f7491ddd60b4f3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:09.822 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 podman[212482]: 2024-04-25 05:57:09.503861375 +0000 UTC m=+0.835689762 volume remove ee9d1421eb72f435a1b025a94dd29b669bbc2882c34d5f035ed729700dbc624e 2024-04-25T05:57:09.822 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-25T05:57:09.822 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:57:09.822 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:Apr 25 05:57:09 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.712s CPU time. 2024-04-25T05:57:10.092 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:09 smithi043 systemd[1]: Stopping Ceph node-exporter.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:10.390 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:10 smithi043 podman[212597]: 2024-04-25 05:57:10.092593951 +0000 UTC m=+0.125531383 container died 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:10.641 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:10 smithi043 podman[212597]: 2024-04-25 05:57:10.598868022 +0000 UTC m=+0.631805464 container cleanup 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:10.641 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:10 smithi043 bash[212597]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a 2024-04-25T05:57:11.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:10 smithi043 podman[212611]: 2024-04-25 05:57:10.866707497 +0000 UTC m=+0.778591408 container remove 9da72b3def8b10dd1404145574a98ce9f64e0d1eb4a86c70ff1dc9c059dc0afa (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-25T05:57:11.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:10 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-25T05:57:11.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:11 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-25T05:57:11.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:11 smithi043 systemd[1]: Stopped Ceph node-exporter.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:57:11.142 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:Apr 25 05:57:11 smithi043 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 5.131s CPU time. 2024-04-25T05:57:11.791 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T05:57:11.990 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T05:57:13.605 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:13 smithi060 systemd[1]: Stopping Ceph iscsi.iscsi.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:13.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:13 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a[58050]: debug Shutdown received 2024-04-25T05:57:23.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:23 smithi060 bash[152770]: time="2024-04-25T05:57:23Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-04-25T05:57:23.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:23 smithi060 podman[152770]: 2024-04-25 05:57:23.629289508 +0000 UTC m=+10.134745836 container died 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T05:57:24.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 podman[152770]: 2024-04-25 05:57:24.104412974 +0000 UTC m=+10.609869300 container cleanup 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T05:57:24.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 bash[152770]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a 2024-04-25T05:57:24.677 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 podman[152797]: 2024-04-25 05:57:24.330364169 +0000 UTC m=+0.701441477 container remove 04baa5483e47c3d31f37b3770b1d23bd060c387aff99a484c6e9165204c4b069 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:53547c167dfd1636c2f97ada3832867c494b7ad3a6f460c09753bd0677c85e72, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-04-24-1625-d6f5f4e) 2024-04-25T05:57:24.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-25T05:57:24.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-25T05:57:24.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 systemd[1]: Stopped Ceph iscsi.iscsi.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:57:24.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi060.stdout:Apr 25 05:57:24 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Consumed 3.491s CPU time. 2024-04-25T05:57:25.999 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:25 smithi060 systemd[1]: Stopping Ceph grafana.a for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:26.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=server t=2024-04-25T05:57:26.120891468Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-25T05:57:26.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=tracing t=2024-04-25T05:57:26.121278805Z level=info msg="Closing tracing" 2024-04-25T05:57:26.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a[61106]: logger=ticker t=2024-04-25T05:57:26.121349238Z level=info msg=stopped last_tick=2024-04-25T05:57:20Z 2024-04-25T05:57:26.499 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 podman[153060]: 2024-04-25 05:57:26.131162107 +0000 UTC m=+0.092531835 container died 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:57:26.962 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 podman[153060]: 2024-04-25 05:57:26.637507889 +0000 UTC m=+0.598877623 container cleanup 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:57:26.963 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 bash[153060]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a 2024-04-25T05:57:27.248 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:26 smithi060 podman[153074]: 2024-04-25 05:57:26.96343187 +0000 UTC m=+0.829828036 container remove 1e0fff93e26e9e88aec95bdb7c7f9f41625a787a2c76ff043cab2d7eb8cd3347 (image=quay.io/ceph/grafana:9.4.12, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-25T05:57:27.249 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:27 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-25T05:57:27.249 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:27 smithi060 systemd[1]: Stopped Ceph grafana.a for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:57:27.249 INFO:journalctl@ceph.grafana.a.smithi060.stdout:Apr 25 05:57:27 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@grafana.a.service: Consumed 16.712s CPU time. 2024-04-25T05:57:27.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:27 smithi060 systemd[1]: Stopping Ceph node-exporter.b for 84884b5e-02c4-11ef-bc93-c7b262605968... 2024-04-25T05:57:27.749 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:27 smithi060 podman[153186]: 2024-04-25 05:57:27.568734363 +0000 UTC m=+0.096855939 container died 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:57:28.075 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:27 smithi060 podman[153186]: 2024-04-25 05:57:27.899857126 +0000 UTC m=+0.427978691 container cleanup 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:57:28.075 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:27 smithi060 bash[153186]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b 2024-04-25T05:57:28.326 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:28 smithi060 podman[153200]: 2024-04-25 05:57:28.075902629 +0000 UTC m=+0.511696636 container remove 2292635b54c545052b9bc95d00ba57789e7371ca7dae6d0539b55ceb77c1ba25 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-84884b5e-02c4-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-25T05:57:28.326 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:28 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-25T05:57:28.743 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:28 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-25T05:57:28.744 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:28 smithi060 systemd[1]: Stopped Ceph node-exporter.b for 84884b5e-02c4-11ef-bc93-c7b262605968. 2024-04-25T05:57:28.744 INFO:journalctl@ceph.node-exporter.b.smithi060.stdout:Apr 25 05:57:28 smithi060 systemd[1]: ceph-84884b5e-02c4-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 4.833s CPU time. 2024-04-25T05:57:29.054 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:57:29.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T05:57:29.114 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T05:57:29.116 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi043/crash 2024-04-25T05:57:29.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash -- . 2024-04-25T05:57:29.172 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T05:57:29.173 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2024-04-25T05:57:29.175 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi060/crash 2024-04-25T05:57:29.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash -- . 2024-04-25T05:57:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T05:57:29.222 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2024-04-25T05:57:29.223 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T05:57:29.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-25T05:57:29.302 INFO:tasks.cephadm:Compressing logs... 2024-04-25T05:57:29.302 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T05:57:29.305 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T05:57:29.353 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T05:57:29.353 INFO:teuthology.orchestra.run.smithi060.stderr:: No such file or directory 2024-04-25T05:57:29.356 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose --/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T05:57:29.357 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T05:57:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T05:57:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T05:57:29.361 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log 2024-04-25T05:57:29.362 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log 2024-04-25T05:57:29.365 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log: 58.8% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-04-25T05:57:29.373 INFO:teuthology.orchestra.run.smithi043.stderr: 91.6% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T05:57:29.377 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.b.log: 67.8% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-25T05:57:29.441 INFO:teuthology.orchestra.run.smithi043.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T05:57:29.457 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T05:57:29.518 INFO:teuthology.orchestra.run.smithi060.stderr: 91.3% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T05:57:29.544 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T05:57:29.577 INFO:teuthology.orchestra.run.smithi043.stderr: 91.3% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T05:57:29.633 INFO:teuthology.orchestra.run.smithi060.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T05:57:29.778 INFO:teuthology.orchestra.run.smithi043.stderr: 94.0% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T05:57:29.803 INFO:teuthology.orchestra.run.smithi060.stderr: 81.6% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T05:57:29.827 INFO:teuthology.orchestra.run.smithi060.stderr: 85.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T05:57:29.865 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T05:57:33.265 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T05:57:33.457 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T05:57:36.160 INFO:teuthology.orchestra.run.smithi060.stderr: 91.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T05:57:58.533 INFO:teuthology.orchestra.run.smithi043.stderr: 90.3% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T05:58:52.406 INFO:teuthology.orchestra.run.smithi043.stderr: 93.1% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T05:59:54.247 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T06:01:18.765 INFO:teuthology.orchestra.run.smithi060.stderr: 93.2% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T06:02:15.204 INFO:teuthology.orchestra.run.smithi060.stderr: 93.3% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T06:02:28.467 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T06:02:28.470 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-04-25T06:02:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:real 4m59.150s 2024-04-25T06:02:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:user 5m34.511s 2024-04-25T06:02:28.470 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m13.414s 2024-04-25T06:02:58.875 INFO:teuthology.orchestra.run.smithi043.stderr: 92.8% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T06:03:51.212 INFO:teuthology.orchestra.run.smithi043.stderr: 93.6% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T06:05:14.405 INFO:teuthology.orchestra.run.smithi043.stderr: 92.9% -- replaced with /var/log/ceph/84884b5e-02c4-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T06:05:14.406 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-25T06:05:14.406 INFO:teuthology.orchestra.run.smithi043.stderr:real 7m45.087s 2024-04-25T06:05:14.407 INFO:teuthology.orchestra.run.smithi043.stderr:user 7m0.566s 2024-04-25T06:05:14.407 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m17.046s 2024-04-25T06:05:14.407 INFO:tasks.cephadm:Archiving logs... 2024-04-25T06:05:14.407 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi043/log 2024-04-25T06:05:14.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T06:06:17.112 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi060/log 2024-04-25T06:06:17.113 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T06:07:05.693 INFO:tasks.cephadm:Removing cluster... 2024-04-25T06:07:05.693 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 --force 2024-04-25T06:07:05.922 INFO:teuthology.orchestra.run.smithi043.stdout:Deleting cluster with fsid: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T06:07:06.689 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 84884b5e-02c4-11ef-bc93-c7b262605968 --force 2024-04-25T06:07:06.907 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: 84884b5e-02c4-11ef-bc93-c7b262605968 2024-04-25T06:07:07.556 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T06:07:07.556 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T06:07:07.573 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T06:07:07.589 INFO:tasks.cephadm:Teardown complete 2024-04-25T06:07:07.589 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T06:07:07.598 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T06:07:07.598 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T06:07:07.615 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T06:07:07.704 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T06:07:07.713 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T06:07:07.713 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T06:07:07.715 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T06:07:07.728 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-04-25T06:07:07.730 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:^- hv01.front.sepia.ceph.com 2 6 377 62 -259us[ -263us] +/- 594ms 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -847us[ -852us] +/- 62ms 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1096us[+1096us] +/- 72ms 2024-04-25T06:07:07.743 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T06:07:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T06:07:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-25T06:07:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:^- hv01.front.sepia.ceph.com 2 6 377 0 -331us[ -331us] +/- 595ms 2024-04-25T06:07:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -940us[ -925us] +/- 62ms 2024-04-25T06:07:07.751 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +1003us[+1003us] +/- 73ms 2024-04-25T06:07:07.751 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T06:07:07.751 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T06:07:07.760 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T06:07:07.761 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T06:07:07.768 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T06:07:07.907 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T06:07:07.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-25T06:07:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:07:08.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-25T06:07:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:07:08.026 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T06:07:08.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T06:07:08.046 INFO:teuthology.task.internal:Duration was 2744.254651 seconds 2024-04-25T06:07:08.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T06:07:08.055 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T06:07:08.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T06:07:08.058 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T06:07:08.113 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T06:07:08.125 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T06:07:08.291 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T06:07:08.291 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-04-25T06:07:08.291 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T06:07:08.312 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-04-25T06:07:08.312 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T06:07:08.339 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T06:07:08.339 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T06:07:08.354 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T06:07:08.405 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T06:07:08.405 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T06:07:08.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T06:07:10.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T06:07:10.161 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T06:07:10.161 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T06:07:10.215 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T06:07:10.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T06:07:10.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:07:10.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:07:10.297 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-04-25T06:07:10.327 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-04-25T06:07:10.350 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:07:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:07:10.369 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T06:07:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:07:10.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T06:07:10.416 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T06:07:10.416 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi043 2024-04-25T06:07:10.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T06:07:10.486 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-04-25_03:53:55-orch:cephadm-wip-adk-testing-2024-04-24-1625-distro-default-smithi/7672832/remote/smithi060 2024-04-25T06:07:10.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T06:07:10.527 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T06:07:10.527 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T06:07:10.529 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T06:07:10.576 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T06:07:10.587 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T06:07:10.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T06:07:10.597 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T06:07:10.598 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T06:07:10.600 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T06:07:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 06:07 /home/ubuntu/cephtest 2024-04-25T06:07:10.631 INFO:teuthology.orchestra.run.smithi060.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 06:07 /home/ubuntu/cephtest 2024-04-25T06:07:10.633 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T06:07:10.643 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T06:07:10.718 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 2744.2546513080597 flavor: default owner: scheduled_adking@teuthology success: true 2024-04-25T06:07:10.718 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T06:07:10.790 INFO:teuthology.run:pass